It makes the ridiculous prices they were quoting make sense. Because giving API access is giving a key to all that data, which they can then turn around and covertly sell access to. So they priced it so that they wouldn’t have to sell the data at wholesale value to apps that could turn around and undercut their AI training prices.
It’s the same reason why they were considering blocking Google search because Google (or any search engine) uses a crawler to look at all that data and you can’t allow Google to continue without leaving it open to any other crawler, like say an AI training data crawler.
Same thing with any push to make users log in to view comment threads (and it wouldn’t surprise me if that’s what Musk was thinking of when he was doing/considering the same with Twitter). If only users can access the comment data, then it’s easier to see when a user is reading too much data, or rate limit them. Also the move towards only showing a bit of a comment thread by default.
But that data is the only reason people visit the site and provide more data, so I don’t see this problem ever fully going away for them. The problem they are trying to solve is how to give access to enough data to engage users enough to provide more data while preventing AI trainers from getting that same data for free. If I wanted to, I bet I could write something that would fill a database with comment data and metadata while browsing normally in less than a day and then a bit longer to automate the browsing entirely (depending on what kind of bot detection the site uses). There’s no way for Reddit to stop the manual browsing version and the automated one will be an arms race that will also end in no way for it to be stopped because it emulates a real user to an undetectable level.
It makes the ridiculous prices they were quoting make sense. Because giving API access is giving a key to all that data, which they can then turn around and covertly sell access to. So they priced it so that they wouldn’t have to sell the data at wholesale value to apps that could turn around and undercut their AI training prices.
It’s the same reason why they were considering blocking Google search because Google (or any search engine) uses a crawler to look at all that data and you can’t allow Google to continue without leaving it open to any other crawler, like say an AI training data crawler.
Same thing with any push to make users log in to view comment threads (and it wouldn’t surprise me if that’s what Musk was thinking of when he was doing/considering the same with Twitter). If only users can access the comment data, then it’s easier to see when a user is reading too much data, or rate limit them. Also the move towards only showing a bit of a comment thread by default.
But that data is the only reason people visit the site and provide more data, so I don’t see this problem ever fully going away for them. The problem they are trying to solve is how to give access to enough data to engage users enough to provide more data while preventing AI trainers from getting that same data for free. If I wanted to, I bet I could write something that would fill a database with comment data and metadata while browsing normally in less than a day and then a bit longer to automate the browsing entirely (depending on what kind of bot detection the site uses). There’s no way for Reddit to stop the manual browsing version and the automated one will be an arms race that will also end in no way for it to be stopped because it emulates a real user to an undetectable level.