The guy is scanning eyeballs for a living, I don’t believe he has any respect for a small text file in your web server.
Why is everyone outraged when Google/Microsoft/Yahoo and others have scraped the whole internet for two decades and are also massively profiting from that data?
There’s a significant difference in the purpose of the scraping.
Google et al. run crawlers primarily to populate their search engines. This is a net positive for those whose sites get scraped, because when they appear in a search engine they get more traffic, more page views, more ad revenue. People view content directly from those who created it, meaning those creators (regardless of whoever they are) get full credit. Yes, Google makes money too, but site owners are not left in the cold.
ChatGPT and other LLM’s works by combing its huge database of known content its “learned” to cook up an answer through fast math magic. Content it scrapes to populate this database can be regurgitated at any time, only now its been completely processed and obfuscated to an insane degree. Any attribution of content is completely stripped in the final product, even if it ends up being a word-for-word reproduction. Everything OpenAI charges for its LLM goes directly to OpenAI, and those who have created content to train it will never even know it was used without their consent.
Essentially, LLM’s operate like a huge middle school plagiarism machine shitting all over any concept of copyright, only now they’re making billions off said plagiarism with no plans to stop. It’s a huge ethical conundrum and one I heavily disagree with.
Google et al. run crawlers primarily to populate their search engines. This is a net positive for those whose sites get scraped, because when they appear in a search engine they get more traffic, more page views, more ad revenue.
This is not necessarily true. Google’s instant answers are designed to use the content from websites to answer searcher’s questions without actually leading them to the website. Whether you’re trying to find the definition for the word, the year a movie came out, or a recipe, Google will take the information they’ve scraped from a website and present it on their page with a link to the website. Their hope is that the information will be useful enough that the searcher never needs to leave the search engine.
This might be useful for searchers, but it doesn’t help the sites much. This is one of the reasons news companies attempted to take action against Google a few years ago. I think a search engine should provide some useful utilities, but not try to replace the sites they’re ostensibly attempting to connect users to. Not all search engines are like this, but Google is.
Because until now they weren’t competing against individual content creators.
Sure but they were competing against all sorts of other jobs by automating them.
They weren’t stealing content until now.
Unsure why this is where the line is drawn, but okay.
Because it’s perverse for someone to create content if successful content will surely be stolen and used against its creator.
Could somebody explain why this is bad? I’m not a fan of all this AI stuff. But I can’t think of an argument besides “Big tech is bad and they should not make money if they use public information to do so.”
I’m genuinely curious. There may be massive amounts of data being processed. But only public data, right? If they can use that data for something, isn’t that something positive? Or at the very least nothing negative? I always thought anything that is posted in public spaces means making it available for anyone to use anyway. So what am I missing here?
If the results were also open and public, it’d be a different conversation.
This is more akin to rain water collection up-hill and selling it back to the people downhill. It’s privatization of a public resource.
This is more akin to rain water collection up-hill and selling it back to the people downhill
Not really, anyone can go and collect the same water they are collecting. And it’s happening, open source LLMs are quickly catching up and a shit ton of other companies are also crawling the exact same data.
“anyone”. I hate when people use this word knowing full well it’s not true in meaning. “Nothing is stopping you from spending millions of dollars on your own LLM.” Ok.
The web is a bunch of information that is public, sure. People don’t have a reasonable expectation of privacy but they used to have a reasonable expectation that their information would be used in a very specific fashion. Especially in the US where there is a default copyright claim on data. And crawling the web may ignore text that states you can’t use the data. Even if you include a clause saying by accessing the data you agree to the claim. That only works against little people. The “anyone” that can’t actually just go and build a LLM.
Sure but that applies to literally a million other things. There is an absolute ton of shit that companies do that individuals can’t which is still built off of others.
A company can go spend 1B on a new state of the art nuclear reactor which will bring in billions over it’s life time. Will the physicist who discovered the underlying math see any of the profit? No, probably not. And if they do it won’t be nearly a “fair share”. Nor will all the publishers and authors who generated the learning materials that the people working for said company used to build it.
There is tons of public knowledge that can only be utilized with a huge investment, that’s just how a lot of innovation works.
And OpenAI also has a ton of competitors. Sure they have the lead for now but thousands of other companies are also scraping and building LLMs.
You’re not really going to win this argument as I’m an anti-capitalist. So I agree a lot of that stuff is wrong too. I don’t believe you should own other’s labor. The employees should own the company. And I don’t believe in copyright, but it does exist and it’s enforced against individuals, so it’s only fair it’s enforced against them as well. I don’t think you should be allowed to blindly scrape when information could be behind an agreement to use it in a specific manner if accessed. Plus I think it should be opt in based off it being a new use and therefore a new right of copyright. Just as suddenly actors need to worry if they’ll be scanned and owned by a Hollywood studio now. It’s something a reasonable person wouldn’t expect. And that’s why past works are protected from that use.
Things behind a third party privacy policy, sure. You agreed to it, whatever. But your own website? I’m not feeling it.
Charitable of you to believe they’d listen to robots.txt.
I just hide the N word, long R word very well several dozen times throughout my site so they have to manually blacklist it.
Stupid question: where is the difference between the long r word and the r word?
I’m sure you can guess the long r word, its pretty well “regarded”, I honestly don’t know what the short r word is, if it even exists. I just said long r to differentiate it from the hard r word.
