Been stewing at it for longer than you have, buddy.

You’re not correcting anything, dude.

The entire point here is that the bots are being disrespectful.

deleted by creator

They’re not trying to do that in particular, but on a site like lemmy the same content might be linked a dozen different ways:

https://lemmy.world/comment/25863752 https://lemmy.world/post/51995780/25862545 https://lemmy.world/post/51995780/25860801 https://lemmy.world/post/51995780/25858204 https://lemmy.world/post/51995780/25857153 https://lemmy.world/post/51995780/25857033

It’s not necessarily the big companies themselves doing it? But, there’s a push right now to scrape as much data as possible to feed the training of the AI models, and if your business model is entirely based on selling that data you don’t really care who you’re going to piss off. So, in turn these new scraper bots basically behave in the same way that attack bots do where they’ll be behind VPNs and switch IPs and stuff if you block them.

Ordinary search crawlers are throttled and they do stuff like respect your robots.txt and sitemap.xml (directives that tell the crawler what links to scan or not and how frequently). These naive crawlers hit everything at max speed, and on a content-heavy site like lemmy that can easily take out a server.

They’re simply not censoring it. The BBC already got thrashed for this bullshit.

Gets us on the other end too as we do newsletters and other web work for small orgs, so our poor little web servers basically get DDOS’d by the scanners when we send out a newsletter with a bunch of backlinks.

Your email service’s scanner might, because some of them go through every link and check what’s on the page.