If you remember the project I would be interested to see it!

But I've seen some AI poisoning sink holes before too, a novel concept as well. I have not heard of real world experiences of them yet.

That's a tarpit that you're describing, like iocaine or nepthasis. Those are to feed the crawler junk data to try and make their eventual output bad.

Anubis tries to not let the AI crawlers in at all.

generates an infinitely deep tree

Wouldn't the bot simply limit the depth of it's seek?

It requires a bunch of browser features that non-user browsers don't have, and the proof-of-work part is like the least relevant piece in this that only gets invoked once a week or so to generate a unique cookie.

I sometimes have the feeling that as soon as some crypto-currency related features are mentioned people shut off part of their brain. Either because they hate crypto-currencies or because crypto-currency scammers have trained them to only look at some technical implementation details and fail to see the larger picture that they are being scammed.

Yes, Anubis uses proof of work, like some cryptocurrencies do as well, to slow down/mitigate mass scale crawling by making them do expensive computation.

https://lemmy.world/post/27101209 has a great article attached to it about this.

--

Edit: Just to be clear, this doesn't mine any cryptos, just uses same idea for slowing down the requests.

There are some detailed instructions on the docs site, tho I agree it'd be nice to have in the readme, too.

Sounds like the dev was not expecting this much interest for the project out of nowhere so there will def be gaps.

Or even a quick link to the relevant portion of the docs at least would be cool

The docker image page has it

For those not aware, nepenthes is an example for the above mentioned approach !

You have a point here.

But when you consider the current worlds web traffic, this isn't actually the case today. For example Gnome project who was forced to start using this on their gitlab, 97% of their traffic could not complete this PoW calculation.

IE - they require only a fraction of computational cost to serve their gitlab, which saves a lot of resources, coal, and most importantly, time of hundreds of real humans.

(Source for numbers)

Hopefully in the future we can move back to proper netiquette and just plain old robots.txt file!

I don't think AI companies care, and I wholeheartedly support any and all FOSS projects using PoW when serving their websites. I'd rather have that than have them go down

What's the ffxiv reference here?

Anubis is from Egyptian mythology.