Cynicus Rex@lemmy.ml to Privacy@lemmy.ml • 3 months agoHow to block AI Crawler Bots using robots.txt fileexternal-linkmessage-square8 fedilinkarrow-up146
arrow-up146external-linkHow to block AI Crawler Bots using robots.txt fileCynicus Rex@lemmy.ml to Privacy@lemmy.ml • 3 months agomessage-square8 Commentsfedilink
minus-squareDa Bald Eagul@feddit.nlhexbear12·3 months agoThat is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered. linkfedilink
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.