Cynicus Rex@lemmy.ml to Privacy@lemmy.ml • 3 months agoHow to block AI Crawler Bots using robots.txt fileexternal-linkmessage-square8 fedilinkarrow-up146
arrow-up146external-linkHow to block AI Crawler Bots using robots.txt fileCynicus Rex@lemmy.ml to Privacy@lemmy.ml • 3 months agomessage-square8 Commentsfedilink
minus-squareCynicus Rex@lemmy.mlhexagonhexbear2·edit-23 months agoLies, as in that it's not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out? linkfedilink
minus-squareDa Bald Eagul@feddit.nlhexbear12·3 months agoThat is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered. linkfedilink
Lies, as in that it's not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.