Cynicus Rex@lemmy.ml to Privacy@lemmy.ml • 3 months agoHow to block AI Crawler Bots using robots.txt fileexternal-linkmessage-square8 fedilinkarrow-up146
arrow-up146external-linkHow to block AI Crawler Bots using robots.txt fileCynicus Rex@lemmy.ml to Privacy@lemmy.ml • 3 months agomessage-square8 Commentsfedilink
minus-squaremox@lemmy.sdf.orghexbear13·3 months agoThis article lies to the reader, so it earns a -1 from me. linkfedilink
minus-squareCynicus Rex@lemmy.mlhexagonhexbear2·edit-23 months agoLies, as in that it's not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out? linkfedilink
minus-squareDa Bald Eagul@feddit.nlhexbear12·3 months agoThat is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered. linkfedilink
This article lies to the reader, so it earns a -1 from me.
Lies, as in that it's not really “blocking” but a mere unenforceable request? If you meant something else could you please point it out?
That is what they meant, yes. The title promises a block, completely preventing crawlers from accessing the site. That is not what is delivered.