Cynicus Rex@lemmy.ml to Privacy@lemmy.ml • 3 months agoHow to block AI Crawler Bots using robots.txt fileexternal-linkmessage-square8 fedilinkarrow-up146
arrow-up146external-linkHow to block AI Crawler Bots using robots.txt fileCynicus Rex@lemmy.ml to Privacy@lemmy.ml • 3 months agomessage-square8 Commentsfedilink
minus-squaremox@lemmy.sdf.orghexbear3·3 months agoOf course, nothing stops a bot from picking a user agent field that exactly matches a web browser. linkfedilink
Of course, nothing stops a bot from picking a user agent field that exactly matches a web browser.