plinky [he/him] to technology • 4 months agoThe Backlash Against AI Scraping Is Real and Measurableexternal-linkmessage-square3 fedilinkarrow-up141file-text
arrow-up141external-linkThe Backlash Against AI Scraping Is Real and Measurableplinky [he/him] to technology • 4 months agomessage-square3 Commentsfedilinkfile-text
minus-squarenat_turner_overdrive [he/him]hexbear10·4 months ago While many companies respect robots.txt instructions, some do not. Several large companies, including Perplexity, have been caught using proxies and surreptitious user agents to circumvent and ignore robots.txt. respecting robots.txt already died link
minus-squarethethirdgracchi [he/him, they/them]hexbear8·4 months agorobots.txt has been dead for over a decade. Massive web scraping is big business at this point. link
minus-squarenat_turner_overdrive [he/him]hexbear8·4 months agoFor sure, I just mean in the context of AI since OP suggested it wouldn't be respected much longer link
respecting robots.txt already died
robots.txt has been dead for over a decade. Massive web scraping is big business at this point.
For sure, I just mean in the context of AI since OP suggested it wouldn't be respected much longer