In the last year, the number of websites specifically restricting OpenAI and other AI scraper bots has gone through the roof.

respecting robots.txt won't last long i suspect

  • nat_turner_overdrive [he/him]
    ·
    2 months ago

    While many companies respect robots.txt instructions, some do not. Several large companies, including Perplexity, have been caught using proxies and surreptitious user agents to circumvent and ignore robots.txt.

    respecting robots.txt already died