Consider https://arstechnica.com/robots.txt or https://www.nytimes.com/robots.txt and how they block all the stupid AI models from being able to scrape for free.

  • space_comrade [he/him]
    ·
    6 months ago

    but if entities like OpenAI say they will respect it then presumably they really will.

    Eh, will they really? It'd be pretty hard to prove they didn't respect it.