Some context about this here: https://arstechnica.com/information-technology/2023/08/openai-details-how-to-keep-chatgpt-from-gobbling-up-website-data/

the robots.txt would be updated with this entry

User-agent: GPTBot
Disallow: /

Obviously this is meaningless against non-openai scrapers or anyone who just doesn't give a shit.

  • jungekatz [comrade/them, undecided]
    ·
    1 year ago

    Idk , I am not all for a private entity scraping comments and then using it , thats basically us doing free labour for them! No cool !