• Onno (VK6FLAB)@lemmy.radio
    ·
    3 months ago

    This does not block anything at all.

    It's a 1994 "standard" that requires voluntary compliance and the user-agent is a string set by the operator of the tool used to access your site.

    https://en.m.wikipedia.org/wiki/Robots.txt

    https://en.m.wikipedia.org/wiki/User-Agent_header

    In other words, the bot operator can ignore your robots.txt file and if you check your webserver logs, they can set their user-agent to whatever they like, so you cannot tell if they are ignoring you.