• SacrificedBeans@lemmy.world
    link
    fedilink
    arrow-up
    12
    ·
    7 months ago

    Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.