minus-squareProntomomo@lemmy.worldtoSelfhosted@lemmy.world•Cloudflare blocking AI crawlerslinkfedilinkEnglisharrow-up5·1 day agoI don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling linkfedilink
I don’t think this blocks crawlers. About 1/5 websites uses cloudflare, the significant thing here’s is that AI scraping is now blocked by default on most of those sites, NOT crawling