Tag:Crawlers
All the articles with the tag "Crawlers".
A Scanner Named ARWP: Tracing a Curious User Agent Back to Its Source
I traced an unknown user agent (arwp-scanner/0.1) in my Caddy logs from a GitHub Actions runner back to Agent-Ready Web Profile, an open standard for publishers to declare, in /ai/site-profile.json, how agents may read a site — and I explain why I'll publish one.