Tag:Crawlers

All the articles with the tag "Crawlers".

• A Scanner Named ARWP: Tracing a Curious User Agent Back to Its Source

  hrbrmstr·26 Aug, 2026

  I traced an unknown user agent (arwp-scanner/0.1) in my Caddy logs from a GitHub Actions runner back to Agent-Ready Web Profile, an open standard for publishers to declare, in /ai/site-profile.json, how agents may read a site — and I explain why I'll publish one.