---
title: "A Scanner Named ARWP: Tracing a Curious User Agent Back to Its Source"
description: "I traced an unknown user agent (arwp-scanner/0.1) in my Caddy logs from a GitHub Actions runner back to Agent-Ready Web Profile, an open standard for publishers to declare, in /ai/site-profile.json, how agents may read a site — and I explain why I'll publish one."
pubDatetime: 2026-08-26T15:30:00Z
author: hrbrmstr
tags: ["Security", "AI", "Web", "User Agents", "Crawlers", "Site Observatory"]
---
> Original: [A Scanner Named ARWP: Tracing a Curious User Agent Back to Its Source](https://ai.rud.is/posts/2026-08-26-a-scanner-named-arwp-tracing-a-curious-user-agent-back-to-its-source)

At 23:12 UTC on 2026-08-25, a user agent I had never seen before started walking this site. It announced itself as `arwp-scanner/0.1`. The first published guess (by the daily `parallax-agent` run) was wrong: it "guessed" that the name suggested a WordPress probe ("ar WordPress"), but this site runs Caddy with no PHP anywhere. A quick Kagi query then Kagi-bang query (to GitHub), revealed the origin/project.

This post follows that trace and also concludes with a decision every publisher now faces.

### Eight sweeps, seven URIs

The Caddy log showed 56 requests (the Site Observatory report said 14, but it is deliberately non-exhaustive since it's only plucking out a specific set of facts and classification misses). The bot did eight full sweeps of seven URIs each (the number is the HTTP status code):

```text
/robots.txt (200)
/sitemap-index.xml (200)
/llms.txt (200)
/rss.xml (200)
/feed.json (200)
/ (200)
/ai/site-profile.json (404)
```

Thankfully, `robots.txt` did arrive first (at leat they're polite?). Since the rules in it gave the bot permission to scan, it hit up the sitemap, then `llms.txt`, then feeds, then the homepage HTML. Each request carried a matching media type in its `Accept` header: `text/plain` for `llms.txt`, `application/json` for the profile path. From [the code](https://github.com/dkharlanau/agent-ready-web-profile/blob/42ae7c6718a6736e0ee3053734047ec010645928/lib/scanner.mjs#L458), I confirmed that a fixed, polite checklist drives the bot, and a 404 response answered the bot's very last question: *does this site publish an agent service map?* Mine did not.

### From user agent to project

`arwp` stands for **Agent-Ready Web Profile**, an open-source project by Dennis Kharlanau at [ `github.com/dkharlanau/agent-ready-web-profile`](https://github.com/dkharlanau/agent-ready-web-profile). The scanner is one CLI subcommand of the aforelinked Node toolkit (`lib/scanner.mjs`, UA hardcoded at line 127 as of this post publish timestamp). The toolkit also ships a resolver engine, MCP servers, and JSON Schema for profile documents.

The eight source IPs that hit this site all sat in AS8075 (Microsoft's block). Censys rated every one benign and flagged them all as hosting. That alone proves little since a decent chunk of the internet's attack traffic rents or has compromised assets in Azure too. When I compared the IPs against GitHub's published Actions egress ranges from `https://api.github.com/meta`, all eight matched. Someone ran an ARWP scan job on GitHub-hosted runners against `ai.rud.is`. One request per IP per URI per sweep, timestamps within milliseconds across regions, fits a distributed batch run, not a botnet.

### What "Agent-Ready Web Profile" is

An increasing number of modern sites expose agent-relevant surfaces piecemeal: `llms.txt`, RSS and JSON feeds, OpenAPI documents, MCP servers, A2A agent cards, Agent Skills. An agent client today "needs" site-specific guesses to know which of those exist and which one matters for a given task. ARWP proposes one convention to end the guessing: a publisher-maintained JSON document at `/ai/site-profile.json`.

The spec requires five fields — `profileVersion`, `id`, `name`, `canonicalUrl`, `description` — and defines optional capability groups: web surfaces (sitemap, robots, `llms`, feeds), data (OpenAPI, dataset catalogs, [Croissant](https://mlcommons.org/working-groups/data/croissant/)), retrieval indexes, Agent Skills, WebMCP pages, MCP servers, A2A agents, identity rules, trust metadata, and namespaced extensions. You declare only what exists. The strongest spec rule is "publishers MUST NOT claim planned features, and a static JSON API must never pose as an MCP server".

The companion resolver is used to ask the following q: for a given intent — read, search, structured data, tools, or agent — which interface should a client use? It merges the profile with evidence from HTTP Link headers, [RFC 9727](https://www.rfc-editor.org/rfc/rfc9727.pdf) API catalogs, [RFC 9728](https://www.rfc-editor.org/rfc/rfc9728.pdf) OAuth metadata, A2A cards, and observed web behavior. Each claim carries its source authority rank, conflicts stay visible, and plain HTML remains an honest fallback. A site without AI metadata still resolves toward ordinary reading.

The ecosystem context explains the timing. RFC 9728 and RFC 9727 landed in 2025. The A2A protocol and Agent Skills moved from vendor concepts toward working implementations during 2025 and 2026. (These discovery formats are multiplying like rabbits.) ARWP seeks to become a meta-layer over that…pile, not another competitor inside it.

### O.K. So, should you/we help agents read your site?

The marketing framing is "make your site agent-ready." The real-world translation of that is "decide what the clankers may take."

Publishing an ARWP profile does not give anything away that a determined crawler already gets. Your sitemap, feeds, and HTML are public now. The profile adds three things: structure (one canonical list instead of surface archaeology), intent declaration (which interfaces are for machines), and trust metadata (license, citation, provenance, review policy). None of that blocks scraping. Nothing in the spec blocks scraping. ARWP grants no permissions and describes surfaces.

My position after tracing this traffic: consider cooperating with attribution, not with extraction-on-the-agent's-terms. Concretely:

1. Publish the profile if your content licensing already permits machine consumption. Mine does for most material, so `ai.rud.is` will likely get one.
2. Put real license and citation guidance in the `trust` group. A declared citation format beats hoping scrapers invent one.
3. Treat the profile as descriptive, but not protective. If you need paywalls or rate limits, enforce those with your server, not with metadata.
4. Keep human-readable HTML primary. The spec itself asserts this as a requirement and warns against promising search-ranking gains. Content behind `llms.txt` alone is content Google may rank down and readers cannot find.

The harsh reality is that the scanners will keep arriving whether you publish this new metadata file or not. The choice is between declaring terms on your own page and letting each agent infer them from whatever it happens to fetch first.

### FIN

An unknown user agent ran a fixed discovery checklist against this blog from GitHub Actions runners. It traced to ARWP, an experimental standard for publisher-declared agent surfaces. The profile costs minutes to adopt, controls nothing, reveals nothing hidden, and removes ambiguity about how agents should read a site. That trade is fine by me. Read the spec at the [project repository](https://github.com/dkharlanau/agent-ready-web-profile) so you can decide if it works for you as well.


```json
[
  {
    "@context": "https://schema.org",
    "@type": "BlogPosting",
    "@id": "https://ai.rud.is/posts/2026-08-26-a-scanner-named-arwp-tracing-a-curious-user-agent-back-to-its-source",
    "mainEntityOfPage": {
      "@type": "WebPage",
      "@id": "https://ai.rud.is/posts/2026-08-26-a-scanner-named-arwp-tracing-a-curious-user-agent-back-to-its-source"
    },
    "headline": "A Scanner Named ARWP: Tracing a Curious User Agent Back to Its Source",
    "datePublished": "2026-08-26T15:30:00Z",
    "description": "I traced an unknown user agent (arwp-scanner/0.1) in my Caddy logs from a GitHub Actions runner back to Agent-Ready Web Profile, an open standard for publishers to declare, in /ai/site-profile.json, how agents may read a site — and I explain why I'll publish one.",
    "url": "https://ai.rud.is/posts/2026-08-26-a-scanner-named-arwp-tracing-a-curious-user-agent-back-to-its-source",
    "keywords": [
      "Security",
      "AI",
      "Web",
      "User Agents",
      "Crawlers",
      "Site Observatory"
    ],
    "author": [
      {
        "@type": "Person",
        "name": "hrbrmstr",
        "url": "https://rud.is"
      }
    ],
    "publisher": {
      "@type": "Organization",
      "name": "hrbrmstr",
      "url": "https://ai.rud.is/",
      "sameAs": [
        "https://mastodon.social/@hrbrmstr",
        "https://bsky.app/profile/hrbrmstr.bsky.social",
        "https://github.com/hrbrmstr",
        "https://sr.ht/~hrbrmstr"
      ]
    }
  },
  {
    "@context": "https://schema.org",
    "@type": "BreadcrumbList",
    "itemListElement": [
      {
        "@type": "ListItem",
        "position": 1,
        "item": {
          "@id": "https://ai.rud.is/",
          "name": "ai.rud.is"
        }
      },
      {
        "@type": "ListItem",
        "position": 2,
        "item": {
          "@id": "https://ai.rud.is/posts/2026-08-26-a-scanner-named-arwp-tracing-a-curious-user-agent-back-to-its-source",
          "name": "A Scanner Named ARWP: Tracing a Curious User Agent Back to Its Source"
        }
      }
    ]
  }
]
```