Last updated: 2026-10-22T00:13:05Z | Log entries analyzed: 512080 | Model: glm-5.3-flash:cloud | Enrichment: Censys

Site Observatory: ai.rud.is — 2026-10-21

What Changed

This is an incremental run for 2026-10-21. The day was loud: 7,890 requests against a trailing 7-day average of roughly 3,900, with 3,838 unique IPs.

The biggest shift is fediverse traffic at 3,862 requests, 3.08× the prior 7-day average. The surge has no matching referrer signal — referrer data for the day shows only a handful of Mastodon clients (phanpy.social, infosec.exchange) — so the surge is unattributable from referrer data. Link-preview fetchers commonly omit Referer, and the hourly data shows the spike concentrated between 10:00 and 17:00 UTC, which is consistent with link sharing during the European and US workday rather than a crawler campaign.

Scanner volume also rose: 1,374 requests, 1.85× baseline. AI crawlers doubled to 715 requests (1.94×). other_crawler fell to 0.65× baseline, and RSS readers held flat at 0.97×.

New scanner IPs worth naming. Three Google Cloud IPs — 35.203.26.153, 35.240.134.159, and 34.141.93.225 — arrived with an identical user agent (Chrome/153 ... Edg/153.0.0.0 on a fake Windows build), roughly 285 requests each, and about 108 404s each. Identical UA, identical volume, three IPs: one distributed campaign. 23.234.96.222 ran 95 requests with 43 404s under a Chrome/106 Mac UA. 64.89.160.91 sent 42 requests with no user agent at all. Enrichment was unavailable for all of these.

New recon URIs are the usual PHP sprawl — /symfony/public/_profiler/phpinfo, /00_server_info.php, /_poopinfo.php (someone’s sense of humor) — plus .env variants with a trailing space (/laravel/.env%20, /app/.env%20), a space-suffix trick that defeats naive exact-match blocks. One curiosity: /posts/2026-10-19-weekly-bulletproof-report/sitemap.xml, a scanner guessing that sitemaps live under post paths.

New user agents, each with its class:

• Claude-User (claude-code/2.1.306 ...) and claude-code/2.1.305 — ai_crawler. Known Anthropic agents; two point releases of claude-code appeared the same day.
• Mozilla/5.0 (compatible; TasteModel-research/1.0; non-commercial research crawler) — other_crawler. Unknown, 11 requests from 1 IP. May need a classifier rule if it persists.
• dwm-gapfill/0.1 (research crawler; robots.txt obeyed) — other_crawler. Unknown, 2 requests. Same note.
• DittaBot/0.1 (+https://lappy.tail0170e0.ts.net:8443/bot) — other_crawler. Unknown, 1 request; the Tailscale hostname in the contact URL suggests a hobbyist.
• HeadlessChrome/155.0.8059.79 — scanner, 2 requests. Recon URIs only, so the scanner classification holds.
• Python/3.14 aiohttp/3.14.5 — scanner, 2 requests.

Traffic Summary

The observation window runs from 2026-03-10 09:10:30 UTC to 2026-10-22 00:09:24 UTC: 512,080 requests from 64,612 unique IPs, 11.3 GB served. Median response time is 4.52 ms; p95 is 31.9 ms. Caddy is not breaking a sweat.

The signal-to-noise picture is unusual for a personal blog. Fediverse link previews dominate at 45.3% of all requests (231,756 from 10,672 IPs). Human traffic — visitor at 15.6% plus RSS readers at 5.8% — is about 21.4%. Scanners are 15.2% (77,783 requests from 2,556 IPs), so noise roughly matches humans request-for-request. AI crawlers add 8.8%.

Resource consumption by class:

Bandwidth figures reflect bytes actually transferred. Conditional and cached responses show 0 bytes, so totals undercount logical content size. The scanner class is the clearest example: 77,783 requests but only 217 MB, because most probes 404 or hit canary stubs.

Temporal Patterns

Scanners and humans keep different hours. Scanner requests peak at hour 0 (5,829) and hour 3 (5,189) UTC, with a secondary bump at hour 12 (7,255). Visitors peak at hour 17 (4,090) and stay in the 3,000–4,000 band from hour 4 through hour 23 — a broad human daytime plateau. AI crawlers are flat around 1,500–2,500 per hour, all day, every day; they do not sleep.

Day of week tells the same story. Fediverse traffic peaks Monday (52,748 requests) and drops on weekends. Scanner traffic peaks Saturday (21,202 requests) — automated campaigns run when nobody is watching. Visitors prefer Monday (14,228) over Sunday (9,950).

Content & Visitors

The top human pages: / (4,671 hits, 2,875 visitors), the Ornith + Claude Code post from 2026-06-27 (2,579 hits), and the weekly bulletproof reports. The observatory page itself pulls 1,254 hits. Note the trailing-slash duplicates: /posts/2026-06-27-running-ornith-locally-with-opencode-and-claude-code/ and the same path without the slash both rank, with the slashless variant averaging 388 ms versus 15.9 ms. Caddy’s redirect costs a round trip.

Referrers over the full window: Google leads (2,908), then Reddit (622), Kagi (408), Twitter (406), and Hacker News (395). In the trailing 7 days, Kagi is the steady daily referrer — 79 on 2026-10-17, 55 on 2026-10-18, 51 on 2026-10-20, 49 on 2026-10-21 — including a few from kagi.com/smallweb/feed. LinkedIn spiked on 2026-10-16 (13 desktop + 3 Android app referrals) and then vanished. The 2026-10-21 visitor surge has no referrer behind it; Reddit, HN, and Twitter strip or omit Referer on outbound links, so a social spike can be invisible here.

HTTP/3 among real visitors: 5,569 of 79,900 visitor requests (about 7%) negotiated h3, against 4,849 h2 and 2,582 h1.1. Site-wide, h3 is 1.3% of negotiated TLS. Real browsers adopt h3 faster than the crawler population, which still lives on HTTP/1.1. Browser families among visitors: Chrome dominates (57,556 requests, 29,977 IPs), then Safari (9,734), Firefox (6,273), Edge (2,176), Opera (160).

RSS: 26,527 feed requests from 177 unique IPs in the rss_reader class, plus 6,669 feed fetches classified as visitor. Daily volume sits around 440–480 requests — a stable subscriber base, not thin, not growing fast.

Agent-Artifact Access Patterns

The post .md files carry nearly all agent-artifact traffic: 8,328 requests from 3,962 IPs, 37.6 MB transferred. llms.txt follows at 345 requests from 243 IPs (2.5 MB). The rest is a long tail: identity.json (16 requests), llm.txt (4), ai.json (6), llms.html (2), and the llms-variant group (7 requests, 0 bytes — all conditional hits).

Who consumes them? The top consumer list is a mix: lan-rss-collector/1.0 (590 requests, 32 artifacts), Barkrowler (542), Baiduspider (292), and — notably — plain Chrome visitors pulling .md files directly (281 requests from a macOS Chrome/133 UA). AI crawlers are present and real: PetalBot (280), Bytespider (238), Amazonbot (177), GPTBot (130), Applebot (112), ChatGPT-User (82).

Per-post .md breakdown confirms the split. The Ornith post’s .md file: 316 requests, 148 from AI crawlers, 74 from visitors, 38 from search crawlers. observatory.md: 293 requests, 170 visitor versus 34 AI — humans read this one in Markdown, which tracks for a technical audience.

The honest comparison: HTML page views still outnumber artifact fetches roughly ten to one, and the designed-for-agents index files (llms.txt aside) get almost no use. But the .md posts are doing their job — AI crawlers request them at rates that match or exceed their HTML counterparts for the popular posts. The agents the artifacts were built for use the artifacts; the exotic index formats mostly wait.

AI Crawler Activity

The aggregate table is long, so here is the shape of it. The top named crawlers by request volume:

• Other AI (unclassified agents): 5,256 requests, 680 IPs
• ClaudeBot (Anthropic): 4,553 requests, 193 IPs
• ChatGPT-User (OpenAI): 4,381 requests, 1,915 IPs — user-triggered fetches, hence the IP sprawl
• Bytespider (ByteDance): 3,459 requests, 1,098 IPs
• Amazonbot: 3,092 requests, 453 IPs
• Meta Crawler: 2,972 requests, 992 IPs
• PetalBot (Huawei): 2,199 requests from only 22 IPs — the most concentrated crawler on the list
• Claude User: 2,192 requests, 1,099 IPs
• OAI-SearchBot: 2,142 requests, 253 IPs
• ExaSearchBot: 2,080 requests, 458 IPs
• PerplexityBot: 1,965 requests from 46 IPs — also concentrated
• GPTBot: 1,735 requests, 94 IPs
• Applebot: 1,377 requests, 875 IPs

Below that, a long tail: GoogleOther, ShapBot, CCBot, DuckAssistBot, and dozens of Chinese and regional agents (DeepSeekBot, Qwen Bot, KimiBot, Hunyuan, MoonshotBot, ChatGLM Spider). One IP (AteveSearchSourceUrlDiscovery) made 329 requests for 329 unique pages — a full crawl from a single host.

For a small personal blog, 44,879 AI-crawler requests over seven months is real bandwidth (776 MB) but not a burden. The interesting number is IP diversity: ChatGPT-User and Applebot spread across hundreds of IPs, while PetalBot and PerplexityBot run tight fleets. If you ever write IP-based rate limits, the concentrated crawlers are the ones you can actually throttle.

Search Engine Crawlers

All the majors show up. Bingbot: 3,587 requests, 402 IPs, 466 pages. Googlebot: 3,067 requests, 412 IPs, 525 pages. Baiduspider: 2,658 requests, 424 IPs, 544 pages — Baidu indexes this site more thoroughly than most Western engines bother to. DuckDuckBot: 1,523 requests, 165 pages. Yandex: 808 requests but last seen 2026-10-13, so it has gone quiet for over a week. The site is well covered by Google, Bing, and Baidu; DuckDuckGo crawls lightly; Yandex paused.

Fediverse Activity

Fediverse link-preview traffic is the site’s largest class, and 2026-10-21 tripled it. The top 20 instances by request volume include mastodon.n41.lat (185 requests), evy.pet and social.louis-vallat.dev (Misskey, 179 and 173), federation.network (159), and veganism.social (150). Multiple distinct Hetzner-adjacent IPs are different instances, not one recurring reader. Each instance fetches / when someone shares a URL; the volume means the site gets shared on the fediverse often enough that link cards are a first-class traffic source.

Scanner & Recon Activity

The all-time scanner leaderboard is headed by 185.219.151.78: 9,636 requests, 8,632 404s, 7,816 unique URIs between June and September. That is a dictionary attack on the URL space, not a targeted probe. The 185.177.72.x trio (185.177.72.17, .69, .38) runs curl/8.7.1 in short bursts — 2,999 requests in six minutes on 2026-10-16, 2,722 in five minutes on 2026-08-28 — nearly all 404s. Fast, loud, and uninterested in stealth.

Recon targets are the standard 2026 menu: /info.php (136 hits from 78 IPs), /.aws/credentials (118 hits from 86 IPs), /config.json, /credentials.json, WordPress batch endpoints (/wp-json/batch/v1 and its /blog/ and /wordpress/ prefixes), and /phpinfo.php. One IP ran 102 requests for /nuclei.svg?VaOD6=x — a Nuclei template marker, so that was a Nucleo scan run, plainly labeled.

The 2026-10-21 campaign from the three Google Cloud IPs repeats the same fake Edge UA across 35.203.26.153, 35.240.134.159, and 34.141.93.225. Enrichment was unavailable for them, so attribution stops at “Google Cloud, coordinated.”

One IP with enrichment context: 172.94.9.253 appears in both the scanner table (698 requests, 639 404s, only 36 unique URIs, March through May) and the enrichment block. Censys scores it ? on reputation, places it on ASN213790 (Limited Network LTD, Netherlands), and lists SSH and HTTP with a REMOTE_ACCESS label — a rented box, not a residential connection. GreyNoise data was absent.

Also enriched: 220.196.160.218 (China Unicom Shanghai FuTe IDC, reputation ?) shows 26 requests across 26 days in the persistence table — an IDC host, so treat it as automation until proven otherwise. 2001:4860:7:219::ff is Google with privacy:anonymous,relay flags — proxy traffic, not a direct client. 207.241.239.62 and 207.241.239.99 are Internet Archive crawl nodes (wwwb-spn12.us.archive.org, wwwb-spn22.us.archive.org), benign by rDNS. 158.69.55.82 is an OVH box exposing Kubernetes ports (6443, 10250, 31452) plus SSH — someone’s misconfigured cluster, and it hit this site.

IP Persistence

70.8.208.88 is the standout: 110 days seen, 1,294 requests, first on 2026-06-26, last on 2026-10-21. That is a fixture. 45.79.135.33 (52 days) and 37.59.21.100 (48 days) are likely self-hosted services on Linode and OVH ranges given their ASNs’ reputations — though enrichment was unavailable for them, so that is inference from the IP ranges alone, and I flag it as such. 92.247.181.45 shows 46 days over six months. A cluster of China Unicom and Tencent IPs (220.196.160.x, 43.155.125.x, 116.128.185.x) persists for 20–33 days each with low request counts — consistent with monitoring or scraping infrastructure, not readers.

This is IP persistence, not repeat visitors. IPv6 privacy extensions and CGNAT both blur the mapping in both directions.

Serendipity

Two things. First, /_poopinfo.php in the new recon URIs — a phpinfo-guessing wordlist with a sense of humor, presumably from an open-source scanner repo. Second, the .env%20 trailing-space probes: the scanner appends %20 to defeat exact-match deny rules. If you block /.env by exact string, /development/.env%20 walks past it. The canary stubs caught these with 200s either way, which is the point of the canaries.

Observations

• The canaries earn their keep. /.env returned 200 to 625 requests from 317 IPs, last hit 2026-10-21 17:33:21 UTC, and the whole .env family (/.env.local, /.env.production, /.env.bak, and ten more variants) plus /.git/config (352 hits) all show fresh activity on the report day. Every one of those 200s is a logged, fingerprintable scanner. The decoys turn noise into signal at zero cost.
• The AI.ROBOTS.TXT cross-check has both kinds of rows. Gaps: wpbot/1.4 (18 requests, classed other_crawler) names a listed agent and never gets called an AI crawler — a candidate classifier rule. Category disagreements, not gaps: Scrapy and Lightpanda landed in scanner because their requests hit credential probes, where URI rules outrank UA rules by design; Sogou and Exabot sit in search_crawler while the upstream list calls them AI data scrapers — an open question, not an error. The Baiduspider rows list Spider as the matched agent, a substring artifact of the check, and I do not count them. One cluster worth noting: on 2026-08-29, single IPs ran Claude-SearchBot, OAI-SearchBot, GPTBot, and GrokBot UAs into scanner — same timestamp window, same IP pattern, one impersonation batch.
• Fediverse is this blog’s real distribution channel, and the classifier proves it: 45.3% of requests, 58.7% of bytes. A personal site that optimizes only for Google is optimizing for its third-largest referrer while its largest one — Mastodon instances fetching link cards — gets no design attention at all. The artifacts section shows the .md posts serving agents well; the fediverse serves humans. Both matter more here than the scanner panic of the week.

Generated by observatory.sh — Caddy logs → DuckDB → Censys → Ollama → Astro