---
title: "What 100 Cybersecurity Vendors Tell AI Agents: An llms.txt Census"
description: "I probed all 100 Cyber 100 cybersecurity vendors for /llms.txt: 34 ship one, 40 return 404, and eight WAFs block it outright — including Cisco, which seems to require TLS fingerprint matches to user-agents. A census of what the security industry actually tells AI agents, plus what a good llms.txt should carry (and what to leave out)."
pubDatetime: 2026-09-04T11:00:00Z
author: hrbrmstr
tags: ["llms-txt", "ai-agents", "seo", "cybersecurity", "web-scraping", "research"]
---
> Original: [What 100 Cybersecurity Vendors Tell AI Agents: An llms.txt Census](https://ai.rud.is/posts/2026-09-04-cyber100-llms-txt-census)

In September 2024, Jeremy Howard of [Answer.AI](https://www.answer.ai/) published a proposal called [llms.txt](https://llmstxt.org/). It defines one file: `/llms.txt` at the root of a web site, written in Markdown and for machine readers.

The file follows a fixed shape: One `# H1` carries the project name. A blockquote under it carries a one-paragraph summary. Then optional `## H2` sections hold Markdown links, each with a short description. A companion file, `/llms-full.txt`, holds the full text of the linked pages in one place.

While the spec is a proposal (i.e., not an IETF standard), adoption still matters, because agents read what sites serve. A help center that ships `/llms.txt` gives a model a curated map whereas a site without one leaves the model to crawl and guess.

The whole thing is designed to answer a real question: when an agent lands on your domain, what should it read first?

## The experiment

The University of San Diego maintains "[The Cyber 100](https://onlinedegrees.sandiego.edu/top-100-cybersecurity-companies/)," a list of 100 influential cybersecurity companies (updated 2026-07-21). The list made for a good sample — large vendors, government integrators, consultancies, and consumer security brands all represented.

So, of these 100 companies, how many ship an `llms.txt`, and what do those files actually tell an agent?

Design constraints:

- Probe exactly one URL per company: `https://<domain>/llms.txt`.
- Use a real browser User-Agent string, as some webservers/CDNs serve different bodies to `curl`.
- Save every response body, including error pages. Classify from the content, not from status codes alone.

## Steps taken

1. Downloaded the article and extracted all 100 company entries (name, article link, domain) with a DuckDB (ofc) query over the anchor tags.
2. Identified and fixed one source error: the article links Cyderes to `herjavecgroup.com` but I used the canonical `cyderes.com` domain instead.
3. Probed `https://<domain>/llms.txt` for all 100 domains with `curl`, a Chrome User-Agent, a 25-second timeout, `--location`, and `--compressed` (frustratingly, one vendor ships raw gzip regardless of request headers).
4. Re-tried domains that returned non-200 against the bare registrable domain, which recovered a few subdomain-hosted sites.
5. Recorded status code, body size, and URL tried for each probe in `results.csv`.
6. Read and classified each response body from its content, not from the status code.
7. Went back to sites with some odd, non-200 responses and tried them again with well-known AI agent UAs.

## Results⁺ at a glance

| Outcome | Count | Examples |
| --- | --- | --- |
| Real llms.txt served | 34 | cloudflare.com, crowdstrike.com |
| 404, no file | 40 | splunk.com, microsoft.com |
| 403, blocked | 9ⁱ | mitre.org, mcafee.com |
| HTML page served with 200 | 6 | cyberark.com, thycotic.com |
| Redirect, no file at target | 5 | trellix.com, pwc.com |
| 202 accepted, empty body | 3 | checkpoint.com |
| Host unreachable | 2 | siemplify.co |
| Real file behind broken format | 1 | esentire.com |

_⁺This table reflects the run results with the original user-agent._  \
_ⁱCisco was originally in this category, but was ultimately resolved._

One-third of the Cyber 100 ships an `llms.txt`! The other two-thirds return errors, block pages, or HTML dressed up in an HTTP 200 response. Eight vendors, including Intel, McAfee, and MITRE, serve a 403 to the probe (their WAFs seem to think `llms.txt` requests are violence of some sort).

The blocked set seemed a bit suspicious, which called for some follow-up. The probe repeated all eight fetches with `GPTBot`, `ClaudeBot`, `ChatGPT-User`, and `Claude-User` User-Agent strings, then again with a plain `curl/8.9.1` identity. The outcomes split four ways:

- **cisco.com serves a real llms.txt, and hides it.** The file exists at `https://www.cisco.com/llms.txt` and returns a 403 to the browser User-Agent and to every bot identity. It returns a 200 to plain `curl`. The edge rejects clients whose TLS fingerprint doesn't match their claimed User-Agent, so honest browser-like agents get locked out of a file written for them.
- **intel.com has no file.** The plain-`curl` probe got a true 404.
- **baesystems.com serves an Imperva challenge page.** The page carries status 200, a `NOINDEX` meta tag, and a JavaScript challenge. No file exists behind it.
- **mcafee.com and whiteops.com drop the connection** for every tested identity — browser, bot, and plain `curl` alike. Akamai, Digital Defense, MITRE, and Tripwire answered 403 to all of them.

File sizes ranged from 1,092 bytes (sera-brynn.com) to 474,305 bytes (fidelissecurity.com).

## The classes

Seven patterns showed up across the 35 files:

### Directive files

The most intentional files in the sample treat the agent as a reader with specific instructions — which is, after all, what the spec was designed for. [Ping Identity](https://www.pingidentity.com/) ships the strongest example: every resource link carries usage rules in the description. Customer stories links read "Use for commercial validation. Do not use as implementation guidance." Newsroom links read "Do not use older announcements as the source of truth for current product details."

[Delinea](https://delinea.com/) ships an `## Instructions for AI Agents` section that tells the agent to prefer Markdown resources over HTML when both exist, then links the Markdown versions directly. [Varonis](https://www.varonis.com/) ships a hybrid — a robots-style allow/disallow block followed by rules for how an agent should represent Varonis content. 

### Curated directories

Fourteen vendors ship the spec shape done properly (i.e., `# H1`, blockquote summary, `##` sections, and described links): Cloudflare, CrowdStrike, Palo Alto Networks, Tenable, Gigamon, Darktrace, Leidos, Lookout, NETSCOUT, RSA, Intruder, ZeroFox, Sera-Brynn, and Cisco.

Quality varies widely within this class. Cloudflare maps its platform areas, developer docs, and company pages in 17 KB — compact and legible. CrowdStrike dates its file (`Last Updated: 2026-04-30`), a practice worth copying because stale guidance is worse than no guidance. Cisco's file covers five Data Center priority pages and nothing else. Sera-Brynn lists eight service links in a mere 1 KB. 

### Docs maps

Duo and OneSpan point the agent at documentation rather than marketing. Duo's file lists over a thousand links — nearly all Cisco Duo doc pages — each as a `.md` URL. OneSpan lists 979 links under a single `## Docs` heading. For an agent doing technical work (e.g., troubleshooting an integration or verifying an API call), these files have the highest value per byte in the sample. They also show the cost of the raw-dump approach: 264 KB and 405 KB respectively, all inside one flat file with no curation hierarchy. Ask what the MFA product supports, or what if a vendor is affected by a given CVE, and the file returns the same pile.

### SEO plugin exports

Four files came from WordPress SEO plugins, not from a human. Infoblox and Verimatrix run Yoast ("Generated by Yoast SEO v26.2"). Transmit Security runs Rank Math. Valimail runs All in One SEO, and its export lists 970 links including every tag archive page on the site. A plugin export tells the agent "here is every URL" and stops there, leaving tag pages weighted equally with product pages. These files definitely carry generator signatures and match common plugin defaults.

### Full-site dumps

Booz Allen's file opens with "Full Site Content for Booz Allen" and ships the _entire_ site as Markdown. Fidelis Security ships 736 links across every CMS content type, and ships each section twice — a generator ran two passes and concatenated the output, producing 474 KB where roughly half is a duplicate of the other half (there goes your context window!).

### Marketing copy

Avast, Kaspersky, and Arctic Wolf serve product pitches. Avast's file leads with "Download Free Antivirus & VPN | 100% Free & Easy." Trend Micro's file routes the agent into a "What Is?" glossary farm — a large SEO content library that generates thin product signal at high volume. I have to wonder if these are designed for Google's (and others') "AI Overview" sections, but they may be of some use for an analyst using AI tooling to get market overviews.

### Malformed and stray

A few files don't fit cleanly into any other class. Checkmarx ships a real product directory in a broken format — `[text] URL:` instead of `[text](url)`, with — oddly — CR-only line endings — so a strict Markdown parser reads the whole thing as one line. eSentire prepends an `<h1>` element and a robots block before the actual `llms.txt` content (`O_o`). KPMG's file is a scrape of the global home page, login and preferences links included, and tells the agent almost nothing useful. Trustwave's file describes LevelBlue, its rebrand, and points at `levelblue.com` — correct, but the file at `trustwave.com` never mentions the word Trustwave. AT&T's file positions the whole corporation, not its security unit (this is likely an issue with the OG list curation, not AT&T).

## Configurations that need work

These are the files where the intent clearly existed but the execution failed quite miserably. Vendors named here can verify each item against their own deployment.

**a10networks.com leaks a staging host.** Its `llms.txt` points at `env-a10networkscorporate-premstaging.kinsta.cloud` — a Kinsta WordPress pre-production environment — for its home page, product pages, and resources. An agent following the file crawls the _password-protected_ staging site. Anyone reading the file learns the staging host name. This is the single worst finding in the census and kind of a rookie CMS configuration mistake tbh.

**fidelissecurity.com ships every section twice.** Their generator likely ran two passes and concatenated the results to serve up 474 KB to your context window.

**checkpoint.com, oneidentity.com, and onelogin.com return 202 with an empty body.** A 202 means "accepted for processing." I'm going to try to dig into this a bit with a more robust crawler setup.

**Six vendors serve a full HTML page with status 200**: CyberArk, Guidance Software, RiskIQ, Thycotic, Zimperium, and i-Sprint, so any agent that trusts the status code ingests a soft 404 as if it were content.

**valimail.com lists every tag archive page.** 970 links of index noise that dilute the links that matter.

**trendmicro.com routes agents to the glossary.** An agent that looks for product capabilities finds "What Is AI?" (I kid you not) explainers instead.

**sailpoint.com returns a 9-byte body** — the string `Not found` — with a 404 after a redirect chain that promises a file and then doesn't deliver one.

## What cybersecurity vendors should put in llms.txt

The census suggests a short list. A security vendor's `llms.txt` should carry, in order:

1. **A dated summary.** `Last Updated:` in the blockquote. Stale guidance misleads agents more than no guidance, and the date is free to include.
2. **A product/services map with one link per product.** Descriptions in one sentence. The agent needs the catalogue, not the campaign.
3. **Documentation in Markdown.** Link the `.md` versions the way Duo does. Use flags, like Delinea's, when both HTML and Markdown exist.
4. **Security info and current disclosures.** CVE list, advisories, and the PSIRT contact. An agent asked "is vendor X affected by CVE-2026-NNNN?" can then answer from the vendor's own record rather than third-party coverage.
5. **Trust artifacts.** SOC 2 status, penetration test summaries, compliance attestations, etc. This is the section, say, that procurement agent might look for.
6. **Status page and changelog.** Makes it easier for agentic systems to know service status and also if they should care about new info.
7. **Usage rules where they help.** Follow Ping Identity's pattern: mark customer stories as marketing, mark docs as canonical.

and, four things to leave out:

- Staging URLs.
- Plugin exports of every tag page, since an index isn't a briefing.
- Consumer download pitches in a file aimed at analysts and enterprise procurement agents.
- Duplicate content (be kind to context windows!).

The spec is one file — one page of Markdown — and the ~17 vendors who ship it well spend less effort than the eight whose WAFs block it.

---

Data & scripts available at: <https://rud.is/git/gists.git/tree/2026/cybersec-llmstxt?h=main>


```json
[
  {
    "@context": "https://schema.org",
    "@type": "BlogPosting",
    "@id": "https://ai.rud.is/posts/2026-09-04-cyber100-llms-txt-census",
    "mainEntityOfPage": {
      "@type": "WebPage",
      "@id": "https://ai.rud.is/posts/2026-09-04-cyber100-llms-txt-census"
    },
    "headline": "What 100 Cybersecurity Vendors Tell AI Agents: An llms.txt Census",
    "datePublished": "2026-09-04T11:00:00Z",
    "description": "I probed all 100 Cyber 100 cybersecurity vendors for /llms.txt: 34 ship one, 40 return 404, and eight WAFs block it outright — including Cisco, which seems to require TLS fingerprint matches to user-agents. A census of what the security industry actually tells AI agents, plus what a good llms.txt should carry (and what to leave out).",
    "url": "https://ai.rud.is/posts/2026-09-04-cyber100-llms-txt-census",
    "keywords": [
      "llms-txt",
      "ai-agents",
      "seo",
      "cybersecurity",
      "web-scraping",
      "research"
    ],
    "author": [
      {
        "@type": "Person",
        "name": "hrbrmstr",
        "url": "https://rud.is"
      }
    ],
    "publisher": {
      "@type": "Organization",
      "name": "hrbrmstr",
      "url": "https://ai.rud.is/",
      "sameAs": [
        "https://mastodon.social/@hrbrmstr",
        "https://bsky.app/profile/hrbrmstr.bsky.social",
        "https://github.com/hrbrmstr",
        "https://sr.ht/~hrbrmstr"
      ]
    }
  },
  {
    "@context": "https://schema.org",
    "@type": "BreadcrumbList",
    "itemListElement": [
      {
        "@type": "ListItem",
        "position": 1,
        "item": {
          "@id": "https://ai.rud.is/",
          "name": "ai.rud.is"
        }
      },
      {
        "@type": "ListItem",
        "position": 2,
        "item": {
          "@id": "https://ai.rud.is/posts/2026-09-04-cyber100-llms-txt-census",
          "name": "What 100 Cybersecurity Vendors Tell AI Agents: An llms.txt Census"
        }
      }
    ]
  }
]
```