About

A bandwidth ledger for small sites

Small content sites (model libraries, wikis, photo archives, hobby forums) often find out about aggressive crawlers only when the hosting bill arrives. One public report described a single crawler pulling well over 100 GB of 3D models from a small community site. Crawl Ledger turns the access log you already have into a ranked list of who took what, and writes the rules to stop the ones you never agreed to feed.

Three ways to use it

In the browser. Open the analyser and drop in access.log (plain or .gz). Parsing happens in the tab, so files of a few hundred MB are fine and nothing is uploaded.
Save and share. When the audit looks right, save it to get a private link like /r/Ab3dEf7hJk for your host, your team or a forum thread. The browser that saved it keeps an edit token, so it can update the chosen rules or delete the report later.
From the server, with curl. Send a log straight from the machine that wrote it, as shown below.

API

Send a log and read a summary in the terminal:

curl --data-binary @/var/log/nginx/access.log \
  "https://crawl-ledger.boogles.dev/api/analyze?format=text"

Get JSON plus a shareable report (gzip is detected automatically):

gzip -c access.log | curl --data-binary @- \
  "https://crawl-ledger.boogles.dev/api/analyze?save=1&title=October&policy=no-ai"

Route	What it does
`POST /api/analyze`	Body: raw log, up to 32 MB (gzip or plain, at most 150,000 lines). Query: `policy` = `balanced` \| `no-ai` \| `bandwidth`, `rate` (1 to 600 requests/min), `save=1`, `title`, `format` = `json` \| `text`. Returns totals, ranked clients with suggested actions and reasons, estimated savings, and robots.txt, nginx, Apache, Caddy and Cloudflare rules.
`POST /api/reports`	Saves a snapshot built by the browser. Returns `{ id, url, editToken }`.
`GET /api/reports/:id`	The saved report as JSON.
`PUT /api/reports/:id`	With `Authorization: Bearer <editToken>`: change `title`, `policy`, `rateLimit` or per-client `overrides`.
`DELETE /api/reports/:id`	With the edit token: deletes the report.
`GET /r/:id`	The dashboard for a saved report.

For bigger logs, send the most recent part (tail -n 100000 access.log) or use the browser, which has no size limit.

What a saved report contains

Only aggregates: per-client request and byte totals, daily totals, up to 25 top paths with query strings removed, the network ranges (/24 or /48) of bots so the IP rules still work, a histogram of requests per minute for the rate-limit maths, up to four user-agent strings per client, and your policy choices and robots.txt. It never contains raw log lines, individual IP addresses, or the networks of human visitors. Logs sent to /api/analyze are processed in memory and discarded. Reports expire after 180 days.

How clients are classified

User agents are matched against about 80 known crawlers, AI agents, SEO tools, link-preview fetchers and HTTP libraries. Any other self-declared bot is listed by name.
Googlebot, bingbot and GPTBot claims are checked against a built-in snapshot of their published IP ranges. Requests from outside those ranges are listed separately as unverified.
Browser traffic is grouped per network. A network is flagged as a disguised scraper when it pulls far more than typical visitors, skips the CSS, scripts and images a browser loads, or requests pages faster than people click.
If you paste your robots.txt, every bot's requests are checked against it using longest-match rules.

Limits and what is approximate

The IP-range check uses a static snapshot, not live reverse-DNS lookups.
Savings assume the rules applied for the whole period in the log. Rate-limit savings replay each client's per-minute request counts.
Generated configs are a starting point. Test them on staging first, because a typo in a server config can take a site offline.