# GroundShade documentation

> Self-hosted Rust reverse proxy for DDoS and scraper defense. Every doc page concatenated into one file for LLM use. Source: https://codeberg.org/groundshade/groundshade

---

# Getting started

> Drop GroundShade in behind Caddy, nginx, or HAProxy. Five minutes to a working request path.

<aside class="note" aria-label="Beta software notice">
  <p class="note-head">Beta · pre-1.0</p>
  <p>
    GroundShade is a young project. It runs in production on
    [kycnot.me](https://kycnot.me) and has CI gates for attack
    survival, but it's still sub-1.0. Expect occasional bugs and small
    config-schema changes between minor versions. If you hit something
    rough, open an issue on
    [Codeberg](https://codeberg.org/groundshade/groundshade/issues).
  </p>
</aside>

GroundShade sits between your TLS terminator and your app. It does
not terminate TLS itself; your fronting proxy keeps doing that and
forwards the JA4 fingerprint as `X-JA4`.

```
client → TLS terminator (Caddy / nginx / HAProxy) → groundshade → your origin
```

## Run with Docker Compose

```yaml
services:
  groundshade:
    image: codeberg.org/groundshade/groundshade:latest
    restart: unless-stopped
    environment:
      GROUNDSHADE_UPSTREAM_URL: http://your-app:3000
      GROUNDSHADE_ADMIN_TOKEN: ${GROUNDSHADE_ADMIN_TOKEN:?set an admin token}
      GROUNDSHADE_TRUSTED_PROXIES: 172.18.0.2/32
    ports:
      - "127.0.0.1:9090:9090"
    volumes:
      - groundshade-state:/var/lib/groundshade

volumes:
  groundshade-state:
```

Generate the admin token once:

```sh
openssl rand -hex 32
```

Put it in your `.env`. The token gates `/admin/*` and `/metrics`.
Startup refuses to bind admin on a non-loopback address without a
token.

Then point your fronting proxy at `groundshade:8080` instead of your
app. Note that, if you are using docker, groundshade and your app need to share the same docker network.

## Check the dashboard

The container's admin port is published to host loopback on
`127.0.0.1:9090`. Open it in a browser:

```
http://127.0.0.1:9090/
```

The login page accepts your admin token and sets a cookie. After login
you land on the dashboard.

If you're not on the host, tunnel over SSH:

```sh
ssh -L 9090:127.0.0.1:9090 your-host
```

## Forward JA4 (optional, recommended)

The JA4 arm of the rate signal needs your fronting proxy to forward
the TLS fingerprint as `X-JA4`. Without it you keep the per-IP/24 arm
and trustless persistence; you lose detection of botnets that share a
TLS library across many IPs.

Stock Caddy has no JA4 placeholder. The build recipe lives at
[`examples/deploy/caddy-ja4.Dockerfile`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/deploy/caddy-ja4.Dockerfile)
and the matching Caddyfile at
[`examples/deploy/caddy-ja4.Caddyfile`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/deploy/caddy-ja4.Caddyfile).
For nginx and HAProxy see
[`nginx-ja4.conf`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/deploy/nginx-ja4.conf)
and
[`haproxy-ja4.cfg`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/deploy/haproxy-ja4.cfg).

GroundShade auto-detects JA4. After 100 requests (or 60 seconds), if
no JA4 has arrived, it logs a single WARN and the dashboard's signals
row reads `JA4: not detected`. Everything else keeps working.

## Trusted proxies

`GROUNDSHADE_TRUSTED_PROXIES` should list only your fronting proxy's
pinned IP. Trusted peers get `X-Forwarded-For` and `X-Real-IP` honored
and bypass the per-IP connection cap. Pin tight: a `/16` would grant
cap bypass to every container on the bridge.

Defaults trust only loopback (`127.0.0.1/32`, `::1/128`).

## Where to go next

- [Operating GroundShade](/docs/operating) covers daily workflows,
  the dashboard, the metrics reference, and the tuning table.
- [Architecture](/docs/architecture) explains the request lifecycle
  and the three defense levels.
- [Deployment hardening](/docs/hardening) is the pre-deploy security
  checklist. Read it before publishing.
- [Behind Cloudflare](/docs/cloudflare) covers the orange-cloud
  recipe and what gets silenced.
- [Building from source](/docs/build) builds local Docker images and
  multi-arch releases.
- [FAQ](/docs/faq) lists the questions operators ask first.

---

# Operating GroundShade

> Daily operator guide. Configuration, defense levels, fast-lane allowlists, the admin dashboard, the metrics reference, and the tuning table.

This page covers running GroundShade in production. The wire formats
and full design live in
[SPEC.md](https://codeberg.org/groundshade/groundshade/src/branch/main/SPEC.md).

## TL;DR

```sh
# Plain run with built-in defaults. Forwards to http://127.0.0.1:3000.
groundshade

# With a config file:
groundshade --config /etc/groundshade/config.yaml
```

The proxy listens on `0.0.0.0:8080`. The admin port and dashboard
live on `127.0.0.1:9090`.

## Configuration

YAML, loaded in priority order:

1. `--config <path>` flag
2. `GROUNDSHADE_CONFIG` env var
3. `/etc/groundshade/config.yaml`
4. `./groundshade.yaml`
5. Built-in defaults (zero-config)

[`examples/config/full.yaml`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/config/full.yaml)
documents every option at its default.
[`examples/config/minimal.yaml`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/config/minimal.yaml)
is the smallest useful starting point.

### Environment overrides

These env vars override the loaded YAML at startup and on every
`SIGHUP` reload:

| Variable | Purpose |
|---|---|
| `GROUNDSHADE_CONFIG` | Config file path |
| `GROUNDSHADE_LISTEN_HTTP` | Inbound HTTP listener |
| `GROUNDSHADE_LISTEN_ADMIN` | Admin listener |
| `GROUNDSHADE_ADMIN_TOKEN` | Bearer token gating `/admin/*` and `/metrics` |
| `GROUNDSHADE_UPSTREAM_URL` | Default upstream URL |
| `GROUNDSHADE_TRUSTED_PROXIES` | Comma-separated CIDRs |
| `GROUNDSHADE_TRUST_STATE_DIR` | Trust-key state directory |
| `GROUNDSHADE_TRUST_SECRET` | HMAC signing key (hex) |
| `GROUNDSHADE_LOG_FORMAT` | `json` or `text` |
| `GROUNDSHADE_LOG_IP_HASH` | `true` hashes client IPs in logs |

The Docker image sets `GROUNDSHADE_LISTEN_ADMIN=0.0.0.0:9090` and
`GROUNDSHADE_TRUST_STATE_DIR=/var/lib/groundshade`. Outside Docker,
the state directory defaults to `$XDG_STATE_HOME/groundshade` or
`$HOME/.local/state/groundshade`.

### Hot reload

`SIGHUP` re-reads the config file and reapplies environment
overrides. The listeners drain gracefully and the proxy restarts on
the same PID with the same trust signing key, so outstanding
`gs_trust` cookies stay valid. Defense state, sliding windows,
connection counts, and metric counters all reset.

A validation failure logs a warning and keeps the old config
running. Bad reloads never take the proxy down.

To catch a bad config before you reload, validate it without starting
the server:

```sh
groundshade --check-config --config /etc/groundshade/config.yaml
# exit 0 + "config OK" if valid, exit 1 + the error if not
```

It applies the same env overrides as a real start, so it checks the
effective config. Run it as a container pre-flight
(`docker exec groundshade groundshade --check-config`) before the SIGHUP.

The inbound port is unbound briefly during reload (under 100 ms in
normal drains, up to 30 s if old connections take their time). For
true zero-downtime, run two replicas behind a load balancer.

In zero-config mode (no config file resolves), SIGHUP is logged and
ignored.

## What it does

Calm state: GroundShade behaves as a normal reverse proxy. No
challenge code runs.

Defense state: when a route's origin pain crosses thresholds (p95
latency or 5xx rate over a 30 s window with at least 50 samples),
the route lifts:

| Level | Who sees a challenge (without a trust token) |
|---|---|
| L1 | UAs matching `l1_ua_patterns`; forged-browser clients (UA claims browser, JA4 disagrees); optionally write methods if `l1_suspicion_methods` is set. v0.7.1's `always_challenge_forged_browser` flag promotes the forged-browser arm to fire at every level, including Open. |
| L2 | L1 scope plus thin clients (no `Referer`, no `Accept-Language`) |
| L3 | Everyone except the fast lane |

Defaults for `l1_ua_patterns`: `headless`, `bot`, `crawl`, `spider`,
`python`, `curl`, `go-http`, `libwww`. `l1_suspicion_methods` is
empty by default; opt in only for write-only APIs that never see a
real browser POST.

Independent of level, two behavioural signals can short-circuit:

- **Rate signal (hard threshold).** The 60 s sliding window per
  `(route, IP /24)` or `(route, JA4)` crossed
  `defense.rate_signals.hard_threshold`. Default: 1,000 requests.
- **Trustless persistence.** The `/24` was challenged
  `defense.trustless_persistence.threshold` times without ever
  solving. Default: 20. Sticks until a single solve clears it.

The rate signal's soft threshold acts as an extra L1 arm on the
firing request only. No permanent state change.

### Per-route policy knobs (v0.7.1)

Two config fields let you pin route policy without operator action:

- **`defense.escalation.min_level`** (default `open`): the level the
  route is allowed to *fall to* on cooldown. Setting `l1` keeps a
  sensitive route in active defense even when the detector reports
  calm. The detector can still escalate above; it just won't step
  below. Useful for admin panels, payment endpoints, and known scrape
  targets. The difference from `shields_up` is that the detector still
  drives the upper levels; `shields_up` pins all the way at
  `shields_up`.
- **`defense.scope.always_challenge_forged_browser`** (default
  `false`): when `true`, requests classified as ForgedBrowser
  (browser-shaped UA paired with a script-tool JA4) are placed in
  challenge scope at every level, including Open. Pairs with the v0.6
  rate signal: rate catches volume; this flag catches polite low-rate
  forgers that fly under the hard threshold. No-op under CF
  orange-cloud, where every request arrives with CF's JA4 and the
  classifier returns Browser for all.

Both are per-route overrides via the `routes` list, so the same proxy
can have a relaxed default route and a strict `/admin/*` route:

```yaml
routes:
  - match:
      path: "/admin/*"
    defense:
      escalation:
        min_level: l1
      scope:
        always_challenge_forged_browser: true
```

### Fast lane

The fast lane always bypasses challenges. First match wins, in
order:

1. Client IP matches `fastlane.allow_ips` (CIDRs, not spoofable when
   `trusted_proxies` is set correctly).
2. Path matches a configured feed glob (`/feed`, `/rss`, `*.atom`,
   etc.).
3. `User-Agent` substring matches `fastlane.allow_user_agents`
   (spoofable; pair with `allow_ips` if the threat model demands).
4. `Authorization: ApiKey id:secret` matches a configured key.
5. UA claims a known crawler (Google, Bing, DuckDuckGo, Apple,
   optionally Yandex) **and** the IP passes reverse-DNS plus
   forward-DNS verification.

`fastlane.crawlers.yandex` is `false` by default. The others are on.

### No-JS passage challenge (opt-in, v0.7.2)

The default HTML interstitial needs JavaScript to solve the
proof-of-work. Clients with JS off (Tor Browser on Safer/Safest,
NoScript users, text browsers, some accessibility setups) hit the
interstitial and have no way through. The passage challenge gives them
a path that does not need JavaScript.

It is a **friction layer, not a bot detector**. It raises cost on no-JS
clients (a one-click form, a server-enforced wait, single-use tokens,
ip-prefix + JA4 binding, a shorter trust TTL) but a patient client can
still pass. The JS proof-of-work stays the stronger proof; do not treat
the passage as a replacement.

Enable it per route with `challenge.mode`:

- `js` (default): JS PoW only; no-JS clients dead-end. No change.
- `auto`: JS clients solve the PoW as before; no-JS clients get the
  passage form in the `<noscript>` branch of the same page.
- `nojs`: the passage form is the whole page, no PoW. For routes you
  know are no-JS (for example an onion address).

```yaml
routes:
  - match:
      path: "/onion/**"
    challenge:
      # A per-route challenge block REPLACES the global one wholesale, so
      # copy any pow/interstitial settings you rely on into it too.
      mode: auto
      nojs:
        delay_secs: 5
        trust_ttl_secs: 600
```

The flow: a challenged no-JS client clicks Continue (a form with an
invisible honeypot), waits out `delay_secs` while a CSS-only progress
bar fills and the page auto-reloads, then is issued a `gs_trust` cookie
and sent back to where it was (query string preserved). The wait is
enforced server-side; there is no JS to gate the button. Tuning lives
under `challenge.nojs`: `delay_secs`, `redeem_window_secs`,
`trust_ttl_secs`, `max_issued_per_prefix`, `issue_window_secs`. Because
the passage is a weaker proof, `nojs.trust_ttl_secs` must be `<=
trust.token_ttl_secs` (validated at startup).

Watch `challenges_issued_total{kind="nojs"}`,
`challenges_solved_total{kind="nojs"}`,
`challenge_failed_total{kind="nojs",reason=...}`, and
`passage_wait_total` to see passage traffic and failures.

A note for Tor: behind one exit node many clients share an IP prefix
and Tor Browser's uniform JA4, so the binding is weak there; the
single-use token and per-prefix issuance cap carry the anti-abuse
weight.

## Operator workflows

### Engage shields under attack

```sh
curl -X POST http://127.0.0.1:9090/admin/shields \
  -H "Authorization: Bearer $GROUNDSHADE_ADMIN_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"level":"up"}'
```

Add `"route": "*|/api/*"` to target one route. Disengage with
`{"level":"down"}`.

### Mint an API key for a partner

```sh
secret=$(openssl rand -hex 32)
echo "secret (give to partner): $secret"
cat <<EOF
fastlane:
  api_keys:
    - id: partner-acme
      secret_hash: "$(printf '%s' "$secret" | sha256sum | awk '{print $1}')"
      label: "Acme webhook receiver"
EOF
```

`SIGHUP` to reload. The partner sends
`Authorization: ApiKey partner-acme:<secret>` on every request.

Shortcut: `make api-key`.

### Let a non-browser client through

Two paths:

1. Issue an API key (above). Use this for known clients.
2. Have the client solve the JSON challenge. Reference solvers ship
   at
   [`examples/solvers/`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/solvers):

   ```sh
   token=$(./examples/solvers/solve.py https://your.site/api/endpoint)
   curl -H "Authorization: ChallengeSolution $token" https://your.site/api/endpoint
   ```

### Allowlist a monitor

For monitors that can't send a custom auth header (UptimeRobot free
tier, Uptime-Kuma, PageSpeed), use the fast-lane allowlists. Two
independent layers; either match bypasses:

```yaml
fastlane:
  allow_ips:
    - "63.143.42.240/28"      # UptimeRobot range 1
    - "69.162.124.224/28"     # UptimeRobot range 2
  allow_user_agents:
    - "UptimeRobot"
    - "Uptime-Kuma"
    - "PageSpeed"
```

> **Security note.** UA substring match is forgeable. Treat it as
> ergonomics, not security. Pair `allow_user_agents` with
> `allow_ips` when the threat model demands it; an attacker then
> has to defeat both.

The dashboard's fast-lane section shows hit counts per reason. If a
monitor's counter never moves, either the IP range is wrong or your
fronting proxy is stripping the real client IP before it reaches
GroundShade.

### Read the dashboard

Open `http://127.0.0.1:9090/` in a browser. With an admin token
configured, you land on `/admin/login`; submitting the token sets an
`HttpOnly` `gs_admin` cookie and redirects to the dashboard.

The page polls `/admin/status`, `/admin/routes`, and `/metrics` once
per second and renders:

- A hero showing the worst-route level and a one-line summary of
  connections, drop rate, and uptime.
- Per-route cards with a 32 s sparkline of request rate, the
  shields toggle, and per-route signal tracked-key counts.
- A traffic proportion bar (forwarded vs challenged) and three
  challenge funnels with drop/pass rates: browser (JS PoW), API
  (JSON), and the opt-in no-JS passage (issued, solved, wait, fail).
- A FAST LANE row with per-reason counters.
- A CLIENTS row classifying traffic by UA + JA4 family (browser,
  script, forged, bot, unknown).
- A SIGNALS row with JA4 state, rate soft/hard hits, trustless
  hits, and tracked-key counts.
- A CONNECTIONS row with active connections, refusals, and the
  self-throttle flag.

Point Prometheus at `http://127.0.0.1:9090/metrics` with the same
`Authorization: Bearer <admin-token>` header.

### Admin endpoints

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/admin/status` | Version + counts |
| `GET` | `/admin/routes` | Per-route defense snapshot |
| `POST` | `/admin/shields` | Engage or disengage shields |
| `GET` | `/admin/login` | Login form |
| `POST` | `/admin/login` | Submit token, get cookie |
| `POST` | `/admin/logout` | Drop the cookie |
| `GET` | `/metrics` | Prometheus exposition |

### Metrics reference

| Metric | Labels | Type | Meaning |
|---|---|---|---|
| `groundshade_requests_total` | `route`, `decision`, `level` | counter | Requests processed by decision (`forward`, `challenge_html`, `challenge_json`, `reject`) and route level |
| `groundshade_route_level` | `route` | gauge | Effective level: `0` Open, `1` L1, `2` L2, `3` L3, `4` ShieldsUp |
| `groundshade_challenges_issued_total` | `route`, `kind` | counter | Challenges minted; `kind` is `html`, `json`, or `nojs` (passage) |
| `groundshade_challenges_solved_total` | `route`, `kind` | counter | Successful solves; same `kind` set |
| `groundshade_challenge_failed_total` | `route`, `kind`, `reason` | counter | No-JS passage redemptions that failed (`reason`: `bad_signature`, `expired`, `replay`, `binding`, `bad_path`, `honeypot`, `cap`, `unauthorized`) |
| `groundshade_passage_wait_total` | `route` | counter | No-JS passage reloads served before maturity (the wait state) |
| `groundshade_tokens_issued_total` | `route` | counter | Trust tokens minted |
| `groundshade_fastlane_total` | `route`, `reason` | counter | Fast-lane hits by reason (`apikey`, `crawler`, `feed`, `ip_allowlist`, `ua_allowlist`) |
| `groundshade_connections_active` | (none) | gauge | Inbound TCP connections held |
| `groundshade_connections_rejected_total` | `reason` | counter | Connections refused at the accept layer |
| `groundshade_self_throttle` | (none) | gauge | `1` while the proxy is in self-throttle |
| `groundshade_client_family_total` | `family` | counter | Coarse classification: `browser`, `script`, `forged_browser`, `bot`, `unknown` |
| `groundshade_signals_immediate_total` | `route`, `reason` | counter | Challenged on sight by a signal (`rate_hard`, `trustless`) |
| `groundshade_signals_soft_noisy_total` | `route` | counter | Requests where the rate soft threshold fired (L1 scope arm) |
| `groundshade_signals_rate_tracked_keys` | `route`, `key` | gauge | Distinct keys tracked by the rate signal (`key` is `ip_prefix` or `ja4`) |
| `groundshade_signals_trustless_tracked_keys` | `route` | gauge | Distinct prefixes tracked by trustless persistence |
| `groundshade_ja4_detected` | (none) | gauge | `1` once at least one request has carried JA4 |

## Behavioural signals

Both signals run after the fast lane on every non-bypass request.

**Rate signal.** Per route, a 60 s sliding window keyed in parallel
by client prefix (IPv4 `/24` or IPv6 `/56`) and JA4. Crossing
`soft_threshold` (200 req/min) widens the L1 scope check by one arm
on that request. Crossing `hard_threshold` (1,000 req/min) issues a
challenge regardless of level.

**Trustless persistence.** Per route, a per-prefix counter of
challenges issued without ever solving. Past 20 (default), the
prefix is challenged on sight. One solve clears it. The "ever
earned trust" bit is sticky for the entry's lifetime.

> Note on prefixes: trust-token IP binding uses v4 `/24` and v6 `/48`
> by default. The rate and trustless signals use v4 `/24` and v6
> `/56`. The v6 numbers differ on purpose; signals use RIPE's
> recommended customer-prefix size.

### SEO safety invariant

Both signals consult **after** the fast lane. Verified crawlers,
operator allowlists, feed paths, and API keys never reach the signal
evaluator. A misconfigured threshold cannot accidentally challenge a
search-engine crawler. Tests in
[`crates/groundshade-proxy/tests/e2e_signals_seo.rs`](https://codeberg.org/groundshade/groundshade/src/branch/main/crates/groundshade-proxy/tests/e2e_signals_seo.rs)
lock the invariant.

### JA4 availability

The proxy auto-detects whether your fronting proxy is forwarding
`X-JA4`. After 100 requests (or 60 s), if no JA4 has arrived, it
logs a single WARN and the per-JA4 arm goes silent. The per-IP/24
arm and trustless persistence keep working. The dashboard's signals
row shows the current state.

## Tuning

| Symptom | Knob | Direction |
|---|---|---|
| Solves too slow on phones | `challenge.pow.leading_zero_bits` | Lower (16–17) |
| Bots find solving cheap | `challenge.pow.leading_zero_bits` | Raise (19–20) |
| False positives at L1 | `defense.scope.l1_ua_patterns` | Trim |
| Headless setups pass too easily | `challenge.probe.min_score` | Raise from `0` to `5` or `10` |
| Origin still hot under heavy traffic | `defense.trigger.p95_latency_ms` / `err5xx_rate` | Lower |
| Proxy running out of FDs | `selfdef.max_connections_total` | Raise (after kernel ulimit) |
| `connections_rejected_total{reason="per_ip"}` climbing behind a fronting proxy | `listen.trusted_proxies` | Add the proxy's pinned `/32` |
| Sensitive route (admin, payment, scrape target) should stay in defense | `defense.escalation.min_level` | Set to `l1` (or higher) on that route |
| Forged-browser traffic walks through at Open level | `defense.scope.always_challenge_forged_browser` | Set `true` per route (no-op behind CF orange-cloud) |
| Real users tripping rate signal on rich pages | `defense.rate_signals.soft_threshold` | Raise |
| Polite scrapers slipping past | `defense.rate_signals.hard_threshold` | Lower (500–800) |
| Trustless flagging real users with stale cookies | `defense.trustless_persistence.threshold` | Raise (30–50) |
| Memory budget tight | `defense.rate_signals.max_keys_per_route` | Lower (20,000) |

## Logs

JSON to stdout. Default fields: method, host, path, status,
duration, JA4, UA, and `client_ip_hash`. Client IPs are hashed with
a daily-rotated salt. To log raw IPs, set `observe.log_ip_hash:
false` and accept the GDPR responsibility.

## Persistent state

Only one file:

- `state_dir/trust.key`: 32 bytes, mode `0600`. The HMAC signing
  key for trust tokens. Persists across restarts so outstanding
  cookies survive a redeploy.

No DB, no Redis, no on-disk logs unless you redirect stdout.

To rotate the key and invalidate every outstanding cookie, delete
the file and restart. Or run shields-up with a fresh
`GROUNDSHADE_TRUST_SECRET`.

---

# Deployment hardening

> Pre-deploy security checklist. Admin port, trusted proxies, body limits, and reload safety. Read this once before publishing.

GroundShade's security guarantees depend on correct network
boundaries. Confirm these settings before publishing.

## Admin port

- Bind admin to loopback on bare metal: `listen.admin: "127.0.0.1:9090"`.
- If admin binds any non-loopback address, set `listen.admin_token`
  or `GROUNDSHADE_ADMIN_TOKEN`. Startup refuses unauthenticated
  non-loopback admin listeners.
- In Docker, publish the host port to loopback only:
  `"127.0.0.1:9090:9090"`. The container itself can listen on
  `0.0.0.0:9090` (the published image does this by default).
- Prometheus must send `Authorization: Bearer <admin-token>` when a
  token is set.

## Trusted proxies

GroundShade ignores `X-Forwarded-For` and `X-Real-IP` unless the
immediate TCP peer is in `listen.trusted_proxies`.

Defaults trust only loopback:

```yaml
listen:
  trusted_proxies:
    - "127.0.0.1/32"
    - "::1/128"
```

For Docker, add your fronting proxy's pinned container IP:

```yaml
listen:
  trusted_proxies:
    - "172.18.0.10/32"
```

or via env:

```sh
GROUNDSHADE_TRUSTED_PROXIES=172.18.0.10/32
```

**Do not set `0.0.0.0/0` or a broad public range.** Any trusted
peer can supply the client IP used for fast-lane IP allowlists,
crawler verification, behavioural signal buckets, logs, and
trust-token binding.

### Connection-cap bypass

Peers in `listen.trusted_proxies` also bypass the per-IP connection
cap (`selfdef.max_connections_per_ip`). This is intentional: a
fronting Caddy or nginx container aggregates many real clients into
one TCP peer and would otherwise saturate the cap.

The global cap (`max_connections_total`) and backpressure still
apply.

If `groundshade_connections_rejected_total{reason="per_ip"}` is
climbing behind a fronting proxy, the proxy's IP is not in
`listen.trusted_proxies`.

**Pin tightly.** Use a `/32` for a single container IP. A `/16`
for the entire docker bridge grants cap bypass to every container
on it, so a compromised sibling container could exhaust the global
cap without per-IP throttling.

**Behind Cloudflare orange-cloud.** The XFF chain has an extra hop
(CF edge → your fronting proxy). Both must appear in
`trusted_proxies`. See [behind cloudflare](/docs/cloudflare) for
the full recipe and the architectural limits you accept.

## Body limits

`selfdef.max_body_bytes` (default 10 MiB) applies to forwarded
request bodies and to GroundShade's own POST endpoints:

- `/.well-known/groundshade/solve`
- `/admin/login`
- `/admin/shields`

Oversized internal-endpoint requests return `413 Payload Too Large`.

`selfdef.max_header_bytes` (default 32 KiB) applies to all inbound
headers.

## Reloads

`SIGHUP` re-reads the config file and reapplies environment
overrides. Docker and systemd-style deployments can keep values like
`GROUNDSHADE_ADMIN_TOKEN`, `GROUNDSHADE_UPSTREAM_URL`, and
`GROUNDSHADE_TRUSTED_PROXIES` outside YAML and they survive each
reload.

A validation failure logs a warning and keeps the old config
running. SIGHUP cannot take the proxy down through bad config.

## Pre-deploy checklist

- [ ] `listen.trusted_proxies` lists every fronting peer with a
  `/32` (or its v6 equivalent).
- [ ] `listen.admin` is `127.0.0.1:9090`, or an admin token is set.
- [ ] `GROUNDSHADE_ADMIN_TOKEN` is from `openssl rand -hex 32`, not
  a sentence.
- [ ] State directory is mounted as a volume so `trust.key` survives
  restarts.
- [ ] Host port `9090` is published to host loopback only.
- [ ] Webhook URLs (if any) point at internal endpoints, not third
  parties.
- [ ] `observe.log_ip_hash: true` unless you have a written reason
  to log raw IPs.

---

# Behind Cloudflare

> Running GroundShade behind Cloudflare orange-cloud. Required trusted_proxies, what works, what gets silenced, and the Caddy recipe.

Cloudflare in front of GroundShade is a supported topology since
v0.7.0 (which fixed connection-cap saturation behind a fronting
proxy). This page covers **orange-cloud** (CF terminating TLS).
Grey-cloud is direct traffic and needs no special setup.

Orange-cloud works, but **two security signals get silenced** when
CF terminates TLS at its edge. Confirm that trade-off matches your
threat model before flipping it on.

## Topology

```
internet → CF edge (TLS) → origin server
                            ├─ fronting Caddy (HTTP)
                            │      ├─ GroundShade
                            │      │      └─ your app
                            │      └─ (or directly to your app)
```

From GroundShade's perspective:

- Immediate TCP peer: your fronting Caddy.
- `X-Forwarded-For` chain: `<real-client>, <cf-edge-ip>`. CF
  appends its edge IP.
- TLS handshake (JA4): CF's edge fingerprint. The real client's
  handshake terminates at CF.

## Required config

`listen.trusted_proxies` must list **both** your fronting Caddy
**and** Cloudflare's edge IP ranges. Without CF in the list, the
XFF parser walks right-to-left and stops at the CF edge IP,
treating it as the real client. Trust tokens, rate signals, and
trustless persistence would then key on CF's edges instead of the
actual user.

```yaml
listen:
  http: "0.0.0.0:8080"
  trusted_proxies:
    # 1. Your fronting Caddy (pinned container IP).
    - "172.18.0.10/32"

    # 2. CloudFlare IPv4 edge ranges.
    #    Source: https://www.cloudflare.com/ips-v4
    - "173.245.48.0/20"
    - "103.21.244.0/22"
    - "103.22.200.0/22"
    - "103.31.4.0/22"
    - "141.101.64.0/18"
    - "108.162.192.0/18"
    - "190.93.240.0/20"
    - "188.114.96.0/20"
    - "197.234.240.0/22"
    - "198.41.128.0/17"
    - "162.158.0.0/15"
    - "104.16.0.0/13"
    - "104.24.0.0/14"
    - "172.64.0.0/13"
    - "131.0.72.0/22"

    # 3. CloudFlare IPv6 edge ranges.
    #    Source: https://www.cloudflare.com/ips-v6
    - "2400:cb00::/32"
    - "2606:4700::/32"
    - "2803:f800::/32"
    - "2405:b500::/32"
    - "2405:8100::/32"
    - "2a06:98c0::/29"
    - "2c0f:f248::/32"
```

Verify the ranges before deploying.
[cloudflare.com/ips-v4](https://www.cloudflare.com/ips-v4) and
[cloudflare.com/ips-v6](https://www.cloudflare.com/ips-v6) are the
source of truth; the ranges shift, so re-check before deploying.

Or set via env:

```sh
GROUNDSHADE_TRUSTED_PROXIES="$(curl -s https://www.cloudflare.com/ips-v4 | tr '\n' ',' | sed 's/,$//'),172.18.0.10/32"
```

## Caddy config

Caddy needs its own `trusted_proxies` pointed at CF, otherwise it
strips CF's `CF-Connecting-IP` and `X-Forwarded-For` instead of
forwarding them:

```
{
    servers {
        trusted_proxies static cloudflare
    }
}

your-site.example.com {
    reverse_proxy groundshade:8080 {
        # Preserve XFF chain so GroundShade can walk through CF to the
        # real client. Caddy appends its own IP to XFF here; the
        # GroundShade-side trusted_proxies handles the rest.
        header_up X-Forwarded-For {>X-Forwarded-For}, {remote_host}
    }
}
```

`trusted_proxies static cloudflare` is a Caddy module that
auto-fetches and refreshes CF's IPs. If you can't use it, list the
CIDRs explicitly: `trusted_proxies static <CIDR> <CIDR> ...`.

## What works behind orange-cloud

| Feature | Status | Notes |
|---|---|---|
| Forwarding | ✓ | Just works |
| Connection-cap saturation | ✓ (v0.7+) | CF edges don't count against the per-IP cap |
| Real client IP in logs | ✓ | XFF chain resolves to the original client |
| Trust tokens | ✓ | Bound to client `/24` from XFF |
| Rate signal, per-IP arm | ✓ | Counts by real client `/24` |
| Trustless persistence | ✓ | Per real client `/24` |
| Defense ladder | ✓ | Detector sees real latency / 5xx |
| Manual shields-up | ✓ | Always works |
| Operator IP allowlists | ✓ | Resolved against real client IP |
| Operator UA allowlists | ✓ | UA is forwarded unchanged |

## What doesn't work (TLS termination limits)

| Feature | Status | Why | Workaround |
|---|---|---|---|
| JA4-keyed rate signal | ✗ silenced | CF terminates TLS; origin sees CF's JA4 | Per-IP/24 arm still works |
| Verified-crawler fast lane (rDNS) | ✗ dead | rDNS runs on the peer IP (Caddy/CF), never resolves to `googlebot.com` | Use `fastlane.allow_user_agents` |
| ForgedBrowser classifier | ✗ degraded | Uses JA4 ↔ UA consistency check; with uniform JA4, every request looks consistent | Rely on rate signal + trustless persistence + UA patterns |

The dashboard will show `JA4: detected` because CF and Caddy are
forwarding *some* JA4. It's CF's, not the real client's. The
per-JA4 counter in `groundshade_signals_rate_tracked_keys{key="ja4"}`
sits at a small number relative to per-IP keys; that's the visible
fingerprint of the limit.

## Recommended fastlane config behind CF

Because verified-crawler rDNS is dead behind CF, swap it for UA
allowlists:

```yaml
fastlane:
  crawlers:
    # rDNS verification can't see past CF; these flags do nothing here.
    google: true
    bing: true
    duckduckgo: true

  allow_user_agents:
    # Substring match, case-insensitive. Spoofable; the rate-signal
    # backstop catches mass abuse from anyone forging these.
    - "Googlebot"
    - "bingbot"
    - "DuckDuckBot"
    - "AppleBot"
    - "UptimeRobot"
    - "Uptime-Kuma"
```

If your threat model requires verified crawlers, **don't use
orange-cloud**. Grey-cloud preserves both JA4 and the rDNS fast
lane.

## Test the setup

1. **Real client IP visible.** Hit your site from a known external
   IP. Check the log:

   ```sh
   docker logs groundshade | grep -E 'client_ip|method=' | head -3
   ```

   The `client_ip` (or `client_ip_hash`) should reflect *your* IP,
   not a `104.16.x.x` / `172.64.x.x` (CF edge) or `172.18.x.x`
   (Caddy bridge) address.

2. **Per-IP rejection counter stays flat.** Under normal load:

   ```sh
   curl -s -H "Authorization: Bearer $TOKEN" \
        http://127.0.0.1:9090/metrics \
     | grep 'connections_rejected_total{reason="per_ip"'
   ```

   Should sit at `0`. If it climbs, your fronting Caddy's IP is
   not in `trusted_proxies`.

3. **JA4 state acknowledges the limit.**

   ```sh
   curl -s -H "Authorization: Bearer $TOKEN" \
        http://127.0.0.1:9090/admin/status \
     | jq .ja4
   ```

   Says `"state": "detected"` with few distinct JA4 keys tracked
   (about one per CF version). Expected.

4. **Trust tokens bind to real prefix.** Solve a challenge from
   one IP, then send a second request from a *different* IP in the
   same `/24` with the same cookie. It should pass (the token
   binds to `/24`). From a different `/24`: re-challenge expected.

## Recommendation

Use grey-cloud (CF DNS-only) when you can. You get:

- Real client JA4. Behavioural signals key on the actual TLS
  fingerprint.
- rDNS verification. Known-good crawlers don't see the wall.
- One less authority in your trust chain.

Use orange-cloud when you need CF's edge protections (WAF, edge
rate limiting, L3/L4 DDoS, image optimisation) and accept losing
JA4-keyed detection. GroundShade still provides per-IP rate,
trustless persistence, and the trust-token wall, on a reduced
signal set.

---

# Architecture

> The mental model. Workspace shape, request lifecycle, in-memory state surfaces, concurrency model, and explicit non-goals.

This page is the mental model. Byte-level wire formats and the full
design rationale live in
[SPEC.md](https://codeberg.org/groundshade/groundshade/src/branch/main/SPEC.md).

## Workspace

Three Cargo crates:

```
groundshade-core/      transport-agnostic brain (no hyper, no sockets)
  config              YAML schema, defaults, validation
  routing             host + path-glob matcher; compiles EffectivePolicy
  defense             origin-pain detector + 3-level state machine
  trust               HMAC-signed bearer tokens with binding + budget
  challenge           SHA-256 PoW, signed challenge tokens, interstitial HTML
  fastlane            verified crawlers (rDNS), API keys, feed paths
  fingerprint         JA4 parsing, UA classifier, client-family bucketing
  signals             rate signal + trustless persistence
  selfdef             ConnectionLimiter (per-IP + global caps + backpressure)
  observe             Prometheus metrics, IP hasher, webhook dispatcher

groundshade-proxy/     hyper-based binary
  proxy.rs            top-level lifecycle (start, shutdown, signals)
  server.rs           inbound + admin accept loops, graceful shutdown
  service.rs          per-request dispatch
  upstream.rs         hyper client, URI rewrite
  ws.rs               WebSocket / Upgrade pass-through
  hop.rs              hop-by-hop header stripping (RFC 9110 §7.6.1)
  body.rs             ProxyBody type alias

groundshade-dashboard/ inlined HTML + JS dashboard
```

## Why the split

`groundshade-core` is transport-agnostic on purpose. Decisions take
a `RequestFacts` value (a borrowed view of the request) and return a
verdict. The v2 plugin work (Caddy module, nginx, HAProxy SPOA,
Envoy filter) wraps the same core. No rewrites.

`groundshade-proxy` owns everything that touches hyper or a socket:
accept loops, process lifecycle, graceful shutdown.

## Request lifecycle

```
                              ┌─────────────────────────────┐
                              │  groundshade-proxy::server  │
                              │    accept loop              │
                              └──────────────┬──────────────┘
                                             │  ConnectionLimiter::try_admit
                                             │  (per-IP /24 + global cap)
                                             ▼
                              ┌─────────────────────────────┐
                              │   hyper auto::Builder       │
                              │   (h1/h2 negotiation +      │
                              │    header_read_timeout)     │
                              └──────────────┬──────────────┘
                                             │  service_fn → handle()
                                             ▼
            ┌────────────────────────────────────────────────────────┐
            │  groundshade-proxy::service::dispatch                  │
            │                                                        │
            │  1. /.well-known/groundshade/* ────▶ challenge/solve   │
            │                                                        │
            │  2. extract host + path                                │
            │  3. Router::resolve(host, path) → EffectivePolicy      │
            │  4. compiled upstream lookup                           │
            │  5. policy.bypass? → forward (no defense, no sample)   │
            │                                                        │
            │  6. FastLane::evaluate_sync                            │
            │     (ip allowlist → feed glob → ua allowlist → apikey) │
            │     FastLane::evaluate_crawler (rDNS-verified bots)    │
            │     ─▶ on match: forward, no sample, no signals        │
            │                                                        │
            │  7. ja4_availability.observe(ja4.is_some())            │
            │  8. signals_evaluate(prefix, ja4, now_secs)            │
            │      → SignalsVerdict { immediate, soft_noisy, rate,   │
            │                          trustless }                   │
            │                                                        │
            │  9. trust cookie / Authorization: ChallengeSolution    │
            │     valid? → signals_note_trust_earned, forward,       │
            │              maybe renew                               │
            │                                                        │
            │ 10. signals_verdict.immediate is Some?                 │
            │     (rate_hard or trustless)                           │
            │     → render interstitial OR JSON 401,                 │
            │       signals_record_challenge                         │
            │                                                        │
            │ 11. level == Open? → forward + sample                  │
            │                                                        │
            │ 12. ChallengeSubject in scope for level                │
            │     OR signals_verdict.soft_noisy?                     │
            │     no  → forward + sample                             │
            │     yes → render interstitial OR JSON 401,             │
            │           signals_record_challenge                     │
            │                                                        │
            └────────────────────────────────────────────────────────┘
```

The challenge endpoints live at:

- `GET /.well-known/groundshade/challenge` issues a SHA-256 PoW
  offer.
- `POST /.well-known/groundshade/solve` redeems a solution for a
  `gs_trust` cookie.

## State surfaces

In memory:

- **Router.** Small, immutable, built at startup. Cloned via `Arc`.
- **DefenseRegistry.** One entry per known `route_id`. Each holds a
  detector ring buffer (capped at 10,000 samples), a state machine,
  and a `RouteSignals` block.
- **RouteSignals (per route).** A `RateState` with two `LruCache`s
  (`ClientPrefix → SlidingWindow` and `JA4 → SlidingWindow`,
  default 50,000 keys each) plus a `TrustlessState`
  (`ClientPrefix → ClientHistory`, default 100,000 keys).
- **Ja4Availability (global).** Atomic counters for requests seen
  and requests with JA4, plus a one-shot warning flag.
- **TrustIssuer.** Stateless except for the signing key.
  Verification is pure compute.
- **ChallengeIssuer.** Stateless except for an LRU replay cache of
  solved `(token, nonce)` pairs.
- **FastLane.** Feed matcher (immutable), API-key table (immutable),
  IP allowlist (immutable, parsed once), UA allowlist (immutable),
  crawler verifier with a bounded LRU (24 h positive, 10 min
  negative).
- **ConnectionLimiter.** Atomic global counter, bounded per-IP LRU
  (cap 16,384 prefixes), and an `Arc<IpAllowlist>` for the
  trusted-peer cap bypass.
- **Metrics.** Prometheus registry with a fixed handful of
  families.

On disk:

- `state_dir/trust.key`. 32 bytes, mode `0600`. Created on first
  start.

That's the entirety of GroundShade's persistent state in v1.

## Concurrency

- **Inbound.** One tokio task per TCP connection (spawned by the
  accept loop). Hyper serves multiple requests per connection on
  h1/h2 multiplexing.
- **Outbound.** One shared `hyper-util::client::legacy::Client`
  pool per `ProxyState`. Upgrades (WebSocket, generic `Upgrade`)
  use their own short-lived `hyper::client::conn::http1`
  connections; the pool can't safely reuse them after a protocol
  switch.
- **Background.** One defense-tick task ticks once per second
  across all routes. One webhook-dispatcher task drains the event
  queue. One RSS-watermark monitor toggles backpressure on Linux.

Every shared data structure with locking is documented at the lock
site. Most are `parking_lot::Mutex` (faster than std, doesn't
poison).

## Non-goals

- **No global mutex.** The router map is read-only after startup.
  The defense registry is per-route mutexed. Counters are atomics.
- **No dynamic dispatch in the hot path.** The decision tree is
  static enums and if-else chains.
- **No `unsafe`.** `#![forbid(unsafe_code)]` in `groundshade-core`
  and the proxy binary.
- **No third-party network calls** at startup or on the hot path
  beyond the system DNS resolver (for crawler verification) and
  configured webhook endpoints.

---

# Building from source

> Build GroundShade locally or for production. Local Docker images, multi-arch builds, Cargo profiles, cache strategies, and image layout.

Three audiences, three paths.

- **Just need a binary.** `make build-release` produces
  `target/release/groundshade`.
- **Self-hosting from Docker.** Use the published image
  (`codeberg.org/groundshade/groundshade:latest`) or build your own with
  `make docker`.
- **Cutting a release** (maintainer only). `make release TAG=vX.Y.Z`
  validates the working tree, runs CI, tags the commit, pushes the
  tag, and publishes the multi-arch image.

The builder image pins Rust 1.95 via the `RUST_VERSION` ARG in the
Dockerfile.

## Local Docker image

For local iteration:

```sh
make docker
docker run --rm -p 8080:8080 -p 127.0.0.1:9090:9090 \
  -e GROUNDSHADE_ADMIN_TOKEN=dev \
  groundshade/groundshade:dev
```

`make docker` uses the `release-fast` Cargo profile: release
semantics, no fat LTO, 16-way codegen. Compiles 2-3x faster than
full release with a small (under 5%) runtime cost. Fine for local;
tagged releases use full `release`.

## Multi-arch (arm64)

`make release-push` defaults to `linux/amd64`. Emulated arm64 via
QEMU on an x86 host takes 4-5x longer than native amd64. To opt in:

```sh
# Emulated. Slow; run once at release time, not in a tight loop.
make release-push TAG=v0.1.2 PLATFORMS=linux/amd64,linux/arm64

# Native via buildx node on an Apple Silicon mac or arm64 runner.
docker buildx create --name multi --driver docker-container --use
docker buildx inspect --bootstrap multi
make release-push TAG=v0.1.2 PLATFORMS=linux/amd64,linux/arm64
```

If you have a real arm64 machine on the network, attach it as a
buildx node so each platform builds natively:

```sh
docker buildx create --name multi --use            # local node (amd64)
docker buildx create --append --name multi \
  ssh://you@arm64-host                              # remote arm64 node
docker buildx build --platform linux/amd64,linux/arm64 ...
```

## Cache strategies

Cold cargo-chef build: 3-5 minutes for amd64 native, 10-20 minutes
for multi-arch via QEMU. After the first build, edits to workspace
crates rebuild in 30-60 seconds thanks to the dep layer cache.

Three ways to persist the cache:

- **Local layer cache** (default). Works on the same machine.
  Cache lives in Docker's layer store.
- **Registry-backed cache.** Pushes the dep-build layer to a
  registry, so a fresh machine (CI runner, colleague's laptop) can
  pull it.

  ```sh
  docker buildx build \
    --cache-to   type=registry,ref=codeberg.org/groundshade/groundshade-cache:buildcache,mode=max \
    --cache-from type=registry,ref=codeberg.org/groundshade/groundshade-cache:buildcache \
    --platform linux/amd64 \
    --tag codeberg.org/groundshade/groundshade:dev \
    --push .
  ```

- **sccache.** Shared rustc cache across crates and projects. Most
  useful when you build several Rust projects on the same machine.
  Set `RUSTC_WRAPPER=sccache` and start `sccache --start-server`
  before `cargo build`.

For a single-project setup, the layer cache is enough. Registry
cache pays off for ephemeral CI runners.

## Cargo profiles

| Profile | LTO | codegen-units | Use |
|---|---|---|---|
| `dev` | off | 256 | `cargo run`, fastest iteration |
| `release-fast` | off | 16 | Local Docker images, dev clusters |
| `release` | thin | 1 | Tagged production releases |

`release-fast` is the right default for anyone building their own
image. Use `release` when you cut a tag for real traffic.

## Image layout

Two stages:

- **builder.** `rust:1.95-bookworm` with `cargo-chef` installed.
  Plans the dependency graph from current sources, cooks deps in
  their own layer, then compiles the workspace.
- **runtime.** `debian:bookworm-slim` with `ca-certificates`,
  `wget` (for the healthcheck), and `tini` (so `docker kill -s
  HUP` reaches the binary cleanly). Runs as non-root UID 65532.
  Final image: around 140 MB.

State persists in `/var/lib/groundshade` (the HMAC signing key for
trust tokens, see [operating](/docs/operating)). The image
declares `/var/lib/groundshade` as a volume; mount it in
production.

Default container env:

- `GROUNDSHADE_LISTEN_HTTP=0.0.0.0:8080`
- `GROUNDSHADE_LISTEN_ADMIN=0.0.0.0:9090`
- `GROUNDSHADE_TRUST_STATE_DIR=/var/lib/groundshade`

The admin listener binds `0.0.0.0` inside the container; publish
the host port to loopback only (`"127.0.0.1:9090:9090"`).

## Why not distroless

The runtime image needs `wget` for the `HEALTHCHECK` and `tini` for
SIGHUP forwarding. A distroless base would drop both. Debian-slim
is 25 MB heavier and worth it for the operational ergonomics. If
you skip the Docker healthcheck and SIGHUP reload, building a
distroless variant is a 10-line change to the runtime stage.

---

# FAQ

> Common GroundShade questions. PoW choice, JA4 setup behind Caddy, dashboard exposure, Cloudflare compatibility, healthcheck cascades, and self-protection.

## Why SHA-256 PoW instead of Argon2id?

v1 ships SHA-256 because the browser computes it natively via
`crypto.subtle.digest('SHA-256', ...)`. No WASM artifact, no extra
crate on the client, no compilation pipeline. The wire format
already carries an `algorithm` field, so a memory-hard option
(Argon2id) can land in a later version behind it.

PoW is the last layer. JA4 fingerprinting, the browser-integrity
probe, and the fast lane all run before it. Even a GPU-friendly
PoW is meaningful work for a scraper that already paid for the
rest of the stack.

## Why doesn't GroundShade terminate TLS?

Terminating TLS makes the install more complex (ACME, key
management, cipher tuning) for little gain. The only thing it buys
is access to the raw ClientHello for JA4. GroundShade already gets
that from the `X-JA4` header your fronting proxy forwards.

See
[`examples/deploy/`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/deploy)
for one-line terminator configs (Caddy, nginx, HAProxy).

## My Googlebot requests are getting challenged

GroundShade fast-lanes a Googlebot UA only when the request's IP
passes **reverse-DNS plus forward-DNS verification** (the procedure
Google documents). If reverse DNS isn't set up correctly, or you're
testing from a non-Google IP, the bot UA alone won't bypass.

Verification results cache for 24 h on success and 10 min on
failure. The first verified request from a given Googlebot IP costs
one DNS round-trip; subsequent requests are free.

## Real users on slow phones see the challenge for too long

Lower `challenge.pow.leading_zero_bits` from the default `18` toward
`15` or `16`. Each bit doubles expected solve time.

## My CI job that scrapes our own API is getting challenged

Two paths:

1. Issue an API key for the CI client. Best for known clients. See
   [Operating](/docs/operating).
2. Set `prefer_json_challenge: true` on the relevant route and
   point the client at
   [`examples/solvers/solve.py`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/solvers/solve.py)
   to solve the JSON challenge once per session.

## Does GroundShade share my traffic with anyone?

No. v1 has zero phone-home. The only outbound calls are:

- Reverse-DNS plus forward-DNS lookups for verified-crawler
  verification, via your system resolver.
- Webhook deliveries to URLs you configure.

Nothing else leaves the proxy.

## Is the admin dashboard safe to expose publicly?

The admin port supports bearer-token auth via `listen.admin_token`
(or `GROUNDSHADE_ADMIN_TOKEN`). With a token set, every `/admin/*`
and `/metrics` request needs `Authorization: Bearer <token>` or the
`gs_admin` cookie that `/admin/login` sets.

Startup refuses to bind admin on a non-loopback address without a
token configured.

The safe default stays loopback (`127.0.0.1:9090`) and tunnels over
SSH or another authenticated channel. The token is the safety net
for deployments that publish admin to a wider network (Docker
inside-the-bridge, an internal VPN). Per-user accounts and RBAC are
not in scope for v1.

## My X-JA4 header is the literal string `{tls_client_ja4}`

Stock Caddy has no JA4 placeholder. The closest built-in is
`{tls_client_fingerprint}`, which is a TLS cert-based fingerprint
used for mTLS, unrelated to JA4. If your Caddyfile contains
`request_header X-JA4 {tls_client_ja4}` and Caddy was built without
a JA4 plugin, the placeholder passes through as text. Every trust
token GroundShade mints then binds to the literal string and a
stolen token works from anywhere.

GroundShade refuses to bind any X-JA4 value containing `{` or `}`
and logs a one-shot WARN the first time it sees one. The JA4 leg of
the binding drops on those requests; the trust token still binds to
`(route, IP-prefix)`, which is weaker but at least correct.

To get JA4 working, build Caddy with the `caddy-ja4` plugin via
`xcaddy`. The recipe lives at
[`examples/deploy/caddy-ja4.Dockerfile`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/deploy/caddy-ja4.Dockerfile):

```dockerfile
FROM caddy:2-builder AS builder
RUN xcaddy build --with github.com/matt-/caddy-ja4

FROM caddy:2-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
```

And the Caddyfile changes to:

```
{
    order ja4 first
    servers {
        listener_wrappers {
            ja4
            tls
        }
    }
}

example.com {
    ja4 { var_name ja4 }
    request_header -X-JA4
    request_header X-JA4 {http.vars.ja4}
    reverse_proxy groundshade:8080 { ... }
}
```

[`examples/deploy/caddy-ja4.Caddyfile`](https://codeberg.org/groundshade/groundshade/src/branch/main/examples/deploy/caddy-ja4.Caddyfile)
is the full version. Two non-obvious points:

- `order ja4 first` is required in the global block. The plugin's
  own README uses `order ja4 before respond`, which fires too late
  if `{http.vars.ja4}` is consumed by `request_header` (which runs
  before `respond`). `first` makes the directive run before any
  handler that reads the variable.
- After any change to the `listener_wrappers` global block, do a
  full Caddy restart. `caddy reload` only refreshes HTTP handlers;
  listener wrappers attach at socket setup. A reload appears to
  succeed but silently leaves the wrapper inactive,
  `{http.vars.ja4}` expands to empty, and JA4 binding stays off.

With this build, the `ja4` directive fails loudly at config-load
time if the plugin is missing.

## Can I run GroundShade behind Cloudflare?

Yes. Orange-cloud (CF terminates TLS) works after v0.7.0 with two
unavoidable limits: JA4-keyed signals go silent (CF terminates the
handshake) and the verified-crawler rDNS fast lane goes dead (rDNS
resolves CF's edge, never `googlebot.com`). Per-IP rate, trustless
persistence, trust tokens, and operator allowlists keep working.

Grey-cloud (DNS-only) preserves everything and is the recommended
mode when possible. Full setup: [behind cloudflare](/docs/cloudflare).

## Behind Caddy, the upstream gets marked dead under load and every request returns 503

Caddy's active healthcheck probes `/health`:

```
reverse_proxy groundshade:8080 {
    health_uri /health
    health_interval 30s
    health_status 200
}
```

GroundShade is transparent, so `/health` proxies straight through
to your real upstream. If that upstream rate-limits per-IP (and
from its viewpoint all traffic comes from Caddy's container IP), a
heavy burst rate-limits the healthcheck too. Caddy gets back `429`,
expects exactly `200`, marks the upstream dead, and every
subsequent request returns `503 no upstreams available` until the
next probe.

It's the same cascade you'd hit with any transparent proxy in front
of a rate-limited app, or with no proxy at all.

The fix lives in the fronting proxy. Switch to passive detection:

```
reverse_proxy groundshade:8080 {
    fail_duration 10s
    max_fails 3
    unhealthy_status 5xx
}
```

Caddy now marks the upstream down only after three consecutive
responses that look like a real server problem (a 5xx). `429`
doesn't count, so the rate-limit bucket can't trigger the cascade.
Recovery is automatic once real traffic starts succeeding again.

## Does the proxy survive its own attack?

Yes, by design. Every data structure is bounded:

- Detector ring buffer: 10,000 samples per route.
- Connection limiter per-IP LRU: 16,384 prefixes.
- Rate signal LRUs (IP prefix and JA4): 50,000 keys each per route
  by default (`defense.rate_signals.max_keys_per_route`).
- Trustless persistence map: 100,000 prefixes per route by default
  (`defense.trustless_persistence.max_keys`).
- Crawler-verifier LRU: 24 h positive, 10 min negative.
- Trust-token replay LRU: bounded.

A CI load gate asserts survival under sustained attack
(100k req/s from 10k synthetic attackers on 2 cores) and any PR
that regresses it doesn't merge.

`selfdef::pressure::spawn_monitor` polls `/proc/self/statm` once
per second on Linux. When RSS crosses `selfdef.rss_high_watermark_mb`
(default 512 MB), the connection limiter's backpressure flag flips,
new connections are refused at the accept layer, and the
`groundshade_self_throttle` gauge goes to `1`. New connections
resume when RSS drops back below. On non-Linux the monitor is a
no-op.