# CVE remediation

## Purpose

`cve_dependency_remediation` is a [built-in workflow](built-in-workflows.html)
that opens pull requests upgrading vulnerable dependencies. Given rows naming a
repository and a vulnerable package, it runs a headless coding-agent CLI in an
ephemeral sandbox to make the upgrade, then pushes a branch and opens a PR.

It is the most privileged thing Seizu does: a coding agent reads untrusted
repository code, and a GitHub token can push. Those two never coexist — see
[How a run works](#how-a-run-works).

## Setting it up

You need three things: a sandbox provider, a coding-agent provider key, and a
GitHub token. Configuring the GitHub token is what enables the workflow; there
is no separate feature flag.

```
SANDBOX_API_KEY=e2b_...              # see Sandbox; SANDBOX_ENABLED is not required
SANDBOX_AGENT_PROVIDER=claude        # or codex, opencode
SANDBOX_AGENT_API_KEY=sk-ant-...     # the CLI's provider key
REMEDIATION_GITHUB_TOKEN=...         # see below
```

The workflow does not use the chat LLM. Add it as an activity to a workflow
whose query returns rows with `repo` and `package` keys — the seeded
**New CVE dependencies requiring remediation** query does this.

**Always verify the token before enabling the workflow:**

```bash
make remediation_smoke SMOKE_REPO=org/repo     # add SMOKE_FORK=1 for fork mode
```

This opens real sandboxes and reproduces the clone, the credential handoff and
the push, ending in `SMOKE PASS`. It does not run a coding agent or open a PR.
In direct mode you can check write access first with
`gh api repos/<org>/<repo> --jq .permissions` — `push` must be `true`.

```{important}
**A public target repository hides auth failures until push.** Cloning a public
repo is anonymous, so the token is first exercised on `git push` — a successful
clone proves nothing. Always push a branch when testing.
```

### GitHub token

Which token you need depends on the mode. In both, the token is used to clone,
push, open PRs, and by the worker-side CI watch — never while the coding agent
is running.

**Direct mode (default)** pushes work branches into the target repositories, so
the token needs write access there. Use a **fine-grained PAT** with resource
owner set to the org that owns the targets, repository access limited to the
repositories you want remediated, and:

| Permission | Why |
|------------|-----|
| Contents: Read and write | Clone and push the work branch |
| Pull requests: Read and write | Open PRs; read PR state for the CI watch |
| Issues: Read and write | The CI watch posts triage comments |
| Checks: Read, Commit statuses: Read | Poll check runs and legacy statuses |
| Actions: Read | Job-log tails for the fix agent (optional; degrades gracefully) |
| Workflows: Read and write | Only if fixes may touch `.github/workflows/` |

Organizations may require fine-grained PAT approval under *Org settings →
Third-party access*. A classic PAT with `repo` also works but grants far more
than needed.

**Fork mode** (`REMEDIATION_USE_FORK=true`) pushes to a bot-owned fork instead,
so the token needs no write access to the targets. A fine-grained PAT cannot do
this — it has a single resource owner and so cannot both write to the bot's
forks and open PRs on targets owned by someone else. Use a dedicated machine
account with a **classic PAT**:

1. Create a machine account for this automation only (e.g. `myorg-remediation-bot`).
   Forks accumulate under it, or under `REMEDIATION_FORK_ORG`.
2. As that account, create a classic PAT with `public_repo` (all targets public)
   or `repo` (any target private); add `workflow` if fixes may modify
   `.github/workflows/`.
3. For private targets, give the machine account read access on each target and
   enable the target org's *Allow forking of private repositories*.
4. Check the target org allows classic PATs, and authorize the token for SAML SSO
   if enforced.

A GitHub App installation token works in either mode if you have tooling to mint
and rotate it — Seizu just reads `REMEDIATION_GITHUB_TOKEN`.

**Keep branch protection on.** Nothing should merge without human review.

### Turning it off

There is no per-user permission: the workflow is reachable only through
workflows, which are admin-managed (`workflows:write`). Disable the workflow,
remove `REMEDIATION_GITHUB_TOKEN`, or drop `cve_dependency_remediation` from
`TEMPORAL_ENABLED_WORKFLOWS`.

## How a run works

Per (repository, package) group the workflow runs four sandbox commands, each
given only the environment it needs:

| Phase | Credentials | What it does |
|-------|-------------|--------------|
| install | **none** | Install `gh` and the agent CLI. No secrets, because npm postinstall scripts are third-party code. |
| setup | GitHub token | Clone and create the work branch, via `gh auth setup-git`. No token is written to disk or embedded in the clone URL. |
| agent | provider key only | Run the coding agent. **No GitHub token exists in the sandbox during this phase**, so a prompt-injected agent has nothing to exfiltrate and cannot push. |
| push | GitHub token | Verify the agent committed, push the branch, open or update the PR. |

The agent upgrades the dependency in every affected manifest, including
compatibility code changes, and writes the PR title and body. It does **not**
run the test suite — the sandbox usually lacks the dependencies — so CI tests
the PR.

Two behaviours worth knowing:

- **Least-change upgrades.** The agent targets the smallest released version
  clearing every vulnerable range, preferring the current major, then minor.
- **Branches are version-keyed**
  (`seizu/dependency-update/{ecosystem}-{package}-{version}`). If an open PR
  already exists for the branch, the run is skipped rather than force-pushing
  over a PR under review; a later fix needing a higher version gets its own
  branch. Manual re-runs are therefore safe.

Groups run sequentially to bound agent spend, and a group is never retried — a
retry would repeat an expensive run and risk duplicate PRs. A failing group
records an error without aborting the rest.

## Fork mode caveats

With `REMEDIATION_USE_FORK=true`, forks are created on demand and the PR is
opened cross-repo. Three things catch people out:

- **The token owner must be able to fork the target.** A repository cannot be
  forked into the account that already owns it.
- **Many organizations restrict Actions on fork PRs** (no secrets, or
  approve-first). If the CI watch keeps reporting `no_checks`, check the
  target's Actions fork policy.
- **CodeQL default setup never analyzes fork PRs.** A target combining default
  setup with a code-scanning merge rule blocks fork PRs indefinitely. Switch
  those repositories to CodeQL advanced setup — a committed `codeql.yml` with a
  `pull_request` trigger, which runs in the base-repo context.

Credential phase isolation is identical in both modes.

## CI watch and fix

An upgrade can break CI, so after pushing a PR the workflow watches its checks,
polling every `REMEDIATION_CI_POLL_SECONDS` until they settle or
`REMEDIATION_CI_MAX_WAIT_SECONDS` elapses (`0` disables the watch). Checks stuck
*queued* past `REMEDIATION_CI_QUEUED_STUCK_SECONDS` are ignored rather than
waited on, as are cancelled and stale runs. A merged or closed PR ends the watch.

If any check fails, the workflow runs up to `REMEDIATION_CI_FIX_MAX_ATTEMPTS`
coding-agent fix runs (`0` → watch and record only), reusing the same
phase-isolated flow against the PR branch. The agent triages each failure:

- **Caused by the upgrade** → it fixes and commits; the new commits are pushed
  from a fresh push sandbox, re-triggering CI.
- **Unrelated** (flaky, already failing on base, infrastructure) → it writes an
  explanation, which the worker posts through the GitHub API. The text is
  rendered into a fixed template as a block quote with `@`-mentions and
  slash commands neutralized, so a prompt-injected agent cannot ping people or
  drive bots under the bot identity.

The result records `ci_status`: `passed`, `fixed`, `failures_commented`,
`ci_failed`, `fix_failed`, `timed_out`, `no_checks`, `merged`, `pr_closed`, or
`error`, with detail in `ci_detail`.

## Protecting the agent's provider key

The provider key is the one credential present while the agent runs untrusted
code. Three options, in increasing order of protection:

1. **A static key** (`SANDBOX_AGENT_API_KEY`) — simplest, but a long-lived key
   in that sandbox is stealable. The worker logs a warning.
2. **A minted short-lived key** (`SANDBOX_AGENT_API_KEY_COMMAND`) — a command
   whose stdout becomes that run's key. Recommended if you have a broker
   (Vault, a gateway's virtual-key issuer). Pair with `SANDBOX_AGENT_BASE_URL`
   to point at your gateway.
3. **The built-in credential proxy**
   (`SANDBOX_AGENT_CREDENTIAL_PROXY_ENABLED=true`) — for Anthropic and OpenAI,
   whose direct APIs have no short-lived tokens. A *separate* sandbox runs a
   LiteLLM proxy holding the real key, and the agent sandbox gets only an
   ephemeral per-run key with a spend cap
   (`SANDBOX_AGENT_CREDENTIAL_PROXY_MAX_BUDGET`). The proxy dies with the run,
   so a leaked key is worthless afterwards. Mutually exclusive with
   `SANDBOX_AGENT_BASE_URL`; for `opencode`, set `SANDBOX_AGENT_MODEL`.

```{warning}
The proxy stands up LiteLLM inside a sandbox and depends on your agent CLI
talking to it correctly. **Verify it against your CLI and LiteLLM versions
before enabling in production:** `make remediation_smoke SMOKE_PROXY=1` boots
the private proxy and confirms a second sandbox can reach it.
```

### Keeping the proxy's LiteLLM pinned

Without a template, the proxy sandbox installs LiteLLM on every run from a
hash-locked requirement file. This is pinned deliberately: LiteLLM leaves
FastAPI, pydantic, aiohttp, openai and httpx on open ranges, and an incompatible
release published upstream is enough to break every remediation run — in the
sandbox holding your real provider key.

To move to a newer LiteLLM:

```bash
make lock_proxy_requirements REQUIREMENTS="litellm[proxy]==1.90.0"
make build_proxy_template              # required if you use a template
make remediation_smoke SMOKE_PROXY=1   # prove it still boots
```

Keep the old pin if the smoke test fails. Three gotchas:

- **The FastAPI pin is a ceiling, not a floor.** LiteLLM accepts a range but
  imports symbols newer FastAPI releases have removed. When bumping LiteLLM,
  check before raising the FastAPI pin.
- **A lock is only valid for the interpreter and architecture it was built for.**
  The shipped lock targets python 3.13, which is what a templateless E2B sandbox
  runs — *not* the `e2bdev/base` image's 3.11. The install checks the sandbox
  against the lock's header and fails with a re-lock instruction rather than a
  wall of resolution errors. For a self-hosted backend on a different runtime,
  compile your own:

  ```bash
  make lock_proxy_requirements PYTHON_VERSION=3.12 PLATFORM=aarch64-unknown-linux-gnu \
      OUTPUT=locks/litellm-3.12-arm.txt
  # then point SANDBOX_AGENT_CREDENTIAL_PROXY_REQUIREMENTS_FILE at it
  ```
- **With no arguments, `make lock_proxy_requirements` re-locks in place**, reading
  the target runtime from the lock's own header, so it cannot overwrite a
  different lock.

### Prebuilding the proxy template

Installing that tree on every run costs time and makes each run depend on PyPI.
`make build_proxy_template` bakes the pins into an E2B template:

```bash
make build_proxy_template                        # or TEMPLATE_NAME=my-proxy
# then, on the temporal worker:
SANDBOX_AGENT_CREDENTIAL_PROXY_TEMPLATE=seizu-litellm-proxy
```

E2B templates are a cloud feature; with a self-hosted `SANDBOX_DOMAIN` there is
nothing to build and the run-time install covers it.

```{warning}
**A configured template is used exactly as you built it — runs install nothing
and check nothing.** Nothing notices a template built from older requirements,
or one missing LiteLLM altogether (that surfaces as a health-check failure at
proxy start). Re-lock and rebuild together when you bump pins.
```

## Configuration

| Variable | Default | Description |
|----------|---------|-------------|
| `SANDBOX_AGENT_PROVIDER` | `claude` | Coding-agent CLI: `claude` (Claude Code), `codex`, or `opencode`. `opencode` is multi-provider — set `SANDBOX_AGENT_MODEL` to a `provider/model` id (e.g. `deepseek/deepseek-v4-pro`) and it uses that provider's key, reusing the same global `*_API_KEY` (e.g. `DEEPSEEK_API_KEY`) the chat assistant uses. For opencode, an explicit `SANDBOX_AGENT_API_KEY` must belong to the model's provider — an Anthropic key with a `deepseek/…` model is exported as `DEEPSEEK_API_KEY` and fails auth. |
| `SANDBOX_AGENT_TEMPLATE` | `""` (official) | E2B sandbox template. Empty → the provider's official prebuilt template (E2B ships first-party `claude`/`codex`/`opencode` images with the CLI installed), which removes the per-run `npm install` and its postinstall scripts from the flow. A template name → that template (e.g. a self-pinned copy). `none` → the plain base image (the run installs the CLI itself). Ignored on self-hosted backends (`SANDBOX_DOMAIN` set): E2B templates are a cloud feature, and the idempotent install step covers those. The template provides tools only, never credentials, so the phase isolation above is unchanged. |
| `SANDBOX_AGENT_API_KEY` | `""` | Static API key for the CLI, exported only to the agent phase. Empty → falls back to `ANTHROPIC_API_KEY` for `claude`. Prefer the key command below. |
| `SANDBOX_AGENT_API_KEY_COMMAND` | `""` | Command run in the worker before each remediation; its stdout becomes that run's agent API key. Use it to mint **short-lived** credentials from a broker (Vault, an LLM-gateway virtual-key issuer, …) instead of handing the sandbox a long-lived key. Takes precedence over the static key. **Recommended for production:** unlike the GitHub token (kept out of the agent sandbox), the agent's provider key is present while it runs untrusted repo code with internet on, so a long-lived key is stealable — the worker logs a warning when a static key is used. |
| `SANDBOX_AGENT_BASE_URL` | `""` | LLM gateway/proxy base URL exported to the agent phase (`ANTHROPIC_BASE_URL` / `OPENAI_BASE_URL`); typically paired with the key command so the sandbox only ever holds a short-lived gateway key. Mutually exclusive with the credential proxy below. |
| `SANDBOX_AGENT_CREDENTIAL_PROXY_ENABLED` | `false` | Run a short-lived LiteLLM proxy in its **own** sandbox holding the real provider key, and hand the agent sandbox only the proxy's ephemeral per-run key with an in-memory spend cap (see below). All providers (`opencode` needs `SANDBOX_AGENT_MODEL` set). |
| `SANDBOX_AGENT_CREDENTIAL_PROXY_MAX_BUDGET` | `5` | USD spend cap (LiteLLM's in-memory global `max_budget`) — bounds real-time abuse of the ephemeral key while the proxy is up. |
| `SANDBOX_AGENT_CREDENTIAL_PROXY_REQUIREMENTS_FILE` | `""` (checked-in lock) | Hash-locked requirement file a **templateless** proxy sandbox installs. Empty → `reporting/services/sandbox_proxy_requirements.txt`. Point it at your own compiled lock to run a different LiteLLM, or one resolved for a different sandbox runtime; see [Keeping the proxy's LiteLLM pinned](#keeping-the-proxys-litellm-pinned). |
| `SANDBOX_AGENT_CREDENTIAL_PROXY_TEMPLATE` | `""` | E2B template for the proxy sandbox, built from those requirements by `make build_proxy_template`. Set → runs use that image **as built** and install nothing. Empty → the plain base image, and every run installs LiteLLM first. |
| `SANDBOX_AGENT_MODEL` | `""` | Model for the CLI. For `claude`/`codex` a bare model override (empty → the CLI's default). For `opencode` it is **required** and takes the `provider/model` form (e.g. `deepseek/deepseek-v4-pro`), which also selects the provider key and — in credential-proxy mode — the LiteLLM namespace. |
| `REMEDIATION_TIMEOUT_SECONDS` | `1800` | Hard cap for one remediation run (all sandbox phases). Also caps each CI-fix run. |
| `REMEDIATION_CI_MAX_WAIT_SECONDS` | `3600` | Total time the workflow watches one PR's checks (including re-runs after a fix push). `0` disables the CI watch. |
| `REMEDIATION_CI_POLL_SECONDS` | `120` | Interval between check-status polls. |
| `REMEDIATION_CI_QUEUED_STUCK_SECONDS` | `1800` | A check still queued (never started) after this long is ignored by the watch instead of waited on. |
| `REMEDIATION_CI_FIX_MAX_ATTEMPTS` | `1` | Coding-agent CI-fix runs allowed per PR (each is a full sandbox agent session — this bounds spend). `0` → watch and record the outcome but never fix. |
| `REMEDIATION_GH_SHA256` | `""` | Expected SHA-256 of the pinned `gh` linux_amd64 tarball. Set it (out of band) for an independent supply-chain pin, or bake `gh` into a pinned sandbox image — since the installed `gh` later handles the token. Empty → verify against the release's own checksums (integrity only). |
| `REMEDIATION_GITHUB_HOST` | `github.com` | GitHub host the target repositories live on — `github.com` or a GitHub Enterprise Server hostname. Used for the clone URL and `gh` (`GH_HOST`/`GH_ENTERPRISE_TOKEN`). |
| `REMEDIATION_GITHUB_TOKEN` | `""` | GitHub token for the setup/push phases and the worker-side CI watch: a minimally-scoped fine-grained PAT in direct mode, a machine-account classic PAT in fork mode — see [GitHub token](#github-token). Required (configured = enabled). |
| `REMEDIATION_USE_FORK` | `false` | Push the work branch to a bot-owned fork (created on demand) and open cross-repo PRs instead of writing branches into the target repositories — see [Fork mode caveats](#fork-mode-caveats). |
| `REMEDIATION_FORK_ORG` | `""` | Organization that owns the bot forks in fork mode; empty → forks live under the token user's account. |
| `REMEDIATION_GIT_USER` / `REMEDIATION_GIT_EMAIL` | `seizu-remediation-bot` / `seizu-remediation@localhost` | git author identity for the remediation commits. |

Temporal and general workflow settings are in
[workflow configuration](workflows.html#configuration). The sandbox provider
settings (`SANDBOX_API_KEY`, `SANDBOX_DOMAIN`) are in [Sandbox](sandbox.html).

```{note}
**The temporal worker does not hot-reload.** It imports workflow code once at
startup, so after changing remediation code a run silently uses the old code
until you `docker compose restart seizu-temporal-worker` (dev) or rebuild the
image.
```