Chat context and cost decisions (CTX)#

Decisions about how much the chat agent sends to a model, what it costs, and why the request is shaped the way it is. For the settings themselves, see the chat install docs.

Primary code: reporting/services/chat_context.py, reporting/services/chat_budget.py, reporting/services/chat_graph.py.

CTX-001 — Context is budgeted in tokens against the model’s own window#

Applies to: reporting/services/chat_context.py

context_window_tokens reads max_input_tokens from litellm (cached per model); unknown models fall back to CHAT_LLM_CONTEXT_WINDOW_FALLBACK_TOKENS. history_token_budget = min(CHAT_LLM_CONTEXT_MAX_TOKENS, window × CHAT_LLM_CONTEXT_WINDOW_SHARE).

Why: the caps used to be characters at an assumed 4 chars/token. Real tool payloads measure 3.0, so those caps admitted a third more than intended. Where something must truncate characters against a token budget, use chars_for_tokens, which calibrates on the text in hand.

The window is a ceiling, not a target. A 1M-token model must not silently multiply per-call cost, which is why CHAT_LLM_CONTEXT_MAX_TOKENS still applies. The fallback is deliberately small: guessing low wastes a window, guessing high fails the turn.

Don’t: count tokens without the cache. count_tokens is content-hash cached because a trim pass sizes every message on every call.

CTX-002 — The whole request is fitted, and overflow is retried once against what was sent#

Applies to: _run_llm_tool_turn, _fit_messages_to_window in chat_graph.py

_fit_messages_to_window subtracts system prompt, tool schemas, reply allowance, CHAT_LLM_CONTEXT_SAFETY_MARGIN and per-message framing from the window. It runs in _run_llm_tool_turn because that is the single scope where the whole request exists, so every LLM call is covered.

A provider overflow (chat_context.is_context_overflow, matched by litellm type or message across the exception chain) is caught once and retried with the conversation halved relative to what was sent, not relative to our allowance.

Why: an overflow is our allowance being wrong, so halving the allowance is no help.

Don’t: retry after text has streamed — it duplicates output. Don’t retry non-overflow errors at all.

CTX-003 — Cross-turn history is compacted, not truncated, and deterministically#

Applies to: _compact_history, HistorySummary in chat_graph.py

The oldest turns are condensed into one block, cut back past the budget in chunks so the block stays byte-identical between compactions. Measured on 40 simulated turns: 5 compactions, request bounded 2,421–3,348 tokens against a 4,000 budget.

Why deterministic: never a summarizing model call. Its output would differ per run and rewrite the prefix, invalidating the prompt cache for the whole conversation (see CTX-004).

Why chunked: cutting back exactly to the budget re-compacts every turn.

The block is fenced with fenced_within. Compaction flattens assistant turns into a user message, and an assistant turn carries whatever tool and graph output it reported on — unfenced, that promotes provider-controlled text into the instruction role, and keeps it there, because the block is deliberately stable.

The block gets a reserved share of the budget (_SUMMARY_BUDGET_SHARE). Without the reserve, a grown summary competed with the history it described and every turn re-compacted.

The boundary is a message id (covers_through_id), not a count, because a count shifts when filtering changes. An empty id must never be looked up: it matches every message that carries none, which silently dropped the first message of a fresh conversation.

Still outstanding: compaction is not append-only, which the original context plan called for. Rewriting the prefix invalidates the prompt cache for the whole conversation.

CTX-004 — Volatile content goes last, never in the system prompt#

Applies to: chat_graph.session_memory_message

Prefix caching matches the longest common prefix, so anything that changes per turn must not sit at the front of the request.

Why: the session digest in the system prompt measured 0% cached against 98% for an otherwise identical prefix. Carried as a trailing message it leaves only the newest exchange uncached (93%/74% on a live two-turn conversation).

This is the provider-agnostic half — automatic prefix caches (DeepSeek, OpenAI, Gemini) need nothing more.

Don’t: add anything per-turn to a system prompt without checking this first.

CTX-005 — Anthropic needs explicit cache breakpoints#

Applies to: chat_context.with_cache_breakpoints

Anthropic caches nothing without them (measured: 0 cached over a five-call turn). Up to three blocks are marked with cache_control, for Anthropic models only: the system prompt (tool schemas are ordered ahead of it, so one mark covers both), the message before the session digest (a prefix containing the digest can never be read back), and the last message (within-turn rolling).

Live two-turn measurement: turn 1 writes ~11,000 and reads 0 (cold; writes cost 1.25x), turn 2 reads 16,467 (56%) and writes 1,801.

Don’t: reshape non-Anthropic requests into content blocks — it risks a provider transformer for no gain.

CTX-006 — What the prefix contains matters more than the marks#

Applies to: chat_orchestrator._worker_system_prompt, _step_contract, _step_declared_tool_names

_worker_system_prompt() takes no step argument and must stay that way. It used to embed the step goal, criteria and required action, so every step had a different prefix head and none could read another’s — measured: the second step read 0 of 2,963 tokens. Step-specific text lives in _worker_user_message via _step_contract, still fenced.

Tool-list churn invalidated the prefix the same way. Anthropic orders tools ahead of the system prompt, so going from 3 to 11 tools made the next call read 0. Fixed by honouring skills’ tools_required up front, scoped and bounded:

  • _step_declared_tool_names takes only the skills this step names (required_action / suggested_tools). Unioning every enabled skill’s declaration is the catalogue, not the need — it took a single-agent turn from 1 bound tool (343 tokens) to 43 (4,666).

  • mcp_runtime.declared_tool_names(prompts, only=…) reads declarations off the prompt _meta of the listing the turn already made — no second store read. Use _meta, not meta, or the alias shadows it.

  • chat_graph.skill_declared_tool_names bounds the result by CHAT_LLM_DISCLOSE_SKILL_TOOLS_MAX_TOKENS, measured in schema tokens rather than tool count.

After the change a turn held one tool list across all four calls, each reading the previous prefix. The single-agent path has no signal for which skills a turn will use, so it still discloses on render.

CTX-007 — Cache diagnostics are ours, because Anthropic’s are unreachable#

Applies to: chat_context.diagnose_cache_divergence, CHAT_LLM_CACHE_DIAGNOSTICS

Fingerprints model/system/tools/messages as hashes per call and logs the earliest divergence with the tokens behind it (tools_changed, ~4000 tokens…). Off by default.

Why not Anthropic’s beta: unreachable through LiteLLM 1.87.0. It builds anthropic-beta from feature detection and drops caller extra_headers/headers (verified on the wire), so the diagnostics body param arrives unauthorised and the API rejects the call outright. Do not wire it up without fixing the header first.

Scoping is load-bearing. Keyed by thread+phase — per delegation for the sandbox, whose delegations all open with the same system prompt — and by lineage (the opening message). A plan reuses step ids across turns, so without both, turn 2’s worker:s2 was diffed against turn 1’s unrelated one and every first call reported a divergence.

Do: reach for this instead of hand-adding hash logging. Both cache bugs in this area were found with it.

CTX-008 — Budgeting is cache-aware, but reservations are not discounted#

Applies to: reporting/services/chat_budget.py

usage_from_message reads input_token_details.cache_read/cache_creation off the response; usage_cost_usd forwards them to litellm, which treats prompt_tokens as the total and subtracts the cached portions. Pass the total, not the difference, and clamp the details to it — a provider over-report otherwise negative-prices the call. Committed cost is exact.

Reservations use the uncached price. project_cost_usd does not apply the observed hit rate.

Why: that ratio spans every model and phase, so a cache-heavy sandbox phase discounted a cold planner call on another model (reproduced 6.6x under-reserved). More fundamentally, a cache hit is never guaranteed, and a ceiling that assumes one is not a ceiling. The ledger self-corrects on commit.

Tokens are counted whole — a cached token still occupies the context window. Only the price differs. cache_read_tokens is in the ledger and per phase.

CTX-009 — Measure with the harness, never single runs#

Applies to: scripts/chat_harness.py

Answer quality on an unchanged configuration varies several-fold. Single-run comparisons of this system have repeatedly shown “clean separations” that vanished under more samples; four settings have been swept without a distinguishable difference between any of them.

The harness runs the same conversation N times per arm and reports medians and ranges. Tokens, cost and cache hit rate are summed across every turn, not read off the last one.

Traps:

  • A subagent detail is re-emitted after every inner tool call carrying the whole children list, so any stream-derived count must be de-duplicated on detail_id or it inflates quadratically in the size of a delegation.

  • Do not edit code while a run is in flight — the dev server reloads and the run is contaminated.

  • summary_chars is non-zero only when history compaction engaged, which needs a long run or a small CHAT_LLM_CONTEXT_MAX_TOKENS.