Seizu Documentation
Chat agent decisions (AGT)#
Decisions about what the chat agent may do, what it is shown, and how a turn is driven. Cost and context shaping live in chat context; sandbox delegation in sandbox.
Primary code: reporting/services/chat_graph.py,
reporting/services/chat_orchestrator.py, reporting/services/mcp_runtime.py,
reporting/services/headless_chat.py, reporting/services/agent_run.py,
reporting/services/chat_turns.py,
reporting/temporal_workflows/chat_turn.py.
AGT-001 — Chat tools fail closed#
Applies to: mcp_runtime.list_tools_for_user(chat_safe_only=True)
Chat sees only built-ins whose required_permissions are all in
_CHAT_SAFE_PERMISSIONS (read/inspection only), plus mutating built-ins that
carry an action-confirmation resolver.
Why: a newly added write or delete tool is hidden from chat by default rather than exposed by default. The failure mode of the alternative is silent.
The exceptions are documented at the tool registration, and there are only two:
reports__create— creates a new private report and modifies nothing existing. It still carries a conditional resolver for the one case where it publishes (filing into a space).sandbox__delegate— isolation is the control; see SBX-009.
reports__clone is gated unconditionally, because whether it publishes
depends on the source’s placement, and reading that is the resolver’s job rather
than the handler’s.
Don’t: add a third exception without recording it here and at the registration.
AGT-004 — Thread ids are namespaced server-side#
Applies to: chat_graph.namespaced_thread_id
user:{user_id}:thread:{client_thread_id}.
Why: clients supply the thread id, so without server-side namespacing one user can reach another’s thread. Everything downstream — checkpoints, the sandbox resume id, session memory — inherits this isolation, which is what makes those safe to persist.
AGT-005 — Report configs are validated at save time#
Applies to: CreateVersionRequest.validate_report_config
Configs are validated against the Report schema when saved, and markdown
panels must carry content in markdown.
Why: without it an agent stores a config that renders empty and gets no signal. An actionable error at save time is the difference between a retry and a silently broken report.
AGT-006 — Headless runs execute as the schedule’s creator, with permission-based bypass#
Applies to: reporting/authnz/headless.py, reporting/services/agent_run.py
Identity: resolve_stored_user() rebuilds the creator’s permissions from the
last role claim seen on an authenticated request (User.role, synced by
get_or_create_user). Archived users hard-stop.
Confirmation bypass is gated by chat:bypass_permissions (Editor+).
call_tool_for_chat(bypass_confirmations=True) re-checks the permission and
AUDIT-logs every bypassed execution. Without it, headless runs keep the normal
confirmation flow, so mutating tools fail closed for the run. Creator RBAC and
chat_safe_only always apply.
The same permission gates the chat UI’s “Bypass confirmations” toggle
(ChatStreamRequest.bypass_confirmations, 403 without it, default off).
Headless turns get a system-prompt addendum (_HEADLESS_PROMPT_ADDENDUM)
telling the model nobody can answer: don’t ask for confirmation, follow the
skills, summarize blocks instead of retrying.
Don’t: weaken any of this without explicit sign-off. A headless run is an unattended agent holding a real user’s permissions.
AGT-007 — Scheduled chats run on Temporal, and their sessions are not chat sessions#
Applies to: reporting/services/chat_schedules.py,
reporting/temporal_workflows/scheduled_chat.py
Execution is Temporal-only; the polling worker is gone. Run sessions are created
with origin="scheduled" + scheduled_chat_id, excluded from
list_chat_sessions, listed via list_scheduled_chat_sessions, and
admitting a turn on one is rejected (403).
Why the rejection: a scheduled run’s transcript is a record of what an unattended agent did. Allowing a user to continue that thread interactively would blur which turns were unattended.
Schedules are per-user (created_by owner; list/get/update/delete are
owner-scoped, 404 otherwise). chat:schedule:read_all (Admin) unlocks viewing
every user’s schedules; mutations stay owner-only.
Overlap is ScheduleOverlapPolicy.SKIP with no mutex, unlike
ConfiguredWorkflow, and runs use maximum_attempts=1. Disabling pauses the
Schedule rather than deleting it. Monthly specs over-fire on days 28–31, and
load_scheduled_chat drops the non-matching firings via schedule_due.
POST /chat/schedules/<id>/run is keyed on run_requested_at so a recovering
reconcile pass is idempotent.
AGT-008 — An interactive turn is detached from the connection watching it#
Applies to: reporting/services/chat_turns.py,
reporting/temporal_workflows/chat_turn.py, reporting/routes/chat.py,
reporting/services/report_store (chat turn log), src/api/chatTransport.ts
A turn used to be the HTTP request: graph.astream was iterated inside the
StreamingResponse generator. Now a turn is a Temporal workflow writing an
append-only log of stream parts, and a request is a reader tailing it. The
two share only a turn id, so a client can disconnect and reattach.
Why: two failures come from the old shape, and both are structural.
gunicorn.conf set no timeout, so the 30s default applied — and under
UvicornWorker that is a heartbeat watchdog, so a loop blocked past it gets the
worker SIGABRTed mid-response and the client receives a truncated body
rather than an error (observed three times under the harness, as
IncompleteRead after ~470 bytes). Separately, a dropped connection destroyed
the turn: Starlette cancels a StreamingResponse generator on
http.disconnect, so closing the tab killed minutes of work outright. An
explicit timeout = 300 originally masked the first; only detaching the turn
fixes the second and makes a web-worker restart recoverable. With production
now owned by Temporal, keeping a wedged web worker alive for five minutes has no
chat benefit. The bundled Gunicorn watchdog therefore follows
API_REQUEST_TIMEOUT (60 seconds by default). A healthy UvicornWorker keeps
notifying Gunicorn during a long-lived stream, so chat duration does not set
this value.
Interactive chat now requires Temporal#
This reverses the scoping in issue #254, which asked for the turn to be
detached without adding a dependency, and the first implementation duly ran the
producer as a detached asyncio.Task in the web process. What that cost is the
point: a detached task has no identity, so everything a workflow gives for free
had to be rebuilt by hand — a renewable lease to prove it was alive, a
process-local registry to find it, crash detection to notice it was
gone, a finalizer for a task cancelled before its coroutine ever ran, and a
first-writer-wins guard so a second cancel could not land inside the first one’s
cleanup. Each of those appeared as a review finding, and several of the fixes
produced findings of their own. Meanwhile a restart of seizu still ended every
turn it was running.
ChatTurnWorkflow replaces all of it. It exists, it is addressable by a name
derived from the turn id (workflow_id_for), and Temporal is what guarantees it
reaches an end — so stopping a turn needs no stored handle, liveness needs no
renewable lease, and a turn survives a web-process restart. The price is that
CHAT_ENABLED now implies a reachable Temporal server, the same as scheduled
chats (AGT-007). In Docker Compose the web service therefore waits
for Temporal’s health check before starting; it need not wait for the worker,
because Temporal durably queues an admitted workflow until a worker polls it.
Don’t: add a retry policy. maximum_attempts=1, for the same reason as
AGT-007 — a turn is expensive and not idempotent, and a retry both re-bills it
and appends a second answer to the same log. What running here buys is that the
turn survives its request and always reaches an end, not that it is repeated.
Two paths reach a terminal state, deliberately. The activity finalizes its own turn, including under cancellation — it is ordinary Python and owns the log it has been writing. The workflow finalizes only when the activity never got to: it died with its worker, or timed out. Between them a turn cannot sit at “running” forever, which is what a reader waits on and what keeps the thread from admitting another.
Identity is intersected, never unioned. An interactive turn’s CurrentUser
comes from a live JWT and is not serializable, so the activity rebuilds it with
resolve_stored_user (AGT-006) and intersects the result with
the permission cap stored in the admitted command. resolve_stored_user reads
the last seen role claim, which can be staler or broader than the live token.
Admission is its own request#
POST /chat/threads/{thread_id}/turns answers with a turn_id before anything
streams; GET /chat/turns/{turn_id}/stream reads it. The client therefore holds
the id from the moment it sends.
Why this is a separate request and not the head of the stream. Folding the
two together meant every command about a turn had to be expressed against a
resource that might not exist yet. That produced, in order: a second identity
(client_token) for the window before the first frame; a cancel route that
accepted either name and 422’d on neither; a stop that could beat its own turn
into the store, and so had to create a turn already canceled to claim the
token; and a uniqueness constraint per (thread, client_token) to settle the
race between that create and the real one. Every one of those is deleted by
answering admission first. Seven rounds of review findings on this feature
trace to that single coupling — if a command here is hard to express, check
whether the resource it names exists yet before adding a mechanism.
idempotency_key makes admission repeatable: asking again resolves to the turn
already made rather than starting a second one, so a lost response is fixed by
retrying the same request.
Admission returns an outcome, not an exception. ChatTurnAdmission.outcome
is one of created / existing / busy / retired. It used to raise one of
four errors inferred from whichever constraint rejected the write, so a single
collision could mean any of them and the store had to guess. The store now
re-reads and says which it was.
A store failure during admission is a 503, never an assumption. A failed write means we do not know whether the conversation is being torn down; refusing costs a retry, guessing costs the conversation.
Admission and the session touch are one write. The turn’s half of the
retirement handshake (SBX-011) happens inside
admit_chat_turn. Touching the session first left a window: a delete could read
the fresh timestamp, claim the session, see no running turn and cascade, all
between the two — and the turn was then created against a conversation that no
longer existed.
Don’t: enforce one-running-turn with a read above the insert. The store
says a thread has at most one running turn: a partial unique index
(status = 'running') in PostgreSQL. Under read-committed two requests can both
observe no running turn and both commit, leaving two producers interleaving two
answers into one conversation.
The handoff is repairable, because the store commits first. Admission writes
the turn and then starts the workflow, so a failure in between leaves a turn
recorded as running with nothing producing it: unreadable, and holding the
thread against a successor until its claim lapses. So start_turn ensures the
workflow for existing as well as created, and treats
WorkflowAlreadyStartedError as the outcome it wanted. The workflow id is
derived from the turn id, which is what makes ensuring it twice a no-op rather
than a second producer.
Don’t: ensure it unconditionally. A repeat can arrive after the turn it
names has finished, and a closed workflow’s id is reusable — ensuring it then
would run the whole turn again and bill a second answer into the same log. The
guard is status == "running".
Temporal deduplicates, not the status check above it. That check is a read
followed by an act, so the turn can finish in between — and under the default
ALLOW_DUPLICATE a closed workflow’s id is free again, so the repair would
start the finished turn over. REJECT_DUPLICATE makes the id itself the guard:
a turn’s workflow can exist exactly once, ever.
A cancellation buffer comes off the top of that remainder. Timing a
workflow out does not stop its activity there and then — the cancellation
reaches it on its next heartbeat — so handing it the claim’s full remainder
still lets a producer run past the instant a successor can be admitted. Too
little left for both the work and the stopping means the turn is not started at
all, and is closed rather than left at running: a turn with no producer is
one a client attaches to and waits on until the tail deadline, for an answer
that is never coming. That case is stored with terminal status expired and
has its own admission outcome (expired → 503), so every repeat of the same
idempotency key remains retryable rather than becoming an existing empty log.
That 503 carries X-Seizu-Chat-Admission: expired: unlike an ambiguous store
503, the old key is definitively spent, so the transport retries the same
logical message once under a fresh key instead of attempting to repair it.
The workflow is bounded by what is left of the claim, not by a fresh
duration. A duration cannot express the invariant: the claim is an instant
fixed at admission, while a timeout starts whenever the workflow is created — so
a handoff repaired minutes later restarts the clock and the workflow can run past
a claim a successor has already taken. chat_turn_execution_bound_seconds takes
the turn’s expires_at and returns the remainder; a turn whose claim has already
lapsed is not started at all.
Both deadlines come from one helper. The bound adds half the lease margin
where the lease adds all of it, so it is smaller by construction rather than by
coincidence. Hard-coding it (timeout + 60) while the lease used the
configurable margin left the invariant true only for the settings the test
happened to run under — a test that looked like it protected the property and
did not.
The execution bound covers queue time, not just the activity. A
start_to_close_timeout starts when a worker accepts the task, so a workflow
queued through a worker outage can begin after the turn’s claim on the thread
has lapsed and a successor has been admitted. execution_timeout bounds the
whole thing and is kept strictly under the lease, with a test asserting that
relationship so a settings change cannot quietly reopen it. Belt and braces on
the worker side: the activity re-checks that its turn is still running and
is still the active unexpired turn for the thread before producing, because a
lapsed claim is exactly the case the bound is protecting against.
The admitted command is immutable. The turn record stores the message, continuation/confirmation fields, bypass flag, permission cap, and timeout. Every first handoff or later repair dispatches that stored command; the retrying request is only a lookup by its required idempotency key. This keeps one durable source of truth, makes repair safe across request or permission changes, and removes a parallel request-fingerprint protocol. A client must mint a key per logical send and reuse it for ambiguous retries.
Cancellation does reach the fallback finalizer. Worth recording because it
reads like a bug: the workflow catches Exception, and Temporal cancellation is
often described as arriving as asyncio.CancelledError, which is not one. In
this path it does not — execute_activity raises ActivityError (wrapping
temporalio’s own CancelledError), which is an Exception, whether the
cancel lands while the activity is running or while it is still scheduled.
chat_turn_test.py pins both cases, so a future change that lets a cancelled
turn stay running fails there rather than in production.
The event log#
The producer renders the parts; the reader only replays them. The log holds
the exact JSON the live stream sent, so the first delivery and every replay are
byte-identical and there is no second rendering path that can drift. message_id
and text_id live on the turn record for the same reason — a replay that minted
fresh ids would read to the client as a second assistant message, not the same
one being rebuilt.
A reader stops on two conditions, not one: a terminal status and a cursor
that has reached last_seq. Status alone races the visibility of the final
batches and cuts the answer off mid-sentence, which is why finish_chat_turn
takes last_seq rather than deriving it (that would mean rewriting the metadata
item on every flush).
Don’t: advance a reader’s cursor past a gap in seq. Store propagation is
per item, so a poll can return 5 and 7 without 6; taking the gap loses 6
permanently rather than late. The store truncates a page at the first gap.
Only the stream route is exempt from the request timeout. The exemption is
matched on the path’s shape (/api/v1/chat/turns/{id}/stream) because the id is
in the path — exempting the whole /turns/ subtree would silently drop the
deadline from admission and cancellation too.
Stopping#
Disconnecting no longer stops anything, so POST /chat/turns/{turn_id}/cancel
puts that back. It only marks: it sets a flag on the record and cancels the
workflow. The flag is what reaches a turn running on a worker that never saw the
request, and cancelling the workflow is what interrupts a turn blocked mid-call.
Don’t: make the turn notice a stop only between chunks. It reads the flag on its heartbeat and cancels its own work, because a turn is most likely to be stopped precisely while it is blocked on a slow model call or tool — where no chunk arrives for as long as the call takes, and where letting the call finish first means its side effects happen anyway.
Stop names the turn, not the thread. The request can be delayed or retried,
and by the time it lands the turn it was aimed at may have finished and the user
started another — a thread-addressed stop would then kill the successor. The
client learns the id from admission, or, after a reload, from
GET /chat/threads/{thread_id}/turns/active (204 when the thread is idle, which
the AI SDK maps to “nothing to resume”).
Deleting a conversation closes it first, then stops the turn, then cascades.
Cancelling alone is not enough: the cancelled turn releases its mutex when it
stops, and another tab can start a successor before the cascade runs. The gate
is the reaper’s own claim (claim_chat_session_for_retirement,
SBX-011), which makes admission fail atomically — one
mechanism for “this conversation is going away”, not a second one.
The record is deleted last, after the checkpoint and sandbox, by the same
delete_session_state the reaper uses. The route used to do the opposite and
swallow a cleanup failure behind a 204: the record is what makes a thread
findable, so that left the transcript stored forever with nothing to retry from
and nothing saying so.
Every uncertainty on that path is a 503, never a delete: a failed cancel, a lost claim, or a turn that does not stop within the internal stop deadline. The claim is re-claimable by design, so the session stays closed and the retry is a plain repeat; a conversation half-removed from under a live producer cannot be put back.
Expiry and sweeping#
expires_at on a running turn is a claim on the thread, and admission
retires a lapsed one. It is therefore derived from the turn’s own timeout
(CHAT_TURN_TIMEOUT_SECONDS plus an internal safety margin, in the shared
chat_turn_lease_expiry), not from the replay retention window. On finish
the record is re-stamped with CHAT_TURN_RETENTION_SECONDS, which is a replay
deadline rather than a claim.
This replaced a renewable lease heartbeated by the producer. Renewal existed only because a detached task had no other way to prove it was alive, and it brought three separate correctness rules with it (moving the record and the active pointer in one transaction, re-checking expiry in the takeover update as well as the read above it, and a sweep that could not assume creation order was expiry order). With the workflow bounding the turn, a fixed lease derived from that bound is sound and all three disappear.
Deletion is transactional. Event batches and their turn row are removed in one PostgreSQL transaction. The idempotency key is constrained on that same turn row, so deleting the row cannot leave a separate key that resolves to nothing or permanently blocks a retry.
Note: expired logs are swept at the end of turns, not by a scheduler. A log
belongs to a turn, so delete_chat_session’s cascade is not enough. The
PostgreSQL expires_at index lets a bounded ordered query read only expired
turns, with no scan cursor or renewal path.
Heartbeat on a timer, never on output. A turn is quietest exactly when it is
slowest — a model call or a tool can run for minutes producing no chunk — so
heartbeating from the stream loop times out healthy turns. The damage is not
just a failed turn: the fallback finalizer marks it failed and frees the thread
while the original activity is still running and still writing the same
checkpoint, which is two producers on one conversation. The activity ticks
independently (_CHAT_TURN_HEARTBEAT_INTERVAL_SECONDS); start_to_close_timeout
is what bounds a turn that genuinely runs too long.
A terminal write is conditional on the turn still being running. Two
writers reach finish_chat_turn for one turn: the turn closing itself, and the
workflow’s fallback closing it after the activity timed out. A turn that timed
out spuriously is still alive, so an unconditional write lets the late one
replace a recorded outcome — including its last_seq, which is exactly what a
reader uses to know it has seen the whole answer. First writer wins, enforced by
WHERE status = 'running' in the update itself.
The loser is handed what is recorded, not what it asked for, so it can tell
it lost; None keeps meaning “no such turn”. produce_turn reports the store’s
status rather than its own, so the workflow result and the log a reader sees
cannot disagree.
The client holds a pending send#
Stop is live from submitted, which is before admission answers, so there is a
window where the user can ask to stop a turn the client cannot yet name.
Aborting the admission request does not close that window — the server may
already have admitted and started the turn, which then runs with nobody watching
it. The transport therefore keeps one PendingSend per logical message
(idempotency key, turn id once known, and a stopRequested flag), and admission
is deliberately not given the abort signal. A stop with no id yet is
recorded and applied the instant there is one.
Three rules fall out of that object, and each was a live defect without it:
The key is minted per logical message, not per attempt. A fresh key per attempt puts the server’s idempotency promise out of reach: a retry admits a second turn instead of resolving to the one a lost response already made.
Ambiguous admissions are retried by the transport, with that key. A 503 or a dropped connection means the turn may well have been admitted, and the server’s repair path is reachable only by asking again with the same key — waiting for the user to resend does not work, because their next message gets a new id and therefore a new key, which admits nothing and is told the thread is busy. This retries an idempotent request; the turn still runs at most once (AGT-007). A 409 or 404 is a decision, not ambiguity, and is never retried.
Pending state is a map keyed by thread, not one slot. The transport outlives any one conversation and the sidebar can switch mid-turn, so a single slot means whichever thread acted last owns it and the others silently lose Stop. Reattaching in an idle conversation must clear only its own entry.
Completion is identified by the turn that ended, not by the thread on screen and not by “the stream currently being read”. Several conversations can stream at once, so the latter is not a single thing either — an earlier thread finishing after a later one starts would be attributed to the later one.
clearFinishedTurn(turnId)takes the id off the finished message; a turn that ended without announcing one is left alone, which keeps it stoppable. The producer therefore carriesturn_idin the opening frame’s message metadata; defining the TypeScript field without emitting it leaves every completion unidentified.Unresolved threads are a set. One value means a second ambiguous send in another conversation hides the recovery the first still needs.
Reconnect prefers a turn this client already holds over asking
/active, which answers 204 once the turn has finished — losing a response nobody has rendered yet. That is exactly the case after a retry resolves to a turn that completed while the connection was down.An unresolved send is recoverable from the UI, or the preserved key is unreachable and the repair path is theatre — typing the message again mints a new key and admits a second turn. The banner offers Retry, which replays the stored body under the stored key. First send and retry share one admission path so they cannot drift.
Don’t hold that state only in the transport. It lives on plain objects React cannot observe, so the button never renders and the feature is invisible in the product while testing green against the transport directly (
onUnresolvedChangemirrors it into state).Recovery is rendered from unresolved state, not SDK error state. Changing threads recreates the SDK chat and its transient error, while the transport’s unresolved key deliberately survives. Nesting Retry under
errortherefore hides the only route back to the turn after navigating away and back.The pending send is scoped to its thread. The transport outlives any one conversation and the sidebar can switch sessions mid-turn, so a turn finishing in a thread the user has navigated away from must not clear the pending state of the one they are watching — that silently disarms Stop for the turn that is actually running.
Admission carries its own deadline. It deliberately ignores the SDK’s abort signal, so without one a response that never arrives pins the send forever: the retry loop cannot advance and a stop asked for meanwhile is never delivered, because it is waiting on a turn id. A timeout is ambiguous in exactly the way a 503 is, and retries with the same key.
“Unknown” is not “failed”. A send whose attempts all ended ambiguously keeps its key (
unresolved), because the turn may exist server-side and the key is the only route back to it; a 409 or 404 is a decision and spends it.onFinishfires for errored sends too, so clearing pending state there unconditionally strands exactly the turn the repair path exists to recover.A deferred stop reports through a callback, not a return value. A stop asked for before admission answers is carried out later, by which point
requestStophas returned and throwing from the send would be swallowed by the SDK as an expected abort — so the exact race the deferral exists for would be the one whose failure is invisible. (A promise settled later was tried first and is worse: it deadlocks any caller that awaits it before releasing the step that settles it, and strands unsettled on an admission failure.)A finished turn stops being the pending one. Otherwise a Stop pressed during the next send — before that one is admitted — cancels the turn that already ended and silently does nothing to the live one.
A refused cancel raises. The reader stops either way, so an unchecked 401/403/5xx looks exactly like success while the turn keeps generating and running the actions it had queued.
Testing notes#
A concurrency test here needs real connections. The SQL store’s fixture uses
StaticPool, which hands every session the same connection, so two “concurrent” sessions interleave inside one transaction and the race cannot occur — a broken read-then-write passes, with both callers reporting success even though only one row lands.test_two_concurrent_admissions_cannot_both_winbuilds its own file-backed engine for that reason.These races are easy to write tests around rather than for. Several tests here passed against the broken code before being fixed, so check a new one fails against the version without the fix.
The route tests stand in for Temporal by running the activity inline (
_fake_temporal), so admission, the producer and the reader are exercised together without a worker. The fake records workflow starts in a list, not a dict keyed by workflow id — a second start against the same id is precisely the bug worth catching, and a dict silently overwrites it. (That mistake made the first version of the “don’t re-run a finished turn” test pass with its own fix removed.)CI’s
unitjob blocks network, which catches unmocked store and Temporal calls that pass locally because a real Temporal and Postgres are there to fall into. A locally green backend suite does not prove the mocks are complete.When you scope one piece of state to a thread, check its siblings. Mapping
pendingby thread while leavingstreamingThreadglobal reproduced the same bug one field over, and the test missed it by having the later thread finish — the order a single slot handles correctly. Order the actors so the earlier one finishes last.Drive client behaviour through the path a user takes. Two defects here were masked by tests that called the transport directly: a
clearPendingscoped by an argument the test supplied (so it never exercised the callback that reads the wrong thread), and a Retry that worked in the transport while never rendering a button. If a test supplies the value the bug is about, it cannot see the bug.A resumed SSE stream cannot be fed to the AI SDK mid-message.
createStreamingUIMessageStateinitialisesactiveTextParts: {}even when it reuses the existing assistant message, andtext-deltathrows when that map has no entry — thetext-startwas before the cursor and is not re-sent. Resuming was implemented and reverted for this reason; replaying from the first frame is what works. Note that no test here can catch it: the environment’sfetchhas no streaming body, soprocessResponseStreamis stubbed and the real parser never sees a resumed stream.In the frontend suite,
jest.mockis applied at call time by Bun rather than hoisted, so it cannot replace the superclass of a class that has already been evaluated — mockingaidoes not change whatSeizuChatTransportextends. Stub the instance method instead.
AGT-037 — The answer is written once, from output that keeps its conclusion#
Applies to: chat_orchestrator._keep_ends,
CHAT_ORCHESTRATOR_SYNTHESIS_STEP_MAX_CHARS, _closing_summary_ids,
_init_plan, _PLANNER_PROMPT
A reachability turn’s final answer did not carry the source-level findings its own steps had produced. Two separate causes.
The bound took the wrong end. Each step’s output reached the synthesizer
through an unnamed 4,000-character cut taken from the front. A step’s answer is
at its end — the reachability skill instructs its sub-agent to state the
verdict last — so the cut removed the verdict and kept the working. Measured on
one turn: three reachability steps produced 3,414, 4,383 and 2,772 characters and
only the longest lost its conclusion, which is the shape of a bug rather than of
a budget. _keep_ends keeps both ends and drops the middle, weighted to the
tail.
The bound is now named and larger, and it is not the real limit: the request
is fitted to the model’s window afterwards (CTX-001),
which is what has to hold. This is only a guard against one enormous step
crowding the others — the same relationship
CHAT_ORCHESTRATOR_SYNTHESIS_EVIDENCE_MAX_CHARS has, and that one applies to a
different thing entirely: steps that produced no output and are shown their
tool trace instead.
The plan summarized, then the synthesizer summarized the summary. Plans ended
with an answer step gathering the earlier findings, and the turn’s answer is
already written from every step’s output — so the same summary was produced
twice, losing detail at the extra hop and paying a worker and a verifier call for
it.
Told not to, the planner kept doing it: three of four samples. So it is
removed rather than asked again. _closing_summary_ids drops an answer step
that waits on other steps and that nothing waits on. Narrow deliberately: a
mid-plan decision later steps consume is kept (selecting three CVEs is an answer
step), and a plan that is a single answer step — the request needing no live
action, or reporting that nothing can obtain the evidence — is untouched. The
prompt rule stays, because it tells the planner why; the code is what makes it
hold.
AGT-036 — Package metadata comes from an external MCP, because nothing else can supply it#
Applies to: the external-mcp-deps Compose service
(ghcr.io/mappedsky/depsdevmcp), MCP_EXTERNAL_PROXIES
Seizu’s graph records which package versions are installed — version and a
resolved requirements pin per manifest — and which fall in a CVE’s vulnerable
range. It does not record what a package declares it needs. There is no
property or edge saying botocore 1.42.91 requires urllib3 <2.0, because that is
registry metadata rather than scan output.
A sub-agent asked to judge reachability filled the gap from memory. Read off its own thinking: “botocore pins urllib3 < 1.27 historically, but modern botocore (>=1.29) requires urllib3 >= 1.25.4, < 2.0? … Actually botocore 1.29.0 was released around Dec 2022 and it still required urllib3<1.27… But this is a fictional future scenario (2026 versions)”. It discarded the speculation that time. A security verdict resting on remembered version history is the failure this exists to prevent — AGT-016’s rule about invented identifiers, one layer down.
The sandbox cannot fetch it. Measured: curl to pypi.org and
raw.githubusercontent.com both time out, and getent hosts pypi.org fails — no
egress, not even DNS. So an instruction to “look it up” would have sent it
somewhere it cannot reach, and its discovery clause would then have had it
conclude the data does not exist.
Not a call from the Seizu backend either. Third-party egress belongs in a
component the operator runs, which is what the external MCP proxy mechanism
already is. The tools arrive as ext__deps__depsdev_*, all carry readOnlyHint
so AGT-010’s gating skips confirmation (verified), and they inherit
the rate-limit retry from AGT-029. Only a package name and version
leave the network.
Transport, not tools, was the integration question. depsdevmcp began
stdio-only, and Seizu’s external MCP speaks sse or streamable_http. Teaching
Seizu stdio was rejected: the no-pooling rule (a fresh transport per call per
user, for identity isolation) would make it a process spawn per tool call,
and it would move the egress back into the Seizu backend. A bridge sidecar was
the fallback; upstream adding a streamable-HTTP mode removed the need for either.
Its default listen address is loopback, so the container is given
DEPSDEVMCP_HTTP_ADDRESS=0.0.0.0:8080 or nothing can reach it.
What it gives. Nine read-only tools: declared requirements, resolved
dependency graphs, package and version metadata (advisory keys, licences,
attestations), OSV advisories, project metadata and a Graphviz rendering. It
holds a process-local LRU cache, which matters because a reachability sub-agent
asks repeatedly — measured cached:false then cached:true, 0.36s then 0.16s.
Known gap. There is no chain-shaped answer: nothing returns “does X pull in
Y, and by what path” directly, so a sub-agent asking that fetches the resolved
graph and walks it in run_python. That works and costs a round trip and some
tokens. Worth adding upstream if reachability becomes a common workflow.
Declaring it in a skill is what made it reachable. Under progressive
disclosure a sub-agent only sees what a skill declares, so until one named these
tools nothing directed it to them. cve_response/dependency_provenance covers
the standalone question, and repo_cve_reachability — where the guessing was
measured — carries find_dependency_path and get_requirements plus a step
telling it never to state a pin from memory.
Verified on a full turn: the sub-agent called
ext__deps__depsdev_get_requirements six times, all succeeding in under a
second, in place of recalling a pin. find_dependency_path was not reached in
that run, so the transitive-path tool is exercised only by its own tests so far.
AGT-035 — The planner believed two false things about its own system#
Applies to: mcp_builtins/sandbox.py::_delegate_description, _PLANNER_PROMPT
Read off the planner’s own reasoning (AGT-033’s instrumentation), against the listing AGT-034 gave it. It uses the tool names as intended — “cve_severity_analysis? That gives CVSS distribution, not individual CVE vectors, and not remote. Not enough.” is the plan-time capability check the names were added for. But it held two beliefs that are simply wrong.
“No graph tool listed… all graph tools are only accessible via skills.” Every
delegation is bound SANDBOX_CORE_TOOLS — graph__query, graph__schema,
graph__validate_query, graph__explain (SBX-003) — and
the tool’s description never said so. It said what a delegation is not for
(“cannot be expressed as a Cypher query… prefer those first”), which points the
same way. The description now names the core tools, built from the setting
rather than written out, because the set is configurable and a description that
drifts from it is worse than none.
“It can likely run one skill only?” Every skill is a tool spec on every
worker (_worker_tool_specs), so a step can render as many as it needs. Believing
otherwise, the planner adds a step to reach a second skill — and a dependent step
waits for its parent and runs in a later wave, so the belief costs wall-clock
parallelism rather than tidiness. The prompt now states it, with that consequence
attached.
Measured: no detectable change in plan shape. Before, one of three samples chained findings into reachability; after, one of four. Dispatch waves 3–5 either way. Both corrections stand on being true — a planner reasoning carefully from false premises reaches wrong conclusions, and this is the third change in a row whose value is correctness rather than a number (AGT-034). Recorded so the shape hypothesis is not re-tested from scratch.
AGT-034 — What the planner is told about capabilities, and what it is not#
Applies to: chat_graph.build_capability_context(for_planner=),
_PLANNER_CAPABILITY_HEADER, _format_skills(with_tools=),
_skill_required_tool_names; chat_orchestrator.planner_node
The planner sees each skill’s name, description, trigger phrases and argument
names, plus the always-disclosed tools. It has no tool-calling at all —
_structured_invoke passes an empty tool list — so it names capabilities and the
worker renders them at execution time. Two things followed from that being
implicit rather than stated.
It was told to do something it cannot do. The capability header is shared with the executing agent and is written in the imperative: “call the relevant skill tool”, “if the current user request matches a trigger phrase, call that skill now instead of describing how to trigger it.” The planner has neither the mechanism nor the tools bound. Its own thinking, read off the span: “This is strange: as planner, if task matches trigger, call skill now instead of describing how to trigger it. But the output expected is a plan.” It now gets a header describing the same catalogue as names its steps may use.
Measured, and it did not help: median reasoning 22,810 chars before against
24,262 after, four samples each, ranges 705–34,107 and 17,564–29,692. The
contradiction is real and worth removing on correctness grounds — an instruction
that cannot be followed is a defect — but it is not what the planner’s thinking
is spent on, and this entry exists so nobody re-derives that hypothesis from the
same quotes. Plan shape was unaffected (3/3 plan_probe samples valid, fan-out
preserved).
It could not see what a skill can reach. seizu_tools_required rides on every
skill listing and was dropped by _format_skills. The cost was concrete: one plan
set a step’s success criteria to an installed package version, gave it a
dependency whose skill has no tool returning one, and the step failed three times
for the gap (AGT-032). The planner’s listing now renders those names.
Names, not descriptions. Measured on this deployment: names inline cost 1,847
characters, names plus a de-duplicated description glossary 6,288, a description
at every mention 6,719 — 75 references over 49 distinct tools, so de-duplication
saves almost nothing. Tool names here are descriptive enough to carry the
mapping (github_security__top_vulnerabilities against
github_security__repo_dependencies), which is what the planner lacked. The
planner’s listing only: the executing agent is given a skill’s tools when it
renders one, and its context is the one SBX-003 measured.
Not done: an interactive planner. Letting it call load_seizu_skill was
probed. The mechanism works — it breaks off after the first round and reliably
loads the three relevant skills — and over four samples it is suggestive but not
established: median 87s and 16,150 reasoning characters against a
non-interactive median of ~23,000, with ranges overlapping almost entirely, and
one early sample at 52s that did not survive repetition. Judging it properly
needs the real structured-output path (schema enforcement, budget accounting,
_plan_problems validation) rather than a probe, because the metric that matters
is plan validity and a probe without the schema cannot produce a valid plan. The
objection that stopped it earlier — “two or three calls at planner latency” — is
wrong and recorded here so it is not reused: that latency largely is the
speculation the loading removes.
AGT-033 — What a stage spends thinking is recorded, and graded for the two that answer once#
Applies to: chat_budget.LlmUsage.reasoning_tokens / usage_from_message,
chat_models.applied_reasoning_effort, the llm span in
chat_graph._run_llm_tool_turn and the sub-agent span in sandbox.py;
CHAT_LLM_PLANNER_REASONING_EFFORT, CHAT_LLM_SYNTHESIZER_REASONING_EFFORT
A slow stage looked like a slow model. Spans carried duration, tokens, cost
and finish reason, and on a reasoning model the thinking and the answer come out
of one output_tokens figure — so a planner emitting 6,621 tokens for a plan
whose JSON is under a thousand is indistinguishable from a planner writing a very
long plan. LangChain reports the split as output_token_details.reasoning and
usage_from_message read only the input side. It reads both now.
Measured on the turn that prompted this (1,194s total): planner 110s, 3,385 in and 6,621 out; synthesizer 229s, 9,401 in and 10,084 out for an answer of about 2,400 tokens. 27% of wall clock for 3 of ~70 calls, most of it thinking.
The sub-agent’s thinking is recorded the other way round. It does not
stream – create_react_agent calls the model directly – so its span reads
reasoning_content off the finished message where the outer path accumulates
deltas. Both end up on the span under telemetry.content, which is what makes a
delegation’s deliberation readable at all: it is the largest single consumer of a
turn’s model time (141-144s inside each parallel step of one measured batch) and
was previously visible only as a token count.
The effort is read off the model, not re-resolved. reasoning_effort reaches
the provider through model_kwargs because ChatLiteLLM swallows it as a
constructor argument (AGT-019), so “what this deployment configured”
and “what the provider was told” are different questions, and only the second one
is worth putting in a trace.
The levels are the provider’s, not ours, and they are not all distinct#
Six samples per level on deepseek-v4-pro, median reasoning tokens:
effort |
median |
range |
|---|---|---|
(unset) |
179 |
83–223 |
low |
104 |
63–164 |
medium |
229 |
89–359 |
high |
212 |
87–388 |
Only low separates. That is not noise and not litellm flattening the level:
DeepSeek collapses medium, high and xhigh into one setting and defaults
to it, so three of those four rows are the same configuration sampled three
times. Its distinct levels are low, the collapsed middle, and max.
So the portable four-level vocabulary is a request, not a guarantee, and a
level that reads like a reduction can be identical to the default. Measure the
dimension that moves — reasoning tokens — rather than trusting the name; that is
what this entry’s instrumentation is for, and it is why reasoning_kwargs sends
DeepSeek’s level through extra_body rather than letting litellm map it.
Both stages are set to low, the one level that measurably reduces thinking
on this provider. The risk to watch is plan shape, not truncation:
AGT-019’s one-step-plan failure was an output ceiling of 4,096
leaving no room to think and then emit, which is a different mechanism from
thinking less within an ample ceiling. scripts/plan_probe.py measures shape in
seconds and is the check before assuming low is free.
Note the prompt. Those samples used a trivial one, not a planning call with a full schema and 3,385 tokens of context. They establish what the levels are on this provider; they do not predict the real workload.
AGT-032 — Every attempt records its trace, not only the ones cut short#
Applies to: chat_orchestrator._run_worker_step (the _persist_step_record
branch), _prepare_retries, _RETRY_EVIDENCE_MAX_CHARS
A retry is handed what the previous attempt established: a digest in the prompt, and — where one exists — the path to the whole trace on the sandbox’s disk (AGT-013). The file was written only when the attempt ended early, on budget or on an execution error.
The commonest retry is neither. A step that finished and was then failed by the verifier is the ordinary case, and it is the one whose previous attempt produced the most: a complete result, judged incomplete. It was the single case that wrote no file.
Measured. A reachability step wrote a seven-CVE assessment in 396s and was
rejected — “claims to review 8 findings but only covers 7 CVEs”. Its retry got
_step_evidence(..., 4000): a 4,000-character excerpt cut off exactly where the
per-CVE detail lives. It re-derived the rest in 307s, only 23% cheaper than
the cold attempt, and passed with “all 7 selected CVEs” — the same seven. That
is 26% of the turn spent re-deriving what was already on disk, for a rejection
whose own count moved between attempts.
The branch now runs whenever the attempt made calls. The cost is one small sandbox write per step, against a re-derivation measured in minutes.
AGT-031 — A sub-agent’s tool signature is the tool’s schema, composite types included#
Applies to: mcp_builtins/sandbox.py::_py_type_for, _JSON_SCALAR_TO_PY,
_build_seizu_tools
A sub-agent’s tools are built as pydantic models from each tool’s JSON Schema.
The type map held the three scalars and fell back to str for anything else, so
every array parameter was declared to the sub-agent as a string.
A model told a parameter is a string passes a string. It then failed at the
far end — parameter fields could not be coerced to []string, is string — which
reads as the model getting it wrong and is the schema telling it wrong. Measured
on one reachability turn: 12 of the turn’s 12 tool errors were this, across
search_code, list_commits and get_file_contents, and it was the only thing
failing in that run.
array maps to list[<item type>] (a list of strings when the schema does not
say what it holds, never a string), and object to dict[str, Any]. The fallback
for a genuinely unknown type is still str, but composite types no longer reach
it.
Found by the tool spans (AGT-029), not by reading the code. The failures were ordinary tool results — the sub-agent absorbed each one and carried on — so before those spans existed they left no trace to notice, and the same bug had been running through every delegating turn.
AGT-030 — The call ceiling only ever fires; it never throttles#
Applies to: chat_budget._refresh_mode_locked, derived_call_ceiling;
chat_orchestrator._grant_for / _calls_are_primary;
CHAT_RUN_LLM_CALLS_PER_STEP, CHAT_RUN_UNPRICED_LLM_CALLS_PER_STEP
AGT-024 established what the call ceiling is: an emergency loop guard, where cost bounds spend (AGT-022) and loop detection catches loops (AGT-017). The code did not treat it that way, and a measured run showed all three ways it did not.
It throttled a healthy run. _refresh_mode_locked took the maximum ratio
across tokens, cost and calls against the soft limit. A reachability turn
crossed 0.75 on calls alone — 135 of 176, with 17.6% of its cost budget
spent — and went degraded, which is not cosmetic: optional steps are skipped
outright, and the worker and synthesizer drop to the economy model. A run doing a
lot of productive work was quietly given a worse model and fewer steps. Calls are
now out of the soft-limit calculation entirely; the hard stop is unchanged, and
is the only thing the dimension does.
It was sliced per step. The call grant went through the schedule divisor,
which is AGT-025’s exact finding one dimension over: a backstop
divided by the schedule stops being a backstop. On the measured plan a 176-call
ceiling became 6 calls for a step that needed about 20, and the step stopped on
budget_step_share before its first delegation finished. Calls now take the
batch-width bound that keeps concurrent grants disjoint — unless neither cost nor
tokens is budgeted, in which case the loop guard genuinely is the budget
(_calls_are_primary) and is fair-shared like one.
It was sized at the median, not above the maximum. 24 calls per step was
chosen before a step could delegate. Measured on a delegating turn: 13.4
sub-agent calls per completed step, plus the worker’s own loop, its summary and
the verifier — around 20 for an ordinary step, more with a retry. So the figure
sat at roughly what legitimate work costs, which is the one place a safety limit
must never sit. It is 96 where cost can bind, and CHAT_RUN_UNPRICED_LLM_CALLS_PER_STEP
(24) tightens it where LiteLLM cannot price the model and cost can therefore never
accrue — the same split derived_token_ceiling makes for tokens. Zeroing
CHAT_RUN_LLM_CALLS_PER_STEP still disables the dimension outright; the unpriced
figure only ever lowers a ceiling, never restores one.
Why not simply raise the number. That would be the third re-fit of the same
constant (64 → 120 → 8 + 24n), each correct for the plan in front of it. The
count is not what should be doing this work: cost bounds what progress may cost,
the turn and sandbox timeouts bound how long it may take, and _looks_stuck
bounds going nowhere. The ceiling exists for what those three miss — cheap, fast,
novel-every-time and useless — and should be sized so that reaching it is itself
the evidence.
Worth knowing: _looks_stuck keys on an exact tool-plus-arguments signature,
so 88 distinct-but-fruitless searches pass it untouched. If a per-step “is this
going anywhere” signal is wanted, widening that is the mechanism; a call count
only proxies it.
AGT-029 — A tool call is traced by its outcome, and a named rate limit is waited out#
Applies to: mcp_runtime._guarded, mcp_builtins/sandbox.py sub-agent tool
wrapper; external_mcp.call_tool / _call_tool_once / _rate_limit_delay,
MCP_EXTERNAL_RATE_LIMIT_*
A failing tool was invisible. Spans covered the turn, its batches, its steps
and every model call, and the mcp-python-sdk contributes its own for external
MCP traffic — but nothing recorded whether a call worked. Across one
reachability turn, no span carried an error status or event at all, while the
stream log showed 21 errored tool results. Seizu’s own tools (graph__query,
github_security__*, the sandbox’s five) had no spans of any kind.
They are traced now at the two places every call already passes: _guarded,
which is where a call’s outcome is decided for built-in, user-defined and
external tools alike, and the sub-agent’s tool wrapper, which is the only path
the sandbox’s own five take. outcome is a string — ok, error, or the block
reason — for the same reason stopped_by is on a step span: “it failed” and “it
was refused” are different questions. Error text is content and stays behind
TELEMETRY_RECORD_CONTENT.
What that made visible immediately. Two runs of the same request, believed comparable, were not:
run 1 |
run 2 |
|
|---|---|---|
external |
92 |
89 |
median call |
1,272 ms |
544 ms |
calls over 5s |
5 (max 15.5s, all |
0 |
hard rate-limit refusals |
not recorded |
5 named, 19 errored calls |
One run absorbed GitHub’s code-search limit as throttling, the other as refusals. Do not compare two runs’ cost without checking this; the difference between those two was substantially upstream conditions, not anything in Seizu.
A refusal is an ordinary result, so nobody was waiting. The upstream says
"GitHub API rate limit exceeded. Retry after 44s.", that string reaches the
sub-agent as tool output, and the sub-agent does the only thing it can — call
again, into the same closed window. 13 of 34 code searches were refused this way
in one turn.
call_tool now reads the delay out of the refusal rather than backing off
blindly: the upstream knows when its window resets and says so, and a guess
either wastes the wait or spends the retry on the same refusal. Bounded three
ways, because a sub-agent’s whole delegation is bounded by
SANDBOX_TIMEOUT_SECONDS and one tool call must not spend it asleep — a retry
count, a cap past which the refusal is handed back unretried, and a default only
for a limit that names no delay. A non-rate-limit error is never retried: a 404
retried is a 404.
A non-zero shell exit is output, not a fault. grep exits 1 when it matches
nothing; curl exits 2 on a bad flag. E2B raises for all of them, and letting
that propagate ended the whole delegation — the sub-agent lost every result it
had gathered because one command reported “no match”. Two delegations died this
way in one measured turn. run_bash now returns the exit code with stdout and
stderr, in the same shape a success returns, and the sub-agent decides what a
failure means. The exit code is stated only when non-zero: noise on every
success, and the entire finding on a failure, where (no output) alone reads as
a broken tool.
There were two exceptions doing this, not one. A command outrunning E2B’s
deadline raises TimeoutException, not CommandExitException, and a fix written
from the one traceback that had been seen covered only the exit code — the next
live run lost a delegation to the other. Both are now reported as text. The
timeout says so in words because E2B keeps nothing the command had printed, so
there is no output to return: what the sub-agent can use is the fact, and the
suggestion to narrow the command or background it and read a file.
run_bash_streaming deliberately still raises. Its caller is the remediation
workflow, which runs builds and tests, and there a non-zero exit is the phase
failing — reporting it as output would let a failed build read as a result.
AGT-028 — Trace the sub-agent, and recognise a rejection it has already had#
Applies to: mcp_builtins/sandbox.py::_ToolMessageNormalizingModel.ainvoke;
chat_orchestrator._same_rejection, _SAME_REJECTION_SIMILARITY,
_SAME_REJECTION_MIN_TOKENS, _prepare_retries
Two defects found by running a delegating turn against a trace (AGT-026).
The sub-agent was untraced. chat_graph._run_llm_tool_turn carries the span
for a model call, and a sandbox sub-agent never reaches it — it runs on
create_react_agent, whose calls go straight to the model. Measured on one
reachability turn: 51 traced llm spans against 107 calls in the ledger, the
missing 56 being every sub-agent call — 52% of the run’s calls and about 60% of
its cost, invisible in the trace while fully present in the budget. That makes
the one ratio worth watching on a delegating turn — cost per step that completed
— uncomputable from traces. The span now sits where the budget reservation
already sits, which the code says is the one place every inner call passes.
A rejection restated in new words was not recognised as the same one. The
guard added by AGT-017 compared a verdict to the previous one with
==, and a verifier writes its verdict fresh every time. A step was observed
failing three times for the one thing its dependency could not supply, each
rejection worded differently, each one passing the guard as new.
Measured, on the verdicts that run actually produced (Jaccard over content words):
pair |
overlap |
|---|---|
the two rejections of the same step, reworded |
0.500 |
two different “incomplete” complaints |
0.103 |
unrelated rejections |
0.022 – 0.023 |
0.4 sits in a five-fold gap. The length floor is not decoration: at three
or four content words one shared word carries the ratio, and the first version of
this scored two plainly different terse rejections at exactly 0.400 — caught by
an existing test, not by inspection. Below _SAME_REJECTION_MIN_TOKENS the
comparison stays exact.
Direction of error. A false positive ends a step that a retry might have saved; a false negative spends another attempt. AGT-017’s finding is that the second is the one that actually happens, so the threshold is set to catch the observed repeat rather than to be safe against every conceivable one.
AGT-027 — Fan out over divergent work, and seed what a batch will all want#
Applies to: _PLANNER_PROMPT, _PlannedStep.map_over / map_reason,
chat_orchestrator._seed_shared_schema, _dispatch_batch_distributed;
mcp_builtins.sandbox_result_dir
Expansion (AGT-023) fans a step out over the items an earlier step found. What it cannot tell on its own is whether those items need different work or are the same call with a different argument, and the planner prompt used to push it one way: “never write a step that loops over a collection internally”. That is right for an agentic loop and wrong for a data one.
Measured, on a request to review one repository’s CVEs. The planner mapped
“query the graph for the repositories and packages carrying this CVE” over eight
CVEs. That is one Cypher query with an IN list. The fan-out took 70 of the
run’s 80 LLM calls and 219,049 of its 297,017 tokens, and returned nothing —
the answer was written from the un-mapped first step.
So the planner is told the test, not a preference. Same call with a
different argument is one step that fetches everything at once; its own tools,
or a decision that depends on what the item turns out to be, is a mapped step.
map_over is paired with map_reason — a planner that has to name what differs
between items writes fewer fan-outs — and the reason rides the chat expand step span, so a wide trace says whether the width was earned.
Measured after, on the same build. The CVE request planned two steps and no
fan-out (0 of 3 samples), finishing in 263s on 18 calls and $0.020 against 433s,
80 calls and $0.044 — with a better answer, covering all 19 findings where the
old one had reported a gap. A reachability request over three CVEs still fanned
out (2 of 3 samples), map_reason: “Each selected CVE can belong to a different
repository and needs its own repository CVE finding call with that repo and CVE
id.” The discrimination runs both ways, which is the whole point.
The batch, not the carry, is what made eight sub-agents fetch one schema#
Each of those eight children opened by fetching the same 52,846-byte schema, writing it to its own file on the disk they were already sharing, and reading it back — two of the four calls each could afford, before reaching its question.
The obvious fix was already built, and was not the fix. Handing a step the
files its dependency saved duplicates two existing mechanisms: EpisodeLog.recall
already advertises a sandbox’s receipts to a sub-agent, and session_digest
already puts the same manifest in the worker’s prompt for the model deciding
whether to delegate at all (SBX-008). A parallel path was
built, measured against them, and reverted — it emitted a near-identical
second copy of the same list, and unlike render_receipts it did not filter on
sandbox_id, so the one case where it said anything new was the case where the
file was not readable.
The steps of a batch start together, and a receipt reaches a sibling only
once the batch it was written in has returned. No carry can close that, because
there is nothing yet to carry. _seed_shared_schema fetches the schema once
before a distributed batch of two or more, writes it to a fixed path under
sandbox_result_dir(), and records the receipt — after _shared_sandbox_id has
opened the sandbox and before the ledger is serialized for the workers, or
the batch it was fetched for is the one batch that cannot see it.
Bounded deliberately: batches of one are skipped (a lone step’s delegations
already carry it to each other), it never opens a sandbox that SBX-015 did not
already open, it re-uses the file across a turn’s later batches, and it is
skipped for a caller without query:execute so seeding cannot hand anyone data
they could not have asked for. Every failure path is a warning and a return.
Not done: seeding anything else. The schema earns it by being the one thing every sub-agent wants, identical for all of them, and the largest single result the graph returns; nothing else on that list is obvious.
AGT-026 — Tracing is diagnosis, opt-in, and content-free by default#
Applies to: reporting/services/telemetry.py, chat_graph._run_llm_tool_turn,
chat_orchestrator._dispatch_batch / _run_worker_step_with_session /
_expand_mapped_steps, the Temporal client and worker;
TELEMETRY_*
Spans cover the turn, its dispatch batches, each plan step, each expansion and
every model call, exported over OTLP to any collector. Off unless
TELEMETRY_OTLP_ENDPOINT is set.
Why it needs the Temporal interceptor to be worth anything. A turn spans at
least three processes — the web service admits it, one worker activity drives
it, and its plan steps run as further activities that may land on other replicas
(AGT-018). Without propagated context those are unrelated trees and
the only question worth asking, where did this turn spend its time, has no
answer. temporalio.contrib.opentelemetry.TracingInterceptor on both the client
and the worker is what joins them.
What it answered on the first traced run, a 208-second turn:
share |
|
|---|---|
|
54% — 112.5s |
both dispatch batches (five steps, parallel) |
30% |
|
12% |
every worker and verifier call |
12% |
One planner call cost more than all the parallel step work put together. That had been invisible: the planner is a single structured call with no step of its own, and the whole of AGT-023’s tuning went into the steps. It is also the stage still on the reasoning model at adaptive effort.
Content is opt-in. telemetry.content() returns "" unless
TELEMETRY_RECORD_CONTENT is set. A trace of this system carries graph rows,
tool output and the user’s own words, and exporting it sends them wherever the
collector is — a different decision from wanting timings. Exceptions are
recorded by type for the same reason: an exception string here routinely
carries tool output.
Never load-bearing. configure() swallows its own failures, a span is a
no-op context manager when tracing is off, and budget decisions read the ledger
(chat_budget) rather than any of this. A run must spend the same whether or
not the collector is reachable.
The local collector has its own Compose toggle. make otel_enable selects
the local endpoint and enables telemetry; make otel_disable clears that
endpoint and deselects the collector. OTEL_COLLECTOR_ENABLED controls the
tracing profile independently of TELEMETRY_ENABLED, because a remote OTLP
endpoint does not need a local collector. Disable preserves remote endpoints,
and make down includes all Compose profiles even when their toggles are
already off; otherwise the documented disable-then-restart sequence leaves
optional services running. This also covers external MCP and auth containers.
Not metrics. Step ids are content-derived (s2-cve-2026-44432-61daf8), which
is fine as a span attribute and unusable as a metric label; anything wanting
aggregate counters should derive them in the backend.
AGT-024 — The call ceiling is derived from the plan, not configured#
Applies to: chat_budget.derived_call_ceiling,
BudgetController.set_planned_steps, _refresh_remaining_estimate;
CHAT_RUN_MAX_LLM_CALLS, CHAT_RUN_LLM_CALLS_PER_STEP
CHAT_RUN_MAX_LLM_CALLS defaults to 0, meaning derive: a ceiling of
8 + CHAT_RUN_LLM_CALLS_PER_STEP x steps, recomputed wherever the plan changes.
A positive value pins it; zeroing both it and the per-step figure disables the
dimension.
Why a constant is wrong here specifically. The call ceiling is an emergency loop guard — what a run may spend is bounded by cost and tokens (AGT-022), and a loop is caught by loop detection (AGT-017). A fixed count does not bound spend or detect loops; what it actually bounds is plan size, wearing a safety limit’s clothing. 64 was chosen when a plan was at most eight steps, and it silently became “at most about five steps’ worth of work” the moment a step could expand into eight (AGT-023).
Measured: the expansion run in AGT-023 ended exhausted on the call ceiling
at 120 calls, having spent 28% of its cost budget — the same failure shape as
AGT-019’s output ceiling and AGT-021’s reservation sizing, one level
up: an expensive, productive run stopped by a number that had no relationship to
what it was doing.
Only ever raises. set_planned_steps never lowers the ceiling. A plan that
shrinks — steps skipped by the budget sweep, a retry cycle that drops one — would
otherwise pull the ceiling below what the run had already spent and finalize it
retroactively.
The per-step figure is deliberately generous (24). It covers a step’s own loop, whatever it delegates to, its summary pass and the verifier’s look at it, across retries; and since every one of those calls is already bounded by the step’s share of cost and tokens, a high count cannot translate into high spend. The number only has to be above what legitimate work does and below what a run making calls without spending or progressing would reach.
AGT-023 — A step maps over what an earlier step discovered#
Applies to: chat_orchestrator._PlannedStep.map_over, _MapItems,
_expand_mapped_steps, _child_step_id, _dependency_satisfied,
_runnable_steps, _step_contract; CHAT_ORCHESTRATOR_MAX_EXPANSION
A planned step may declare map_over: <dependency id>. When that dependency
passes, the dispatcher extracts the items from its output and replaces the step
with one step per item; the parent takes the status expanded and never runs.
The children are ordinary steps, so the ready set, the fan-out
(AGT-018) and the retry cycle carry them with no changes.
Why the planner cannot do this. planner_node runs once, before anything
executes, so it can only fan out over items the request names. Measured on
deepseek-v4-pro, same graph and session: “investigate these four CVEs: …”
planned 5-7 steps, 4 wide; “find the four highest-severity CVEs, then
investigate each” planned 1 step. The second is the shape the work actually
has, and the planner was right — the ids did not exist yet. The cost of being
right was that one step rendered a skill five times and made 57 tool calls in
sequence, at the one level where nothing could parallelise it.
Ids are derived from the item, never from position. _child_step_id slugs
the item and appends a digest of it. The fan-out’s idempotency key is built from
the step ids in a batch, which is what stops a repeat from paying for the same
batch twice (AGT-018); ids minted from list order or a counter would
move when the model returns the same items in a different order, and the
guarantee would quietly stop holding.
The budget re-allocates itself. _budget_divisor and _remaining_waves
(AGT-020) are computed from the live plan every time a batch is
granted, so children are budgeted as the steps they are the moment they join it.
Nothing subdivides the parent’s slice, because the parent never had one — it had
not run. An expanded step is excluded from the outstanding set, so it is not
counted as work the run still has to pay for.
Expansion is bounded where the items arrive, not by trusting the model:
CHAT_ORCHESTRATOR_MAX_EXPANSION (default 8) cuts the list, and the run records
what it dropped in run_errors so a partial sweep is stated rather than implied.
Items are deduplicated first — two labels for one thing would run the work twice
and charge for both. 0 disables expansion entirely.
A step that cannot be expanded runs as written. No items, or a failed
extraction, clears map_over and leaves the step pending. The alternative — a
step that disappears because its collection came back empty — loses work the plan
said was necessary, and the failure would be invisible.
A dependent of an expanded step waits for every child
(_dependency_satisfied). The parent’s passed never arrives, so without this a
synthesis step that depends on it would either block forever or, worse, run with
nothing.
Each child carries its item, not the collection. The child’s goal names the
item, its contract says siblings cover the rest, and its depends_on drops the
step it was mapped over — so it is not handed, and does not pay for, the whole
list it was sliced from. Its success_criteria is rewritten for the item as
well: the verifier judges a step against that text, and the parent’s was
written for the collection, so a child covering one CVE was failed for “not
covering all four” and the run reported partial with every step complete.
Retry works per child. Children are ordinary steps: _prepare_retries resets
a failed one and the siblings that passed are not re-run, while the parent stays
expanded and is never expanded a second time (it is no longer pending).
A half-expanded plan needs no rule of its own. Its children are pending, so
_has_pending_plan and AGT-011’s discard-unless-resumed treat it
exactly like any other unfinished plan.
Mapping over an already-expanded step chains 1:1. The planner reliably
produces this shape — find the CVEs, then per CVE confirm the finding, then per
CVE judge reachability — so a map_over whose source is expanded takes that
source’s children as its items: no extraction call (the items are already
steps), and each child depends on its own counterpart rather than on the whole
previous stage, so s3-for-X starts as soon as s2-for-X passes. Without this
the second mapped step degrades to one step looping internally, which is the
thing the construct exists to remove.
Measured#
scripts/plan_probe.py on the discovery-shaped request (“find the four
highest-severity CVEs, then investigate each one separately”) now plans a
map_over step in both samples, where it previously planned one step. A full
headless turn on the same request:
expansions: s2 -> 4 (CVE-2017-12791:…, CVE-2017-14695:…, CVE-2017-7893:…, CVE-2019-17361:…)
s3 -> 4 (chained 1:1 off s2's children)
ready sets: [s1] [s1] [4 x s2-child] [1 x s2-child + 3 x s3-child]
widest ready set: 4
Four wide — the same width the issue measured for the enumerated-ids baseline,
now reached without the ids appearing in the request. The last batch mixes a
first-stage child with three second-stage ones, which is the per-item edge
working: s3-for-X started while s2-for-Y was still running.
The call ceiling is what binds now — which is why it became derived
(AGT-024): that run ended exhausted on CHAT_RUN_MAX_LLM_CALLS
having spent 28% of its cost budget, and expansion multiplies calls by design.
What a wide plan actually costs, once it is allowed to finish#
A complete three-stage expansion (15 steps, four CVEs, every step producing
output) on deepseek-v4-pro for planning and synthesis and
deepseek-v4-flash at reasoning_effort=low for the per-step stages:
22 minutes, 274 calls, $0.18, batches of four starting together, step
durations 39-320s.
The same request with the reasoning model on every stage and
CHAT_ORCHESTRATOR_MAX_PARALLEL=3 did not finish in 42 minutes: steps ran
317-600s, three hit the 600s step timeout, and a stage of four children was
split into two sequential batches. Two things were wrong and both mattered —
adaptive reasoning on work that is mostly tool-calling, and a concurrency width
narrower than the expansion it had to carry. The per-step stages are the ones to
put on a fast model: they are the ones expansion multiplies.
Not done: a general map/reduce planner, and reduce steps other than an ordinary step that depends on the expanded one. This is one construct.
AGT-022 — A run is budgeted in cost; tokens are the backstop#
Applies to: chat_budget.BudgetController.open_scope / scope_exhausted /
scope_cost_spend / remaining_normal_cost_usd / _authorize_locked,
chat_orchestrator._StepThresholds / _step_thresholds / _grant_for /
_dimension_share, chat_step_worker; CHAT_RUN_COST_BUDGET_USD,
CHAT_RUN_TOKEN_BUDGET
CHAT_RUN_COST_BUDGET_USD is the limit an operator tunes. It bounds the run,
and a share of it bounds each plan step. CHAT_RUN_TOKEN_BUDGET is the backstop
for a model LiteLLM cannot price, defaulted high (2,000,000) so that cost
normally binds first on a model it can.
Why cost is the right denomination. Tokens vary about a hundredfold in price
across the models LiteLLM reaches, and a run that crosses its soft limit
switches to CHAT_LLM_ECONOMY_MODEL and so changes its own token price
part-way through — a token ceiling therefore bounds spend only for one model at
a time, and has to be retuned for every model change. Dollars are what a
“runaway” actually is.
Measured, on the run that prompted this (AGT-021, run B): a four-step turn exhausted its 400,000-token budget at 88% while spending 5.6% of the cost limit — $0.056 of $1.00. The token ceiling was not protecting anything; it was ending useful work at an arbitrary point that happened to be denominated in the wrong unit.
Enabling cost was not enough on its own. Per-step fair-share
(_step_thresholds, open_scope) was token-only, so raising the token ceiling
to let cost bind would have removed step-level bounding entirely: one step could
spend the whole run’s money while every token check passed. A scope therefore
now carries a cost ceiling and cost spend alongside the token pair, and
scope_exhausted / scope_soft_limit_reached are true when either binds.
The worker loop asks the controller rather than comparing tokens itself.
A dimension left at zero does not bound. _step_thresholds used to return
the complexity floor when remaining_normal_tokens was None, which meant a
run deliberately budgeted on cost had every step capped at a guess made before
any work happened (the thing AGT-017 removed). It now returns no
token bound in that case and lets cost do the work; the floor applies only when
neither dimension is budgeted.
The same schedule divisor everywhere. Cost and call grants for a distributed
step were an equal split by batch size while tokens used the wave-aware divisor
(AGT-020); _dimension_share now takes the divisor, so all three
dimensions slice a bottleneck the same way.
What this does not change. The distributed grant is still a hard cut (AGT-018) — it now carries a cost ceiling as well as a token one. Contention on the cost dimension waits exactly as it does on tokens (AGT-021).
The backstop is derived, because a fixed one can never let cost bind#
CHAT_RUN_TOKEN_BUDGET defaults to 0 = derive: no token ceiling at all
when a cost budget is set and litellm can price the model, and
CHAT_RUN_UNPRICED_TOKEN_BUDGET when it cannot. A positive value pins it;
zeroing both it and the cost budget is an explicit “no limit” and is respected.
Why a fixed backstop cannot work. Measured on two real turns: a run cost $0.126 per million tokens, so the $2.00 cost budget is worth ~16M tokens and a 2,000,000-token backstop is eight times tighter than the budget it backs up. It ended the run at 10% of the cost limit. On a frontier model the same 2M figure would be ~$30 — five times too loose. There is no single number that is a backstop for both, which is the same argument as AGT-019 for output ceilings and AGT-024 for the call ceiling: a constant that has to track a model’s properties belongs to the model, not to configuration.
Pricing is asked, not assumed. ModelCapability.priced comes from litellm’s
input_cost_per_token/output_cost_per_token. If it says priced and the price
turns out to be zero at runtime, the run keeps the derived call ceiling
(AGT-024) and each step keeps its cost share, but nothing bounds total tokens —
the case for setting CHAT_RUN_TOKEN_BUDGET explicitly on a gateway that
proxies a priced model under a private name.
Verified on a real turn#
The same four-step request, run with a deliberately tight cost budget ($0.03) and the 2,000,000-token backstop:
status : budget_exhausted
reason : Step worker:s4 reached its share of the run cost budget ($0.0056).
cost : $0.0290 / $0.03 tokens : 89,084 / 2,000,000
per-step ceilings: tokens=529,584 cost=$0.0056 (soft $0.0019)
Every worker scope was handed both ceilings, cost was what stopped a step, and the run ended on cost at 4.5% of its token backstop — the inverse of AGT-021’s run B, which died on tokens at 5.6% of its cost limit. The run still delivered a 3,397-character partial answer, so the finalization reserve behaved as AGT-012 requires.
AGT-020 — The plan is a validated DAG, and an invalid one is replanned once#
Applies to: chat_orchestrator._Plan / _PlannedStep, _plan_problems,
_init_plan, _truncate_plan, _replan_invalid_graph,
_fail_unreachable_steps, _remaining_waves / _budget_divisor;
CHAT_ORCHESTRATOR_MAX_STEPS, CHAT_ORCHESTRATOR_MAX_PARALLEL
depends_on is the edge set of a directed acyclic graph. The contract is stated
in the planner’s schema (the field description and the _Plan docstring,
both of which travel to the model as JSON Schema) and in the prompt, and it is
checked on the way in: unique non-empty ids, every edge naming a different step
of the same plan, no self-edges, no cycles, and nothing left waiting on one.
Why validate at all: a cycle failed silently and answered anyway. _init_plan
dropped dangling references and self-edges, so those were survivable. A
multi-node cycle (s1 -> s2 -> s1) was not: _runnable_steps waits for every
dependency to reach passed, so no step is ever runnable, route_from_dispatcher
finds nothing in ran and goes to the synthesizer, and the turn produces a
confident answer from zero evidence with no error anywhere. Same failure
class as the planner fallback in AGT-019:
the expensive machinery reports success while doing nothing.
Reject and replan, rather than repair first. A repair is not neutral —
dropping the edge that closes a cycle invents an execution order nobody chose,
and dropping a dangling reference hands a step none of the data its goal assumes
it has. So _plan_problems reports, and the planner is asked once more, shown
its own graph and what is wrong with it. Once, not in a loop: a second planner
call is a second paid call against the turn’s budget, and a model that cannot
honour the contract when shown its broken graph will not honour it on the fourth
attempt.
Repair is the floor, not the single-step fallback. If the replan is also
invalid, _init_plan forces a DAG — dangling and self edges dropped, duplicate
ids renamed rather than dropped (the step is real work; an edge naming the id
binds to the first claimant either way), and cycles cut by freeing their
earliest member in plan order. Falling back to one step instead would throw
away every step the planner wrote in order to punish an edge. The single-step
fallback stays where it was: for a planner that returned nothing usable at all.
Truncation is our edit, so it is repaired, not replanned. Cutting the plan at
CHAT_ORCHESTRATOR_MAX_STEPS orphans any edge pointing into the tail;
_truncate_plan removes those edges itself, so the cut does not present as the
planner having emitted a dangling reference and cost a replan.
An invalid plan never looks like a valid empty one. Every diagnostic goes to
run_errors (persisted in the assistant message’s metadata) and into the plan
detail’s body on the stream. And when the dispatcher finds nothing runnable with
nothing in flight, _fail_unreachable_steps fails each still-pending step with
the dependency that stopped short, so it reaches _terminal_errors instead of
disappearing from the plan the synthesizer answers from.
Budget slicing had to move from breadth to depth. _grant_for and
_step_thresholds divided what the run has left by the number of outstanding
steps. That is the right denominator only for a chain or a single wide batch,
where the step count and the schedule agree. A DAG breaks it: at a bottleneck one
step runs alone while five wait behind it, and dividing by six starves the only
step that is running — the steps behind it will divide their own remainder among
themselves concurrently when their turn comes. _budget_divisor divides by
remaining waves × the width in flight instead: each remaining dispatcher pass
gets an equal share, and the steps within a pass split it. For a chain (n
waves of one) and for a flat batch (one wave of n) this is exactly the old
number; it differs only where the graph has depth.
MAX_PARALLEL matches MAX_EXPANSION (8). It was 3, on the reasoning that a
wider batch made every concurrent step poorer — true while the token budget was
fair-shared by width. It is not true now: tokens keep only the safety bound and
cost is what steps share (AGT-025), and a step’s cost slice is far
from binding. Measured against it: a stage of four expanded children ran as two
sequential batches, so the stage cost twice its slowest step — about ten
minutes — purely because the setting was narrower than the expansion it had to
carry.
Measure shape with scripts/plan_probe.py, which now prints the dispatch
wave count beside the widest independent batch — the depth the budget divisor
keys on. Measured on deepseek/deepseek-v4-pro with the DAG contract in place:
a chained request (“find the highest-risk CVE, then trace it, then write it
up”) planned 5–6 steps one wave wide across three samples, and a request with
independent parts planned 4–5 steps three wide, fanning into one answer step, in
both samples. Neither shape produced an invalid graph, so the replan path costs
nothing on the normal case. The second shape is exactly where the divisor
changed: its lone first step is granted a third of the run rather than a fifth.
AGT-019 — What a call may spend is derived from the model, never a constant#
Applies to: reporting/services/chat_models.py, chat_graph.build_chat_model
/ get_chat_model, chat_context.max_output_tokens,
chat_orchestrator._structured_invoke; CHAT_LLM_MAX_TOKENS,
CHAT_LLM_MAX_OUTPUT_TOKENS_CAP, CHAT_LLM_*_REASONING_EFFORT,
CHAT_ORCHESTRATOR_PLANNER_MAX_TOKENS
A ModelSpec — model id, output ceiling, reasoning effort — is resolved once per
call and passed down. The ceiling is min(cap, the model's own max_output_tokens) read from litellm, not a configured constant.
Why: a constant here fails silently and catastrophically. On a reasoning
model the thinking and the answer come out of one allowance, so a ceiling
that is too low does not produce a shorter answer — it produces no answer, and
nothing distinguishes that from a model that could not satisfy the request.
Measured on deepseek-v4-pro at the old CHAT_LLM_MAX_TOKENS=4096: every
planner call returned chars=0, finish_reason=length twice, and planner_node
fell back to a single step carrying the user’s whole request.
That fallback is invisible everywhere downstream — identical in the stream, the
checkpoint and the harness (steps_total: 1), with run_errors the only record
— and it disables the orchestrator’s parallelism entirely, because
parallelism (in-process and distributed, AGT-018) operates on
independent plan steps and there was only ever one. So a feature can be
correct, tested, and completely unreachable. Measured, 3 samples per arm:
|
real plans |
widest batch |
|---|---|---|
4,096 |
0/3 |
1 |
16,384 |
3/3 |
4 |
32,768 |
3/3 |
4 |
The planner emits 5,798–13,058 output tokens per call on this model, so 4,096 could never have worked: the plan and the reasoning that produces it do not fit.
A constant is wrong in both directions at once. The models we run report
ceilings from 16,384 (gpt-4o) to 393,216 (deepseek-v4-pro). A constant large
enough for the second is refused outright by the first — providers reject an
over-ceiling request rather than reducing it. Only derivation is right for both,
and it needs no per-model configuration.
0 means derive. CHAT_LLM_MAX_TOKENS and
CHAT_ORCHESTRATOR_PLANNER_MAX_TOKENS both default to 0 and are still honoured
when set — and still clamped to the provider’s ceiling.
Don’t: reintroduce a default output size at a call site.
_structured_invoke defaulted to 1,024, which is why the router and verifier
sat one hard question away from the same silent failure the planner hit.
reasoning_effort is the intent; the provider’s own parameter is what ships#
max_tokens is genuinely portable — litellm renames it to
max_completion_tokens for OpenAI on its own. Effort is not. Its mapping is
lossy on half the providers we run: every level collapses to
thinking: {"type": "enabled"} on DeepSeek and {"type": "adaptive"} on
Anthropic, so a “high” and a “minimal” arrive identical. That is why sweeping
effort measured nothing on DeepSeek — there was no dial attached.
Graded control does exist on all four, so chat_models.reasoning_kwargs renders
the level into whatever that provider grades on:
Provider |
Rendered from |
|---|---|
OpenAI |
|
Gemini |
|
Anthropic |
|
DeepSeek |
|
budget_tokens is a share of the call’s own ceiling, not a constant, so one
profile means the same thing on a 16k model and a 393k one — floored at
Anthropic’s 1024 minimum and capped at half the ceiling, because thinking and the
answer come out of the same allowance.
This is provider-specific code, which the previous revision of this entry argued against. The argument was wrong: keeping only the portable parameter did not avoid provider knowledge, it silently discarded control. The knowledge lives in the one resolver that already knows the model, so no call site learns a provider name.
Two generation parameters are refused, and neither says so#
OpenAI reasoning models reject temperature outright. With
litellm.drop_params False that raises, so CHAT_LLM_TEMPERATURE=0.2 fails
every call on gpt-5 and o3. There is no capability flag for it —
get_supported_openai_params reports temperature as supported for gpt-5,
which then refuses it — so chat_models asks litellm’s parameter transform
directly and caches the answer.
Anthropic fixes temperature at 1 once extended thinking is on, and litellm
does not strip a different value. temperature_for therefore returns 1.0 for
that combination and None where a temperature may not be sent at all.
Both are the same shape as the ceiling bug: a flat generation constant applied to models that do not all accept it.
Only the router has a measured default, and only because of its shape#
CHAT_LLM_ROUTER_REASONING_EFFORT defaults to none. Measured across 21 cases
on deepseek-v4-pro: routing stayed 21/21 correct while median output
halved, 78 → 37 tokens, on every turn.
The planner looks like a far bigger win and is not takeable on that evidence. With reasoning off it used 7.7x fewer tokens (8,439 → 1,093) and was 12x faster, at the same plan width. But reading the plans showed they are not equivalent: with reasoning the planner produced a shared up-front fetch step that the four per-CVE steps depended on; without it, four independent steps each re-derived that data. Width is a structural metric and hid this entirely.
The asymmetry is the point, and it generalizes. A stage whose output is a single value can be measured by a cheap probe; a stage whose output is the structure of everything downstream cannot. The router’s reasoning can change nothing but one label. The planner’s changes what every worker then does, so only an end-to-end run can price it — a saving at one call that adds work to dozens is not a saving.
Measured end to end, the planner saving reverses. Three samples per arm on the same conversation: turning planner reasoning off cost more overall – median 1,107k tokens against 846k, and $0.329 against $0.271 – despite saving 7,300 tokens at the planner call. Exactly the mechanism the plans predicted: four steps re-deriving the data the shared prefetch would have fetched once, and the workers are where the tokens are. (Ranges overlap at n=3, so read this as directionally confirmed rather than settled; the mechanism is what makes it credible, not the sample size.)
It is faster – 684s against 1,019s – because those four steps start immediately instead of serializing behind the prefetch. So it is a real cost-versus-latency trade, and a deployment that wants responsiveness over spend can take it deliberately. It is not a free win, which is how the probe made it look.
Don’t: convert the per-stage rationale table into defaults. It is where to look, not what to set.
Effort keys on stage, not only role#
A worker’s ReAct loop is deciding what to do next; its summary pass is writing
down what the step already established, and every “reasoning ate the
allowance” failure in this codebase is in the latter. Both used to resolve
through get_chat_model("worker", …), so the distinction could not be
expressed. worker_summary and worker_summary_retry are now stages that run
on the worker’s model with their own effort, inheriting the worker’s when unset.
It must go through model_kwargs. ChatLiteLLM does not declare
reasoning_effort, so passing it as a constructor argument is silently
swallowed — no attribute, absent from model_kwargs, absent from
_default_params. It shipped that way first, and every measurement taken
against it was measuring nothing.
litellm.drop_params is False, so an unsupported parameter raises rather
than being dropped. That makes the supports_reasoning gate in
chat_models.resolve load-bearing rather than tidy.
Effort is per role because the stages want opposite things in principle: reasoning is what decomposition and judgment (planner, verifier) are for, while classification (router) and transcription (worker summaries, synthesis) only lose answer allowance to it.
Every per-role default is empty, because on this deployment the knob does
nothing measurable. LiteLLM collapses minimal/low/medium/high to a
single value on DeepSeek (thinking: {"type": "enabled"}) and Anthropic
({"type": "adaptive"}) — only OpenAI and Gemini are genuinely graded. Measured
on DeepSeek with the parameter actually reaching the wire: the router routes
21/21 correctly at every level with ~70 output tokens either way, and the
planner produces a width-4 plan 3/3 at every level. Turning reasoning off
(none) did not help either — it produced more output (11k vs 6.5k median),
because the model writes its thinking into the visible answer instead.
So the recommended-values table in the install docs is guidance for graded providers, not a measured win. Do not ship it as a default without measuring on the provider in question.
Reasoning has to survive back into the next request#
A tool-calling assistant message must be replayed with its reasoning intact, and
the shape differs: DeepSeek needs reasoning_content, Anthropic extended
thinking needs the signed thinking_blocks, and a tool-use turn replayed
without them is rejected. _strip_reasoning_context flattens list content to
plain text, which would destroy Anthropic’s blocks, so both shapes are preserved
in additional_kwargs — where litellm reads them
(prompt_templates/factory.py) and where the flattening cannot reach.
Anthropic + reasoning + tool loops is unverified. This deployment runs DeepSeek, so that path is written to what litellm reads rather than to an observed failure. Verify it against a real Anthropic key before recommending reasoning on Anthropic.
The spec is the cache key, and it travels#
build_chat_model is memoized on the spec, not on (role, economy). That
matters ahead of user-selected models: a role-keyed cache would hand one user’s
chosen model to another in the same process.
For the same reason a distributed plan step carries the resolved spec
(AGT-018) instead of the economy: bool it started with. A step must
run on the model its turn was admitted with; re-resolving worker-side reads
that worker’s settings and can produce a different model — the failure
permission_cap travelling already prevents (AGT-006).
Measuring this needs scripts/plan_probe.py, which runs the planner alone —
one LLM call, nothing executed — and reports the widest independent batch plus
whether the plan was the fallback. A full harness turn costs ~553s and ~$0.20 to
learn the same integer.
AGT-018 — A plan’s independent steps are Temporal activities, not coroutines#
Applies to: reporting/temporal_workflows/chat_step_fanout.py,
reporting/services/chat_step_worker.py,
chat_orchestrator._dispatch_batch_distributed / _distribution_eligible /
_grant_for, chat_budget.grant_ledger / BudgetController.absorb,
report_store.append_chat_turn_events (now allocating), chat_turn_payloads;
CHAT_ORCHESTRATOR_DISTRIBUTED_*, TEMPORAL_MAX_CONCURRENT_ACTIVITIES
The turn is still one activity. Its independent plan steps are not: each batch is
handed to a seizu_chat_step_fanout workflow that schedules one
run_chat_worker_step activity per step.
Why not leave them as asyncio.gather. They were already parallel, so this
buys nothing in wall-clock for a batch that fits on one worker. What it buys is
everything that follows from a step being a scheduled unit: steps are placed
across the fleet rather than sharing one process’s CPU, memory and event loop; a
step gets its own start-to-close timeout instead of being bounded only by the
turn’s; a step that fails or whose worker dies takes only itself down, where an
unhandled exception inside the gather took the batch; and each step is separately
visible and separately cancellable. The shape that motivated it is an
investigation whose steps are genuinely independent and individually large — one
per CVE, after a first step has fetched the finding set from the graph — where
one worker holding all of them is the bottleneck and one crash is the whole
answer.
Only the middle is distributed. Routing, planning, verification and synthesis stay in the turn’s own activity: they are sequential, they share a model context, and distributing them would add a serialization boundary per stage for no concurrency. Synthesis in particular stays single-producer, so the user-visible answer still has exactly one writer.
Not for headless runs. A distributed step reports progress into the turn’s
event log and is scheduled by a workflow derived from the turn id; a headless run
has neither, and is already an activity of its own. build_turn_config therefore
carries turn_id, and an empty one is the switch.
The store allocates seq, because a turn now has several producers#
append_chat_turn_events no longer takes a sequence number. It takes the batch,
locks the turn row, and returns the number it assigned. AGT-008 requires one
append-only log whose live delivery and replay are identical, and
read_chat_turn_events truncates at the first gap — so a counter held by any
one producer is not an option once there are several: two writers picking their
own next number either collide or leave a hole, and a hole is a reader that stops
mid-answer. Allocating under the row lock makes the numbering total across every
writer and makes “row N exists” mean rows 1..N-1 were already committed.
The coordinating turn’s closing flush is still the highest number, because it
happens after it has awaited every step — which is what keeps last_seq an
honest end-of-log marker.
Don’t: restore caller-chosen sequence numbers for idempotency. Nothing here
retries: the turn is maximum_attempts=1 (AGT-008), the fan-out and each step
are maximum_attempts=1, and a batch is named by a workflow id derived from the
turn, the retry cycle and the step ids, so asking twice resolves to the batch
already running rather than starting a second paid copy.
Results come back by reference when they are large#
A step’s result carries every call it made and what each returned, bounded per
call by CHAT_TOOL_RESULT_MAX_BYTES — megabytes for a step that made dozens.
Returning that through Temporal copies it into workflow history on the way out of
the activity and again on the way in. Anything over
CHAT_ORCHESTRATOR_DISTRIBUTED_INLINE_MAX_BYTES is written to chat_turn_payloads
and the reference travels instead; the payload is keyed within the turn, so the
existing expiry sweep collects it. Truncating instead is not available: the trace
is the evidence the synthesizer answers from and the retry resumes from
(AGT-013, AGT-014).
What a worker rebuilds, and what it must never be handed#
Everything a step read from ambient context in-process is an explicit field on
ChatWorkerStepInvocation: the step, the trimmed plan, its dependencies’
outputs, the conversation excerpt, the disclosed tool set, session memory, the
sandbox id, the grant. The payload is versioned and a worker refuses one newer
than it understands, because misreading a field is worse than not running.
Identity is rebuilt worker-side and intersected, never carried. The payload
names a user_id and the permission cap the turn was admitted under; the worker
resolves the stored user and intersects (AGT-006, AGT-008). A resolved permission
set travelling in a payload would be one nobody re-checked. The worker also
re-verifies that the turn is still running and still owns its thread before doing
any work, so a step scheduled just before a cancellation cannot keep spending or
keep writing into a closed log.
Confirmation gating and chat-safe filtering are unchanged, because the step
reaches tools through the same mcp_runtime path — a mutating tool still fails
closed (AGT-001) and progressive disclosure still is not an authorization
boundary (AGT-002).
Concurrency is bounded twice#
CHAT_ORCHESTRATOR_MAX_PARALLEL bounds one turn’s batch, as before. Once steps
are distributed that is no longer enough: N conversations fanning out at once can
saturate the provider, Neo4j, the MCP proxies or the sandbox account even though
each is individually well behaved. TEMPORAL_MAX_CONCURRENT_ACTIVITIES is the
cluster-wide bound, and Temporal queues the overflow rather than dropping it.
Falling back is safe only before anything is scheduled#
_FanoutUnavailable — no Temporal client, or the start call failed — runs the
batch in-process, which bills it exactly once. Anything after the fan-out has
started propagates instead: the steps are running somewhere, and a local rerun
would pay for the same work twice and re-apply its tool side effects. A step the
fan-out could not produce comes back as a recorded execution error for the
verifier to judge, never as a silently missing step.
The first question of a conversation admits its own turn#
SeizuChatTransport.startTurn admits a turn without going through the SDK’s
send path, and the page then attaches to it (resumeStream). Only the chat
landing uses this: it creates the session and asks the question in one gesture,
so sendMessage would be called on a chat keyed to that session in the same
commit that created it — racing the SDK’s own reattach probe, its message state
and the transport’s pending slot. Measured by its absence: no chat_turns row
was ever written for those sessions, so nothing was sent at all.
A session created here is not hydrated at all. History is fetched
concurrently and applyHistory replaces the whole message list whenever what it
fetched is at least as long as what is there, so fetching alongside the stream
took the answer back off the screen. Gating the attach on the fetch instead only
moved the problem: the attach then waited on a request that had no error path,
and the conversation appeared only when history polling picked up the finished
turn. There is nothing on the server a newborn session does not already have —
so it does not ask. The skip is released in onFinish, after which the thread
hydrates like any other.
Admitting first makes the question the server’s before the UI has to be right about anything. It is the same pair a send makes and keeps everything a send gets — the idempotency key, the 503 retry, expiry recovery, a stop asked for before the turn had an id.
The attach is the ordinary reattach, not a second route into it. Calling
resumeStream by hand from an effect did not show the turn at all, while a
reload of the same conversation attached and streamed correctly — so the landing
uses the reload’s path: resume is left alone, and the turn is already in the
transport’s pending slot, so reconnectToStream finds its id without probing.
The question is put in the transcript by seeding useChat({messages}), which is
read when the Chat for the new thread is constructed; writing it from an
effect is always either too early for that construction or late enough to
overwrite what the attach has already pushed. The seed remains until the turn
finishes rather than being cleared when navigation settles, because the SDK may
read the options again while reconnecting.
Admission can also precede the worker’s first checkpoint by an arbitrarily long queue delay. During that interval the history endpoint projects the active turn’s immutable command as a pending user message unless the checkpoint already contains it. A reload must show the question the server accepted even when the producer has not emitted its first event yet.
Related, and a hazard in its own right: reconnectToStream deletes this
thread’s pending slot on a 204, and a turn admitted while that probe was in
flight is newer than the answer. It now deletes only the slot it asked about.
The conversation opens on the session’s response, not on the admission’s. Both are small writes — locally 10-70ms to create the session and ~30ms to admit the turn — but waiting for the pair meant the question left the composer and nothing took its place: no route change, no sidebar entry, no transcript. The session is enough to be right about everything the page shows, so navigation, the seed and the sidebar entry happen on it, and the admission is asked for before anything re-keys but awaited after. Only the failure report waits: a turn that was never admitted leaves the question on screen in a conversation that exists, so asking again is retyping nothing.
And the first turn a process serves is not a small write at all. Measured
against a freshly restarted web service, admission took 3.5s on the first
request of each worker and 30ms on every one after it. Almost all of it is the
deferred import litellm behind the first model-capability lookup (~3.2s in
this image), with the Temporal client connect another ~100ms; both are one-time
per process, and both were being paid by whoever opened the first conversation
after a deploy or a worker recycle. chat_turns.warm_chat_dispatch, called from
the app lifespan when chat is enabled, moves them to startup: the first
admission after a restart now answers in 60ms.
What it warms is deliberately narrow. The import is the cost, not the lookup, so
it imports litellm (chat_models.warm_model_metadata) rather than resolving a
model — resolving reads the profile store, and a startup that queries the
database to warm an import is doing something it was not asked to. It is a
warm-up, not a precondition: each half is logged and left to the lazy path it
stood in for, because Temporal being unreachable at boot is not a reason to
refuse every other request.
That reordering is why the transport records an admission while it is in flight
(PendingSend.admission). The reattach now lands inside that window, and the
204 hazard above is exactly what it would hit — the probe would answer about a
turn being admitted right then and discard the record of it. Waiting on the
admission it is already holding is the only answer that is not a guess; when
that admission fails there is no stream to hand back, and whoever asked for it
reports the failure rather than the reattach reporting it twice.
The session is also named by the question at creation, rather than by the auto-title PATCH that follows the first turn. Same text, same truncation — but the sidebar entry arrives readable instead of blank, and a new conversation costs one fewer write.
AGT-009 — Answer-only plan steps require complete evidence#
Applies to: chat_orchestrator._PLANNER_PROMPT
The planner may reuse facts established earlier, but an answer-only step is valid only when those facts satisfy the step’s success criteria. A prior answer mentioning the subject is not evidence for a missing property. If a request asks to determine, verify, investigate, cross-check, or trace something and the conversation identifies an evidence gap, the plan must gather that evidence with an available tool/skill or explicitly say the determination cannot be made.
Attack-path and internet-exposure work illustrates the distinction: selecting
CVEs from a prior ranked list may be answer-only, while claiming reachability
when the prior result contains no deployment or network data may not. Use a
direct graph tool for a bounded lookup and sandbox__delegate for iterative
exploration. Do not manufacture tool activity merely for display; tool and
subagent details represent actions that actually ran.
Why: an attack-path follow-up asked which previously ranked CVEs were accessible from the internet. The prior result contained vulnerability and repository facts, but no deployment endpoints or network-exposure metadata. The planner nevertheless made both worker steps answer-only, recorded the missing evidence as an assumption, and then presented an accessibility conclusion without making an action call. The execution trace was accurate—the absence of tool/subagent rows reflected that no evidence gathering occurred—but the answer overstated what the available evidence could establish.
AGT-010 — External MCP confirmation uses annotations with a local override#
Applies to: reporting/services/external_mcp.py,
mcp_runtime.list_tools_for_user(chat_safe_only=True)
External MCP tools are agent-only capabilities, namespaced as
ext__<proxy>__<tool>. Confirmation policy is evaluated per tool. An exact
match in MCP_EXTERNAL_CONFIRMATION_REQUIRED_TOOLS always requires
confirmation. Otherwise an explicit readOnlyHint:true does not; a complete
mutating profile of destructiveHint:false, idempotentHint:true, and
openWorldHint:false also does not. An explicit mutation or risk hint requires
confirmation. Missing or incomplete guidance falls back to the proxy’s
require_confirmation value, which defaults true. The same confirmation bypass
permission and audit path used by built-ins applies to external calls. The
autonomous sandbox subagent receives only the individual external tools that
this policy classifies as confirmation-free.
The client creates a fresh transport and header dictionary for every discovery or call. It does not pool an authenticated connection across users. Detached interactive turns and headless runs use a service credential plus target-user delegation because the browser bearer token is deliberately not persisted in a Temporal turn command (AGT-008).
Why: the MCP protocol provides standard behavioral hints at the individual tool boundary, which is more precise than treating an entire proxy as mutating. They remain advisory, so Seizu accepts them only from an operator-configured proxy, keeps an exact local force-confirm list for known-sensitive tools, and uses a fail-closed fallback by default when annotations do not establish a clear profile. Per-operation connections prevent one worker’s pooled headers from turning a subsequent user’s call into a confused-deputy request.
The web tool catalog surfaces configured proxies and their dynamically
discovered tools as read-only synthetic toolsets. This is observability and a
skill-authoring aid, not federation: external tools remain absent from Seizu’s
own MCP tools/list, and the catalog REST routes do not execute them. When a
proxy returns an OAuth challenge, its catalog contains the synthetic
seizu_authenticate tool until credentials are available. Catalog parameter
metadata preserves the external JSON Schema property names verbatim (including
names such as perPage); the lower-snake-case rule remains limited to
Seizu-authored Cypher tool definitions.
AGT-011 — An unfinished plan is discarded unless the next turn resumes it#
Applies to: chat_orchestrator.router_node, _forced_route,
_abandoned_plan_reset, reporting/temporal_workflows/activities.finalize_chat_turn
plan/step_results round-trip through the checkpoint so an orchestrated run
can survive a turn boundary, but only synthesizer_node clears them. A plan is
therefore meant to outlive its turn in exactly one case: the run stopped at
confirmation_pause and the next turn carries the approval (or a
continue-the-answer request). Both arrive as a marked HumanMessage.
router_node — the graph’s single entry point — clears an unfinished plan when
the incoming turn is not one of those, and _forced_route pins a turn to
the orchestrated path only for a genuine resume. A new user message therefore
always gets a new plan.
Why: a user stopped an investigation of one repository mid-run and asked for
a different one. Cancelling a turn cancels the graph task wherever it is, so the
dispatcher’s last checkpointed write — steps at pending/ran — stayed in the
thread state. On the next message _has_pending_plan forced the orchestrated
route and the planner kept the stale plan, so the agent resumed the abandoned
repository and never read what had just been asked. The same hole is open to any
turn that does not reach synthesis: a crashed worker, a timeout. Clearing at the
entry point covers all of them with one rule, rather than asking each producer
to unwind state it was cancelled out of.
A cancellation is recorded as a cancellation, whichever writer wins. Stopping
a turn reaches both finalizers at once — Temporal cancels the activity and the
workflow immediately schedules finalize_chat_turn — and the fallback usually
wins by a second or two because the activity is still publishing its closing
frames. It wrote a blanket failed, so a user-initiated stop surfaced as an
error and the activity’s own canceled lost the first-writer-wins race. The
fallback now reads cancel_requested from the turn, which is set before either
writer runs, so the two agree instead of racing over the outcome.
AGT-014 — A step that made calls never reports nothing, and neither does a run#
The synthesizer gets the same treatment as the step summary. A run whose two steps had both passed still opened with “could not produce a final summary” and handed the user raw step output: the synthesizer call ran, spent its allowance and returned no text. From the user’s seat that is a failed answer whatever the internals say. An empty synthesis is now retried once, asking for the answer and nothing else, before the fallback renders step output.
Its output allowance is no longer capped at a concision ceiling either. 2,048 tokens is enough for the answer but not for a reasoning model to think and answer, and the observed result was a blank one — concision achieved by saying nothing. Length is the prompt’s job; the allowance only has to leave room to answer at all.
The step-level half#
Applies to: the summary pass in chat_orchestrator._run_worker_step
When the summary pass returns nothing, a narrower retry asks for three things only — what was established, what was unfinished, what is still unknown — since that is far smaller to produce than a full summary and a model that spent its allowance thinking has a better chance at it. If that is also empty, the step reports its state deterministically: goal, completion condition, how many calls across how many tools, an explicit “still unknown”, and only then the evidence. A raw dump was the first version of this and is not a report — it leaves the verifier and synthesizer to work out what the step was for, and an absent finding reads like a negative one unless something says otherwise.
When a step’s summary pass returns no text, its result is rendered from the calls it made and what they returned, rather than left empty.
Why: the summary pass is a step’s last chance to say what it found, and it
can come back empty — refused by the budget, or a reasoning model spending its
whole output allowance without emitting text. Observed on a step that had made
90 successful calls: output=0, partial_output=0, which fails verification for
“Step produced no output”, is retried from scratch, and loses the work. The
allowance is also no longer a constant: chat_context.max_output_tokens returns
the smaller of CHAT_LLM_MAX_TOKENS and what the model reports it accepts, and
every call site that used to pick a number now goes through it. A hardcoded
1,024 bore no relation to what that model could have given, and asking above a
provider’s ceiling is refused outright rather than quietly reduced — so the
clamp matters in both directions. The synthesizer keeps its deliberate concision
ceiling as min(model_limit, 2048); structured calls (router, planner,
verifier) are clamped once inside _structured_invoke, where the model is
chosen.
The fallback is still the load-bearing half: it does not depend on knowing why the model went quiet.
AGT-017 — Stop useless work; do not ration all work#
Applies to: _step_thresholds, _looks_stuck / _note_call_signature in
_run_worker_step, _prepare_retries, _stuck_notice in sandbox.py;
CHAT_ORCHESTRATOR_STEP_SHARE_HARD_MULTIPLE,
CHAT_ORCHESTRATOR_STUCK_CALL_WINDOW, SANDBOX_STUCK_REPEAT_LIMIT
A token ceiling cannot tell a run that is looping from one that is working: both spend. So the ceiling is no longer where a long investigation ends, and the looping is detected as itself.
The per-step share is a signal, not the execution cut. Its purpose is that no step starves its siblings — a scheduling concern — and at a hard multiple of 1.0 it was also what ended the step. Measured across four consecutive CVE-reachability runs, every one stopped on the step share while the run budget sat ~80% unspent and the cost budget at ~16%. The default multiple is now 3.0: crossing the share still degrades the step and tells it to converge; what changes is that a step with no sibling contending may use what the run can actually spend.
This re-breaks a tie that a three-arm sweep had left open. The sweep found no quality difference between multiples, so the tie went to sibling protection; it did not measure the case that matters here, which is a plan with one genuinely large step.
Three loop detectors, at the level each loop happens.
Within a step: a full window of tool calls (
CHAT_ORCHESTRATOR_STUCK_CALL_WINDOW, default 8) with no call the step had not already made. The step stops, keeps what it gathered, still runs its summary pass, and is marked terminal — a step that has run out of new calls to make will run out again. A full window is required so ordinary repetition (polling, re-reading a file just written) does not trip it.Across attempts: a rejection the step has already been given once and not addressed is terminal. Three of four attempts in one measured run were the same verdict restated, and they cost the rest of the run’s budget.
Inside a delegation: consecutive already-answered calls (
SANDBOX_STUCK_REPEAT_LIMIT, default 3) escalate from the per-call note to an instruction to stop and report. The per-call note says one call was pointless; it does not say the task is.
Why this direction, and not a tighter cap: a cap hit while answering a hard
question does not save the tokens it appears to. The work is re-done in the next
turn or the next session, from a cold context, and the failed run is a total
loss on top. Cheap detection of useless work is what makes an expensive
useful run affordable; a cost ceiling (CHAT_RUN_COST_BUDGET_USD) remains the
outer guard against genuine runaway, and is the one an operator should set.
AGT-016 — The planner does not supply identifiers the request did not#
Applies to: _PLANNER_PROMPT, the repo_cve_reachability skill
A plan step must not name a repository, organization, account or host the request did not give it. Identifiers in this graph are whatever was scanned, and a familiar-looking name is the trap: the resource is very unlikely to be the upstream project it shares a name with. A bare name stays bare in the goal and is resolved against the graph at execution time.
Why: asked about “the confidant repository”, the planner wrote
lyft/confidant into both step goals from its own knowledge. The findings step
correctly resolved mappedsky/confidant and was then failed by the verifier
for not reporting on lyft/confidant — the invented identifier had become the
thing the step was judged against. Worse, the reachability step believed it:
all 100 of its GitHub reads went to lyft/confidant, an unrelated public
repository, so it was judging one codebase’s recorded vulnerabilities against
another codebase’s source. That failure mode produces confident, cited,
wrong-target verdicts, which is worse than producing nothing.
The skill carries the same guard where the calls are actually made: if the repo it was handed disagrees with the one the findings step resolved, it uses the resolved one and says so, and it refuses to read a repository the graph has no record of.
AGT-015 — What a retry is told, and what it is then judged on#
Applies to: _worker_user_message (the resume block), the required-action
guard in _run_worker_step, _dependency_context, the verifier prompt in
_verify_step
Four rules, each from the same observed run: a reachability step produced a correct, cited review on its first attempt and was retried three times until the budget was gone.
A “cannot be determined” that names its missing evidence is a finding. The
verifier failed a review of 19 CVEs because one was Undetermined — a verdict
the skill defines, and requires evidence for. AGT-009 already allows
a plan to “explicitly say the determination cannot be made”; the verifier now
applies that to part of a result as well as the whole, and still fails anything
left silently unaddressed or asserted beyond its evidence.
A rejected attempt is told it was rejected. The resume block said “ran out of budget before finishing … do not re-gather what is already here” for every carry. Told that, a worker whose result had been rejected reasonably skipped the work the rejection asked for — including its required skill.
A step’s contract is satisfied once, not once per attempt. Having skipped the
skill, the retry was failed for not calling it: the guard runs per attempt while
required_action is a property of the step. It is now remembered on the step,
so a later attempt is not failed for a contract an earlier one met. A first
attempt that skips its required action still fails.
A dependency gets a budgeted share, not a fixed 2,000 characters. The
19-finding list reached the dependent step truncated; the worker said so, and
the verifier held the incomplete coverage against it. Split across the step’s
dependencies (CHAT_ORCHESTRATOR_DEPENDENCY_CONTEXT_MAX_CHARS, default 16,000),
and a slice now says it is one — silent truncation is how a step comes to report
missing coverage without knowing what it is missing.
AGT-013 — A retry carries what the attempt fetched, not only what it wrote#
Applies to: chat_orchestrator._prepare_retries, _worker_user_message
A failed step’s retry resumes from partial_output when the worker wrote one,
and otherwise from a bounded digest of the calls it made and what they returned.
Why: the two conditions were mutually exclusive in practice. A worker cut at
its step ceiling never gets to write a partial summary — that is precisely what
produces Step produced no output., which is what fails verification and sends
the step back for a retry. So the carry-forward path existed for a case that
could not reach it, and the retry re-gathered from scratch. Observed on a
reachability step: output=0, partial_output=0, budget_capped=True, and the
second attempt re-fetched files the first had already read.
An interrupted attempt also leaves its full trace in the sandbox. The
digest above is bounded by what fits in a prompt, which is the wrong shape for
what a step that made ninety calls has to hand on, so _persist_step_record
writes the whole trace to a file and records it as a receipt — the machinery
that already tells a delegation about result files then tells it about this
(SBX-008). Best-effort, and only into a sandbox that is already
open: it is a convenience for the next attempt, never a reason to open one or to
fail a step that has otherwise finished.
tool_details is thin for a delegating step — it records the delegations, not
the sub-agent’s calls — so this helps a directly-working step most. The
sandbox layer has its own carry: the session digest and receipts already tell a
later delegation what is on disk (SBX-008), which is why the
observed retry repeated 13 of 99 calls rather than all of them.
AGT-012 — Running out of budget must not delete what the run already found#
Applies to: chat_orchestrator._dispatch_batch (the degraded/finalizing
sweeps), _budget_stop_result, _step_evidence, _rendered_step_status
When the run budget enters finalization the dispatcher marks every unfinished
step skipped, which is what stops the retry loop. It must annotate the
step’s existing result rather than replace it: the stub it used to write
(output: "", tools_used: [], no tool_details) deleted the findings the
step had already gathered.
A step with retained findings is never labelled “skipped” either. skipped
is the routing status; as a label above real evidence it tells the reader to
discount it, which is the same failure one layer up.
Why: a CVE-exploitability run made 33 tool calls, read the repository’s
manifests, lockfile and source, spent 302,679 input tokens — and answered “the
step was skipped and produced no output or supporting evidence”. Nothing
hallucinated: the worker is killed at its share of the run budget before it
writes its summary, so the result carried evidence and an empty output; the
verifier failed it for having no summary; the retry pass met it at failed and
the sweep overwrote it. _synthesis_context forwards tool_details precisely
so a missing summary cannot take a step’s findings down with it
(AGT-009 is the same concern from the planner’s side), and the stub
deleted that safeguard’s input. The checkpoint shows it exactly: step_results
went 110,930 bytes → 390 bytes → the answer. Replaying the real state through
the fixed path hands the synthesizer 12.5k characters of evidence instead of
“(no output)”.
Identical tool results are charged once. A worker that re-runs a tool with the same arguments records the result again, and an equal split of the evidence budget then pays repeatedly for one fact while genuinely new evidence falls off the end — 33 recorded calls, 25 distinct, on the run above.
Don’t: treat “the budget ended the run” as “the run found nothing”. The
terminal status is budget_exhausted and the answer must be the partial one the
evidence supports, with the limit stated.
AGT-036 — Agent Skill allowed-tools is a dependency declaration in Seizu#
Applies to: plugin_packages.py, mcp_runtime._resolve_plugin_allowed_tools
For tool names Seizu recognizes, allowed-tools means the skill requires and
discloses that tool. The skill is hidden for a user whose ordinary tool listing
does not contain it. Unknown portable tokens are retained and ignored.
Why: the existing tools_required field already meant that the workflow
could not run without those tools. Keeping a second Seizu-only field would make
portable packages declare the same dependency twice and allow the declarations
to drift. Treating the standard field as permission instead would conflict with
AGT-002: disclosure is not authorization, and RBAC plus confirmation
remain the only execution boundary.
Logical mcp:<server>/<tool> names resolve only to operator-configured proxies;
the package endpoint is never contacted. URL matching has three modes. none,
the default, ignores the package URL and binds an equally named proxy. lax
prefers a configured or advertised URL alias and then falls back to the proxy
name. strict requires exactly one URL alias match. Every mode also requires
the user’s discovered inventory to contain the exact remote tool.
Why: in a server-side agent the operator-controlled proxy configuration is the execution boundary, while the package URL is deployment-specific metadata and neither grants authorization nor selects a network destination. URL matching still offers optional provenance and configuration checking. Name fallback remains narrower than matching a tool leaf globally: both the plugin server name and remote tool name must match, while RBAC determines whether the discovered tool is present.
AGT-037 — A resource listing is a catalogue of skills, not of files#
Applies to: mcp_runtime.list_plugin_resources_for_user,
list_plugin_resource_templates_for_user, mcp_server._handle_list_resources
resources/list returns one resource per enabled plugin skill — its SKILL.md
URI, title, description and declared allowed-tools — not one per packaged
file. resources/templates/list advertises
seizu://plugins/{plugin_id}/versions/{revision}/files/{path}, and
resources/read still accepts any file in a published revision.
Why: a listing exists so a caller can decide what is relevant, and that
decision is made from a skill’s identity and description. A skill’s
references/, scripts/ and assets/ are named by its own instructions and
fetched by URI, so enumerating them made every file of every installed plugin
the price of asking what was available — a listing bounded by
plugins x files rather than by skills. It was also a query per plugin, and it
was the one surface here that would have needed cursor pagination to stay
bounded. Narrowing it removed all three at once.
Don’t: resolve allowed-tools against the caller’s inventory to build this
listing. Resolution fans out to every configured external MCP proxy, and a
catalogue read must not pay for that; the names are carried as declared, and
availability is still decided at render time (AGT-002, AGT-036). Don’t:
re-add file enumeration “for discoverability” — the template is the
discoverability mechanism, and the render already hands over the skill’s prefix.
AGT-038 — External discovery is memoized per turn, cached across turns only on request#
Applies to: external_mcp.discovery_scope / begin_discovery_scope /
discover_proxy_tools / invalidate_discovery_cache,
MCP_EXTERNAL_DISCOVERY_TTL_SECONDS
Discovering one proxy’s tools costs a transport, an MCP initialize and a
paginated tools/list. A single turn asks for that answer from the system
prompt’s capability listing, the planner’s, and every skill render that resolves
dependencies — five to ten identical fan-outs. Two layers now sit in front of it:
A scope memo, opened by
chat_agent_node,dispatcher_node, the distributed step worker and the MCP ASGI middleware. Valid by construction: one identity, one turn, seconds wide, removing only duplicate work. Always on.A TTL cache (
MCP_EXTERNAL_DISCOVERY_TTL_SECONDS, default0= off) that makes a cold turn cheap. Bounded to 512 entries, LRU-evicted, and dropped for a user whenever an upstream refuses their identity.
Why the split: the memo cannot be wrong; the TTL cache can. A tool the user just lost stays listed and one they just gained stays hidden until it expires. Neither is an authorization decision — the call is still checked by RBAC and by the upstream, and disclosure was never a boundary (AGT-002) — but both are visible, which is why the cross-turn layer is opt-in rather than a default.
Both key on the user. A proxy’s listing is what that delegated identity is authorized to see, so a cache keyed by proxy alone would hand one user another’s view. This is AGT-010’s rule against pooling the transport, applied to what the transport returned. Don’t widen the key to make the hit rate look better.
This is also what made resolving allowed-tools affordable in
resources/list (AGT-037): the catalogue now resolves against the same
inventory the rest of the request already discovered, and omits a skill whose
dependencies are unreachable, exactly as the prompt listing does.
AGT-039 — A rendered skill is two messages: static instructions, then its inputs#
Applies to: render_skill_parts, render_skill_inputs,
mcp_runtime._get_prompt_core, plugin_packages.parse_package
prompts/get returns the skill body unchanged and this invocation’s argument
values as a second message. GetPromptResult.messages is a list; the body is
the same bytes on every run, and a skill refers to a value by name rather than
having it substituted in.
Why: the argument mechanism was always standard — MCP prompts take
arguments — but the template lived in the portable SKILL.md body, and a
consumer without Seizu’s parameter extension reads {% $repo %} literally. The
package was portable while its instructions were not. Keeping values in their
own message makes the file readable anywhere and removes a second discrepancy:
materialize_plugin_skill writes the raw package bytes into the sandbox, so a
sub-agent re-reading SKILL.md used to see placeholders where the outer agent
had values.
Substitution still runs, so packages written the old way render exactly as
before; publishing one records a non-blocking templated_skill_body warning.
The legacy skillset projection is exempt — its bodies are generated from records
whose author cannot restructure them — and the legacy REST render endpoint still
returns the body alone, because a legacy skill inlines its values and an inputs
block there would both duplicate them and change a response callers parse.
Not a caching change. A rendered skill arrives as a tool result at the tail
of the conversation, and only Prompt.description reaches the system prompt, so
neither shape moves the cached prefix. Don’t cite caching as the reason for
this; the reason is that the body travels.
AGT-040 — A package and a skill each have one identity#
Applies to: plugin_packages.derive_seizu_id, SeizuPluginExtension,
PluginCreateRequest, PluginSkillEditor.tsx
A plugin’s Seizu id is derived from the package name, and a skill’s from its
portable name: hyphens and dots become underscores, and the result must be a
valid MCP name component. skillsetId and skillId do not exist: the
extension forbids unknown keys, so a package stating either is refused rather
than reconciled. The whole com.mappedsky.seizu extension is optional, so a
stock Agent Plugins 1.0 package installs unmodified.
Why: the pair was immutable in both directions — a package carried a
portable name and a Seizu id forever, with nothing keeping them related and
no way to change either. skill_id already derived by default, so the two
halves of the same idea disagreed. Naming a thing once is the whole feature;
an author who wants a different id renames the package or the skill directory.
A name that derives nothing valid — leading digit, over 31 characters — is refused at publish and at create, naming the constraint rather than silently inventing an id.
Known trade: STO-009 used an explicit same-ID package to bind a
production cutover to existing skillsets, and a derived id could in principle
adopt a legacy skillset that happens to match. Accepted deliberately: the
legacy surface exists for one release and the collision needs a legacy
skillset whose id is exactly the derived one. If that release is extended, put
the check back before the derivation, not the field.
Consequence: a revision published before this, whose stored manifest states
either field, cannot be restored — restore republishes that revision’s files
and they no longer validate. Both fields were introduced unreleased, so this
was settled by discarding the affected history rather than tolerating the keys.
AGT-041 — Whether a skill is on is an operator’s choice, not package content#
Applies to: SeizuSkillExtension.enabled, publish_plugin,
set_plugin_skill_enabled, PUT /api/v1/plugins/{id}/skills/{skill_id},
PluginDef.skills
A package does not say whether its skills are on. Enablement is store state,
chosen when a plugin is installed or updated: every skill a revision introduces
starts on, publish_plugin carries existing values forward, and an operator
changes one through the API, the CLI, the seed configuration or the plugin
detail dialog. enabled is not a field of the extension at all — a package
carrying one is refused, because the extension forbids unknown keys.
Why: it was neither in the Agent Plugins spec nor meaningful to any other consumer — only Seizu’s extension carried it — and it sat on the wrong side of the authoring/runtime line. Disabling one skill meant editing a manifest and publishing a revision, while the plugin it belonged to was already toggled at runtime and already an install-time seed argument. The two halves of the same question worked differently.
Removing it also removes a rule nobody would have got right: if a package ships a skill off, an operator turns it on, and a later revision ships it off again, what wins? With enablement outside the package there is nothing to reconcile — a republish simply never touches it.
Don’t: reintroduce it as a “default for first install”. That is the rule above wearing a hat, and it makes a package’s meaning depend on whether the store had seen it before.
AGT-042 — Seizu’s own tools are named like any other MCP server’s#
Applies to: plugin_packages.mcp_tool_ref, SEIZU_MCP_SERVER_NAME,
_resolve_plugin_allowed_tools
allowed-tools entries are mcp__<server>__<tool>, and Seizu is the server
named seizu: mcp__seizu__graph__query, mcp__github__get_file_contents.
Anything that is not an MCP reference — Read, Bash(git:*) — is the
consumer’s own built-in, preserved and never resolved by us. seizu is
reserved: an external proxy or an mcp.json entry of that name does not answer
for it.
Why: the package previously used three vocabularies at once — bare
graph__query for Seizu, mcp:<server>/<tool> for external MCPs, and
ext__<proxy>__<tool> for their internal names — and only the first was even
the tool’s real name. mcp:<server>/<tool> was ours alone; nothing else reads
it. mcp__<server>__<tool> is what the ecosystem uses (it is Claude Code’s
permission-rule syntax, which is what allowed-tools values are), and Seizu’s
tool names are already group__action, so mcp__seizu__graph__query parses as
server seizu, tool graph__query and resolves in a client that has Seizu
configured. Neither the Agent Skills nor the Agent Plugins specification defines
a tool-naming convention, so the one in use is the one worth matching.
The field means something different at each end, and we keep our meaning.
The Agent Skills spec calls allowed-tools “tools that are pre-approved to
run” and marks it experimental; Claude Code grants those tools for the turn.
Seizu treats it as a dependency: a skill is absent from a user’s list when a
listed tool is unavailable to them (AGT-002 still applies — it grants nothing).
Gating is kept deliberately, because not offering a skill whose tools the caller
cannot reach beats offering one that fails halfway. The consequence to know: a
package authored elsewhere that lists tools defensively becomes unavailable here
if any one of them is missing.
Don’t: resolve bare names as Seizu tools again “for convenience”. That is a second spelling for a tool that already has one, and it collides with the built-in names the spec’s own example uses.
AGT-043 — Every stage resolves its model and effort the same way#
Applies to: _get_sandbox_model, chat_models.model_id_for_role,
_STAGE_PARENT, CHAT_LLM_ROUTER_MODEL, CHAT_LLM_SANDBOX_REASONING_EFFORT
The sandbox sub-agent built ChatLiteLLM itself whenever SANDBOX_LLM_MODEL
was set, so it never reached reasoning_kwargs. Effort travels through
model_kwargs (AGT-019), which means no setting could reach the wire: the
highest-volume stage in the system ran on provider defaults by construction, and
there was no configuration that could have said otherwise. It now resolves
through chat_models.resolve("sandbox_subagent") like every other stage, which
carries the derived output ceiling and temperature rules with it.
Measured. Grading it low took the sub-agent from 955 reasoning tokens and
11.2s per call to 422 and 7.4s – across a delegating turn, roughly half the
tokens and a third off the wall clock, and the first sample in seven to finish
the reachability step without exhausting its budget. Read the reasoning tokens,
not the setting: DeepSeek collapses several levels into one value (AGT-033), so
the token count is the only evidence the grade arrived.
A stage’s own model now wins before its parent’s, or SANDBOX_LLM_MODEL would
be unreachable — the sub-agent’s parent is the worker, and the parent remap ran
first. The router also gets its own model setting: it shared the planner’s, so
“put the planner on the strong model” silently moved a binary classifier that
emits 48 tokens there too.
Don’t: trust reasoning_effort on a trace to tell you what a stage is
graded at unless it is read off model_kwargs. The sub-agent’s span reported
None while the stage demonstrably ran at low, which is the one attribute
someone would check.
AGT-044 — Thinking is streamed as a detail, and filed by the phase that produced it#
Applies to: _run_llm_tool_turn_inner, _reasoning_detail_data,
_append_reasoning, detail_writer, buildDetailTree
The details pane shows a thinking entry per model call. It is emitted while
the reasoning arrives (paced by _REASONING_STREAM_MIN_CHARS /
_REASONING_STREAM_INTERVAL_SECONDS, never re-sent unchanged) rather than once
at the end, and it carries the step it belongs to, so it renders inside that
step’s section instead of at the root of the turn.
Three things had to change together, and each is load-bearing.
The channel. writer was both the text channel and the detail channel, so a
stage that must not ship prose — a worker step, a summary pass, a post-action
turn in the single-agent loop, the synthesizer — switched off its details along
with its text. Those are exactly the stages that spend the longest thinking and
show the least while doing it. detail_writer is the second channel; a call
with a writer still uses it for both.
The section. The phase a call already carries is what files the entry:
worker:s2, verifier:s2, worker_summary:s2 — the grammar
chat_budget._observation_key documents, which drops the step id again for
estimation. _structured_invoke grew step_id/writer for the same reason.
Nothing new is threaded through the call graph to say where a call belongs.
The order. The pane grouped a detail under “the step we last saw”, which is
only correct when steps are sequential. They are not (AGT-018): two steps’ tool
calls and thinking interleave in one log, and the verifier thinks about a step
after every step has opened. buildDetailTree now keys the step nodes by
step_id, so ownership is stated rather than inferred from arrival order — the
same fix a reload always needed.
Streamed thinking keeps its tail, not its head (_append_reasoning, the one
place in chat_graph that truncates that way). At the display bound a
head-truncated body stops growing, which is indistinguishable from a stage that
stopped thinking — and it happens precisely on the long calls this exists for.
The block is ordinary content at the top of its turn. A thinking entry starts
expanded — it reads as prose and is why the block is open — while a tool call
starts closed, its value being the one line that names it. The block itself is in
normal flow and scrolls with the conversation. It was briefly position: sticky,
which pinned it over the answer it sat on top of, so it had to collapse and
re-expand itself from a measurement of when it had pinned — inferring the user’s
intent from scroll position, and unpredictable to use. Nothing moves it now but a
click, and its height is bounded (min(300px, 40vh)) rather than reserved, so a
two-entry trace takes two rows.
The pane follows streamed output while the reader is near the bottom, pauses when they scroll up, and resumes when they return to the bottom. That position is recorded on scroll, before the next content update: measuring it after an update mistakes a large chunk of new output for the reader having scrolled away and stops following. Reopening the pane also applies the recorded follow state.
What is not persisted: a plan step’s thinking is live-only. Worker
LLMTurnResult.details are dropped, and _orchestration_details rebuilds a
reloaded turn from the plan and the step results, which would have to carry up
to 6,000 characters per step through Temporal history to keep it. The
single-agent path and the synthesizer persist theirs, as they always have.
A native structured call has no thinking to show. with_structured_output
is one ainvoke and yields no reasoning chunks, so the planner’s entry appears
only on the JSON-prompt fallback. Do not stream the structured runnable to get
one: it yields parsed objects, not chunks.
AGT-045 — Model families lock on first use; reasoning remains selectable#
Applies to: model_profiles.py, chat_models.ModelSpec,
ChatTurnCommand.resolved_model_profile, ChatInterface.tsx
Admins manage versioned model profiles in the database. Every holder of
chat:use selects a profile plus one of its admin-configured user reasoning
levels; profiles default to low, medium, and high. The UI groups those
levels beneath each profile, then shows only the locked profile after the first
admitted turn atomically locks its family on the session. Its reasoning level
remains selectable between turns. The resolved pair is copied as a complete
snapshot into each admitted turn. Admission expands inheritance and deployment
fallbacks into primary and economy specs for every runtime stage. Call sites
only select the stage and whether the run is degraded; a missing stage or an
incomplete spec is invalid and never falls back to a worker’s environment.
Scheduled chats and agent_chat workflow activities use the same catalog and
snapshot their selection before Temporal dispatch. A deleted or disabled
explicit choice never silently falls through to another profile: schedules and
workflows require a replacement, while a locked interactive conversation must
be restarted under a new profile.
A profile has one primary base model. Direct assistant calls use that base, and
every runtime stage inherits it unless explicitly overridden. Router, planner,
worker, worker-summary, sandbox-subagent, verifier, and synthesizer may each
override the model and may either inherit the user’s selected reasoning level
or fix an admin-configured value. There is no separate assistant override.
One economy model and reasoning value applies wherever budget degradation needs
the fallback; it is not duplicated per stage. The run cost ceiling is the lower
positive value of the profile cap and CHAT_RUN_COST_BUDGET_USD.
Admins choose the user-visible subset from LiteLLM’s fixed default, none,
minimal, low, medium, high, and xhigh vocabulary. default is
rendered by omitting an effort so it remains safe for provider adapters whose
nominal default mapping is not accepted by every underlying model.
Why: changing effort keeps the same model family, while changing profile families moves the request to another model and guarantees that the next turn cannot reuse that model’s prior cached prefix; its carried context is uncached input. A new conversation makes that cost boundary explicit. Capturing the expanded specs at admission keeps a profile edit or effort change from changing a running turn or one of its distributed workers. A single user level cannot safely flatten the profile: stages such as worker-summary are transcription passes where extra thinking crowds out the answer, while planner and worker may benefit from it (AGT-019). Router and verifier are structural classifiers, so a request for a stronger answer should not silently move them to a more expensive model.
No built-in profiles are seeded. Until an admin creates the first enabled profile, chat
uses the environment configuration unchanged; the first enabled profile becomes
the default. Once profiles exist, exactly one enabled profile is the default.
CHAT_LLM_MODEL remains the concise environment base and fallback, but startup
accepts an enabled default profile in its place. Both the web process and the
Temporal worker validate this after initializing the report store, so a
profile-only deployment does not need a placeholder environment model.
Why no built-ins: unmeasured provider/model combinations would present an operational guess as a safe cost preset. The environment fallback preserves an existing deployment without making that claim.
AGT-046 — Seeded model profiles match by name and select the default first#
Applies to: ReportingConfig.model_profiles, seizu seed, seizu export
Model profiles are portable configuration. Their YAML keys are local handles; seed matches stored profiles by exact name, the same value protected by the API’s uniqueness constraint. Export preserves an existing local key by name and otherwise derives one. Reconciliation processes the declared default before every non-default profile, and profiles precede the rest of the seed pipeline.
Why: profile ids are generated by the server, so exporting them would make a configuration deployment-specific. Changing away from the current default first is rejected by the store’s exactly-one-enabled-default invariant, while selecting the target first atomically transfers that role without an invalid intermediate state. Running the section first gives later seedable resources a stable profile catalog to reference.
AGT-047 — Trace content separates prompts from results and identifies skills#
Applies to: telemetry.py, chat model/tool spans, skill prompt metadata;
TELEMETRY_RECORD_CONTENT, TELEMETRY_RECORD_PROMPTS,
TELEMETRY_CONTENT_MAX_CHARS
Skill identity is non-content trace metadata. A listed skill carries its stable skill id, display name, and immutable revision into the chat tool spec; the primary rendered skill is inherited by the step and its descendant model/tool spans. This makes runs from two revisions comparable without exporting their instructions.
Tool arguments and successful results follow the existing content opt-in. System prompts, model inputs, and rendered skill bodies use a separate prompt opt-in because they are larger and expose the complete instruction context. Both switches remain off by default and share one operator-configured per- attribute character bound.
Why: output-only traces identify a bad answer but cannot reproduce what led to it, and spans without skill revision identity cannot form the before/after populations needed to improve a skill. Prompt capture is materially more sensitive than bounded action evidence, so accepting the latter must not imply accepting the former. A shared limit makes the amount exported an operator choice instead of a collection of code-site constants.
AGT-048 — Gateways own upstream user grants; service tokens renew automatically#
Applies to: external_mcp, external_mcp_tokens, chat_connections,
ExternalMCPProxy.client_credentials / user_authorization
Per-user delegation and recovery are experimental pending the end-to-end validation in #312. ContextForge v1.0.9 transport tests passed, but bearer-plus-target-header requests retained the bearer owner’s authority; trusted-header mode needed email identity mapping and exposed a server-scoped CSRF incompatibility. Transport success therefore does not establish per-user grant isolation. Shared-token external access is outside this experimental designation.
Per-user external authority is an explicit gateway contract. By default Seizu
supplies the stored run owner’s durable OIDC (issuer, subject) pair, never its
local user ID, over either M2M bearer authentication or trusted mesh
authentication. A gateway may instead map a nonempty identity-provider claim
such as email into a target header. The gateway maps that identity to its own
upstream grant and must never substitute a shared credential when the grant is
missing. Shared static API tokens remain supported through bearer plus
token_env.
Why: Temporal deliberately holds no browser bearer (AGT-008), and Seizu’s OIDC refresh token is an encrypted browser cookie, not a worker-accessible vault. An existing gateway already owns consent, refresh, account linking, and upstream policy. Its user directory cannot identify a Seizu-local ID; the durable OIDC pair is the identity boundary (AUTH-001). Duplicating that store in Seizu would introduce another grant lifecycle without proving the upstream account belongs to the asserted user.
M2M client credentials acquire and renew service tokens in process memory;
existing token_env configurations remain compatible. A token acquisition is
coalesced per configuration and a rejection invalidates only the rejected cached
token. Calls are not replayed for renewal because a transport failure need not
prove a tool had no effect. Mesh-authenticated deployments need no second bearer.
Why explicit error classification: a service-account 401 is an administrator
problem, not evidence that the target user needs consent. HTTP 401 is a service
failure and 403 is a permission denial. The initial custom X-Seizu-Auth-Error
contract is replaced by standard MCP URL elicitation: elicitation/create and
the 2025-11-25 -32042 error, plus 2026-07-28 InputRequiredResult requests.
The SDK’s capability builder is narrowed to URL-only because its callback API
otherwise advertises form support that this client does not provide.
Detached workers cancel legacy callback requests and do not continue modern
input-required results. Neither response represents human consent. Recovery is
manual on Chat Connections; no tool is automatically replayed and completion
notifications cannot establish a grant or resume a finished run. URL elicitation
can also request payments or other interactions, so its status is generically
interaction_required, not an assertion that the user needs OAuth consent.
Recovery URLs are bounded and restricted to the operator-configured account page’s origin, revalidated on read, and never included in model diagnostics. The owner sees the gateway’s plain-text explanation and target host before choosing whether to navigate. This permits nonce-bearing protocol URLs without making arbitrary server redirects trusted destinations. Gateways must validate browser identity independently and must not put credentials in URLs/messages. Legacy non-opted-in proxies keep their existing OAuth path.
Connection observations survive worker restarts (STO-013), including failed
discovery that hides a skill. Their owner-facing page is gated by chat:use, so
a custom chat-only role can recover without acquiring toolset administration.
AGT-049 — External Streamable HTTP negotiates modern-first#
Applies to: external_mcp, external_mcp_elicitation.ClientSession,
ExternalMCPProxy.protocol_mode
Streamable HTTP uses SDK auto negotiation, trying modern server/discover
before a bounded legacy handshake fallback. SSE remains legacy. Operators may
force protocol_mode: legacy. Both paths create a fresh owner-scoped transport
per operation and accept metadata from the negotiated result.
Why: initializing unconditionally limits the client to the handshake era, even when the installed SDK and server support modern stateless requests and input-required results. The SDK supplies version selection and request stamping; its private auto-negotiation helper is isolated in our session adapter and covered by wire-level tests. The adapter restricts fallback to compatibility failures: authentication, rate limits, outages, timeouts, and positively disjoint version sets must not become attempts to negotiate a different protocol.
We use the session API, not the high-level client’s automatic input-required continuation driver, so negotiation cannot change the detached recovery and no-replay contract in AGT-048. The legacy override accommodates peers that reject requests made before initialization without treating their errors as consent to downgrade automatically.
AGT-050 — MCP forms decide existing action confirmations#
Applies to: mcp_server._handle_call_tool, mcp_runtime._ensure_tool_confirmation
MCP 2026-07-28 clients advertising form elicitation receive an
InputRequiredResult for pending Seizu action approvals. The confirmation ID
is the continuation state and input-response key. The runtime resolves the
confirmation after current permission and argument validation, using the
existing caller/session/tool/target/argument-hash scope. The form carries no
fields: the elicitation action is the decision, so a matching continuation
that accepts approves the record, decline denies it, and cancel leaves it
pending. Execution consumes the ordinary atomic grant. Other clients retain
the browser URL flow.
Why: these approvals need a decision, not credentials, so a client form
removes the browser round trip. The modern input-required flow works with our
stateless HTTP transport; legacy server-to-client callbacks require a
back-channel that this deployment does not retain. Reusing confirmation records
preserves ownership, expiry, decision attribution, and the execution claim;
trusting a bare accept response or setting confirmation_pre_approved would
discard those checks. A required confirm boolean defaulting to false was
tried and removed: it restates the accept/decline the protocol already
carries, and a client that submits the form without toggling it silently
denies the action and then holds the denial for the confirmation window. The
SDK’s capability checker tests elicitation presence but does not distinguish
form from URL support, so the transport checks the per-request mode
declaration explicitly.
Don’t: turn a form response into a permission grant or bypass flag, or replace upstream account authorization URL elicitation (AGT-048) with a form.
AGT-051 — Keep HTTP connections alive between MCP requests#
Applies to: gunicorn.conf, MCP Streamable HTTP clients
Gunicorn uses a five-second HTTP keep-alive interval with UvicornWorker. Legacy MCP clients still initialize normally, independently of the modern protocol and form-elicitation capability gates.
Why: with keepalive = 0, Codex 0.153.4 received Seizu’s legacy
initialize response, then repeatedly failed sending
notifications/initialized with a closed transport. Separate curl requests
returned 200 and 202, concealing the connection-reuse failure. The same native
Codex client, with mcp_2026_07_28 explicitly disabled, completed initialization
and listed all 105 tools against the same application with --keep-alive 5.
This is an HTTP connection-lifetime issue, not a failure to negotiate an older
MCP revision. After applying the change to the normal endpoint, native Codex
discovered all 105 tools with both 2025-06-18 (flag disabled) and 2026-07-28
(flag enabled). The in-process ASGI tests do not exercise TCP connection reuse.
Don’t: disable keep-alive to make MCP stateless. Protocol session state and HTTP connection reuse are independent.
AGT-051 — Confirmation elicitation mode is an operator’s choice, defaulting to URL#
Applies to: mcp_server._elicitation_mode, _elicitation_params,
settings.MCP_CONFIRMATION_ELICITATION_MODE
MCP_CONFIRMATION_ELICITATION_MODE selects how an MCP client collects approval
for a mutating action: url (default) points the client at Seizu’s own
confirmation page, form (AGT-050) renders a dialog in the client, permission
picks form for callers holding chat:bypass_permissions and url for the
rest, and off returns the payload as content. In url mode the client’s
response never carries a decision — the responder re-reads the record rather
than writing one, so a continuation that claims approval without one gets the
pending confirmation back. A client that cannot do the configured mode receives
content, never the other mode. Only a first attempt elicits; a continuation that
arrives unapproved is answered with the payload.
Why: the protocol has no way to show that a person saw a form. The client
reports the decision, so a client that answers automatically approves every
action its caller is otherwise permitted to take, which is exactly what the
confirmation exists to prevent — and an operator running clients it does not
control cannot detect the difference. Deciding in Seizu costs a round trip and
removes the client from the trust path entirely, so it is the default; form
stays available where the client is trusted, and permission reuses the
judgement a deployment already made about who may skip confirmations. Falling
back from url to form when a client lacks URL support would let any client
opt itself into the weaker flow, so the fallback is content instead. A bare
elicitation: {} counts as form support but not URL support: Claude Code 2.1.263
advertises it and then rejects a URL request outright, which fails the call
rather than degrading it.
Don’t: let a url-mode response decide a record, or treat a missing client
capability as licence to downgrade the mode.
AGT-052 — Modern URL elicitation has a Codex compatibility gap#
Applies to: mcp_server._elicitation_params
Keep the SDK’s protocol-version serialization for URL elicitation. Codex 0.153.4’s modern URL path is not validated as compatible with Seizu.
Why: a native Codex call failed with Unexpected response type before
displaying the URL. The captured Seizu response contained a URL elicitation
inside InputRequiredResult; Codex’s generated schema requires elicitationId,
while the installed Python SDK marks it as removed at MCP 2026-07-28 and strips
it on serialization. Adding the field to the request object was tested and did
not put it on the wire. The report remained present and its confirmation stayed
pending. This is distinct from the working form flow and from capability-based
fallback: Codex advertises URL support, so Seizu offers the configured URL mode.
Don’t: downgrade to form elicitation to work around a URL-client failure.
AGT-053 — One converter writes allowed-tools, and the parser reports the other spellings#
Applies to: plugin_packages.allowed_tool_entry, _allowed_tool_diagnostics,
routes/skillsets._legacy_skill_markdown, legacy_skillset_package,
report_store.reconcile_legacy_projection, src/pluginAuthoring.toolDeclaration
AGT-042 made
mcp__seizu__<tool> the only spelling that resolves, and deliberately left
every other token alone as the consumer’s own built-in. Three writers inside
Seizu kept emitting the internal spelling into packages they generate — the
startup projection of a legacy skillset, the legacy skill routes (which are
what the skillsets__* built-ins call), and the plugin skill editor’s tool
picker, which wrote the tool catalog’s mcp_name verbatim.
The failure mode is the one AGT-042 accepts for a foreign token, applied to
our own. The entry resolves to nothing, reports nothing missing, and the
skill stays listed and renders. Under progressive disclosure the model then
reads instructions naming tools it holds none of, because tools_required came
back empty. Every skill projected from a legacy skillset lost every dependency
this way, the skill-authoring skill included.
Two rules keep it from returning:
Every writer goes through
allowed_tool_entry, which qualifies only a name shaped like one of Seizu’s own and is safe to apply twice. The frontend mirror istoolDeclaration, and the editor normalizes on load, so opening a stale skill and saving it heals the declaration.parse_packagereports what it will not resolve. A bare Seizu tool name, anext__<proxy>__<tool>, and amcp__<server>__<tool>naming a servermcp.jsondoes not declare are each a warning, not an error: a wrong declaration is not a broken package, but it must not be an invisible one.
The projection is reconciled, not just created. _migrate_legacy_skillsets
re-projects every legacy skillset at startup and republishes when the digest
moved, so a change to what the projection writes reaches deployments whose
packages already exist. A package published over the projection is left alone.
An unchanged skillset re-projects to the same digest, so the settled cost is
still one read per skillset.
An ext__<proxy>__<tool> in a legacy skill stays in its legacy spelling.
The package form needs an mcp.json server entry, and a legacy skillset has
nowhere to carry one; synthesizing it from the configured proxy would take
skills on http upstreams from “this tool is not disclosed” to “this skill is
unavailable”, since mcp.json accepts only https or loopback URLs.
The seed ships skills as packages. The dev seed’s four legacy skillsets are
now plugins: entries with package sources, which is what let
cve_response/dependency_provenance declare the deps MCP it has always needed:
as a skillset it could not, so its four external tools were dropped on every
render. seizu export no longer writes an authored package back out as a
skillset either — the projection has nowhere to put mcp.json, scripts/ or
references/, so re-seeding that export would have replaced each package with a
lossy copy of itself. A package whose manifest carries the projection marker is
still exported as a skillset, because there the legacy record is the source.
A capability the listing does not mention is a capability the model will not
find. Under progressive disclosure the agent chooses from skill descriptions
and triggers; the body is only read once a skill is loaded. reports__delete
was declared by the create/update skill, where it exists for the clone-cleanup
step, and nothing in that skill’s description or triggers said “delete”. Asked
to delete a report, the agent read the listing, found no such capability and
answered that it had none — never loading the skill, so no confirmation was ever
offered and the failure looked like a missing tool. The package now has a
delete-reports skill that says so in its description and triggers. Declaring
the tool is necessary; advertising it is what makes it reachable.
Don’t: re-derive the qualified name at each call site. The contract is held
by tests/unit/reporting/services/allowed_tools_contract_test.py, which drives
every writer off the built-in registry rather than a fixed list, and by
seed_config_test.py, which resolves every seeded package’s declarations
against the registry and the seed’s own toolsets.
AGT-054 — Denial recovery has action and session bounds#
Applies to: action_confirmations, report_store.sql, ConfirmationPage
An identical action gets one extra prompt after denial by default. Five live
denials scoped to the user, source, and session refuse further prompts with
confirmation_denial_limit. Both limits are configurable; the window uses the
existing record expiry rather than a new timer or migration. Counts are SQL
aggregates over unexpired denied rows, without the UI list’s 500-row limit.
Existing approved grants remain consumable. Already outstanding confirmations
can still receive decisions; this is a denial budget, not a cap on concurrent
pending requests. A reversed denial no longer contributes to the live count.
Only the owner decision route enables denied-to-approved reversal, with an expiry- and status-conditional database update. Ownership implies this ability; execution still validates the caller’s current permissions and claims the grant once. MCP form continuations never enable reversal or create retry prompts in response to a decline; a fresh call is needed for the bounded retry.
Why: #314 showed that fingerprint stickiness locked out honest identical retries while changing one argument produced unlimited prompts. One extra prompt makes a mis-click recoverable with the smallest retry allowance, while the session budget survives argument variation. Five leaves room for several distinct actions without allowing an unbounded sequential denial loop. Owner reversal provides recovery after either budget is exhausted without adding a model-driven prompting surface. Exponential backoff would add state without bounding a patient caller. The existing TTL gives operators one window to tune.
The new budget refusal is an MCP error and a distinct chat block reason.
Existing pending/denied result isError semantics remain unchanged; changing
those for every client is separate from the denial policy in this issue.
AGT-055 — External input parks a call and resumes through an owner action#
Applies to: chat_elicitations, external_mcp, chat resume commands,
orchestrated step results, ChatElicitationCard
Interactive MCP 2026-07-28 input requests are stored as a bounded group, end the turn, and resume with the original arguments and opaque continuation state after the owner answers. Legacy callbacks still cancel. Form capability is advertised only for an opted-in interactive top-level or orchestrated call; discovery, detached runs, and sandbox subagents keep it stripped. URL capability remains available for AGT-048 recovery, including when interactive elicitation is disabled. Sampling and roots input requests are unsupported.
Why: a Temporal activity has no inbound form channel (AGT-008), and a legacy callback needs an open upstream session. The modern continuation survives both the activity ending and a browser reload without holding a worker slot. Reusing the confirmation pause keeps distributed worker results portable: the coordinator resumes the parked call without restarting the remote step’s model loop.
Response records are owner-scoped and forwarded through input_responses,
rather than inserted directly into model arguments or messages. Upstream tool
results may include those values and enter chat history and model context.
Forms warn users not to enter passwords, API keys, access tokens, or verification
codes. Credential collection belongs in URL elicitation on the external site.
Why: MCP forms are for ordinary input, not secrets. Substring redaction corrupted legitimate results, while length/type heuristics and exact matching could not guarantee confidentiality. Removing redaction preserves results and makes the UI and documentation state the actual data flow.
Form schemas remain a bounded flat primitive subset rendered as escaped text. URLs retain the operator-pinned origin and are revalidated with the proxy configuration before display. Global and per-proxy opt-ins do not make upstream schemas trusted UI.
Replay checks current permissions, current tool discovery and confirmation policy, and atomically consumes the group once. An approval needed during resume is linked to the continuation, so approval cannot run a fresh call. Declined, cancelled and expired requests are terminal without re-prompting. Resumed reads undergo normal verification, but are not retried.
Why: answering a form is not an authorization grant. Verification may reject evidence without repeating a call that may already have had effects. This refines AGT-048 only for owner-initiated interactive continuations; detached runs still never replay. A one-shot decline path needs no additional denial budget.
A sandbox delegation that unexpectedly elicits records the request and returns. After the owner answers, a later delegation may consume the continuation for the exact same tool, arguments, owner, thread and proxy. Its sandbox files and receipts survive; its earlier model loop does not (SBX-005, SBX-008).
Why: retaining or reconstructing a subagent transcript would introduce a second continuation lifecycle. The persistent sandbox already carries work into a new delegation.
Input-required tool details are awaiting input. A continuation records its outcome against the original elicitation group; the UI applies that outcome to the paused detail across turns and history reloads. Within a step’s own trace the outcome is folded into the parked entry rather than appended, so a call is one row carrying both its arguments and its result. An answered card confirms in place and then closes, before the turn it releases starts; dismissal is local to the view, so a reload brings an unclaimed card back and a delivery that never dispatches restores it immediately. Consumed cards are hidden, while accepted cards remain available until the continuation is claimed.
Why: consumption is not proof that an upstream operation succeeded. Using the recorded outcome avoids showing a failed continuation as successful, and retaining unconsumed answers preserves recovery after interrupted delivery. Appending instead replayed the wait beside its own answer on every reload, which live delivery never showed. Closing the card on the delivery promise holds it open for the whole turn: that promise settles when the turn does, not when it is dispatched.
AGT-057 — An elicitation resume offers the finish tool, and never answers with nothing#
Applies to: chat_graph._resume_elicited_tool_turn
The turn that summarizes an answered external-input request is offered
respond_to_user, and reads its answer from that call before falling back to
the message content. If both are empty it answers with the tool result itself
rather than persisting an empty assistant message.
Why: the resume turn is post-action, and the base system prompt requires a
post-action answer to arrive through respond_to_user. The turn ran with no
tools at all, so the model obeyed the prompt, its call was dropped with the
tools it was never given, and content was empty — one LLM call, a stream
carrying text-start/text-end with no delta, and an empty message in the
checkpoint. An empty assistant message is then dropped from history, so the
turn disappeared on reload and the parked tool detail, having no recorded
outcome to reconcile against (AGT-055), reverted to awaiting. Live delivery
showed the settled result and a reload took it away, which reads as the answer
never arriving. Observed intermittently, on whichever turns the model chose the
tool over prose. The confirmation resume never showed this because it already
falls back to the combined tool results.
Don’t: run a post-action summary turn with no terminal tool while the system prompt demands one, or let any resume path persist an empty answer — the message is what carries the turn, and history drops it when it is blank.