Expose per-turn token usage and cost in the after_llm_call hook payload

[kimizuka]

Overview

Add per-turn token usage and cost to the after_llm_call hook payload. Today the payload carries only:

&hooks.Input{
    SessionID:       sess.ID,
    AgentName:       a.Name(),
    StopResponse:    responseContent,
    LastUserMessage: sess.GetLastUserMessageContent(),
}

This proposal adds, additively:

• usage — the per-call chat.Usage (input / cached input / cache write / output tokens), nested to avoid colliding with existing flat token fields
• cost — the per-call cost in USD
• model_id — the model actually used for this call (edit: already landed in fix(runtime): populate ModelID in after_llm_call hook payload #2911 — remaining scope is usage + cost only)

Motivation

There is currently no supported way to observe per-LLM-call usage or cost from outside the core. The only place spend is materialized is the session store, and the session store only records a sub-session when it completes successfully.

When a sub-session (delegated agent / skill / background agent) fails, its turns are dropped from persistence:

• pkg/runtime/agent_delegation.go runForwarding returns early on the first ErrorEvent (~line 289), so parent.AddSubSession(s) and SubSessionCompletedEvent are never reached.
• runCollecting has the same shape (returns on errMsg != "" before AddSubSession).
• The persistence observer only writes a sub-session in response to SubSessionCompletedEvent, so a failed sub-session's turns never reach the DB.

This is reasonable for the session DB (a session is resumable conversation state, and a failed sub-session isn't part of it). But the API calls were already billed. The cost the session reports can be far below the actual provider invoice.

Concretely: in a real run against Vertex AI with several failing sub-sessions, the actual invoice was ~9× the cost shown by the session. Each failed sub-session made multiple billed model calls that never appeared in any total. This makes spend impossible to reconcile and is a real safety concern when running against a metered provider.

The root cause is that cost recording is coupled to session lifecycle (SubSessionCompletedEvent). The cleanest place to expose spend is the layer where it actually happens — once per LLM call. executeAfterLLMCallHooks already fires inside the shared turn loop (pkg/runtime/loop.go:572) on every successful model call, for sub-sessions too, and before the sub-session failure handling in agent_delegation.go. A per-turn payload with usage and cost therefore captures failed sub-session spend automatically, with no coupling to session completion.

Use Cases

1. Spend reconciliation / cost ledger — a sidecar (agent YAML + a shell / Node / Python script) appends every billed turn to its own store, so the total matches the provider invoice even when sub-sessions fail.
2. Budget guard — a handler that warns or stops a run once cumulative cost crosses a threshold.
3. Alerting / telemetry — forward per-turn usage and cost to an external monitoring system, independent of session persistence.

Proposed Solution

Widen the after_llm_call payload at executeAfterLLMCallHooks to include usage and cost. The data already exists locally at the call site in the turn loop; this is plumbing, not new computation. With this primitive, a cost ledger lives entirely outside the core — no new core subsystem and no new storage owned by the core.

Because this widens a public, irreversible payload contract, the schema shape should be settled first. Open questions:

1. Unpriced vs free. A flat cost: 0 can't distinguish a free call from a model with no pricing. Prefer a structural signal (e.g. nullable cost = unpriced) over a documented "check usage != nil" convention.
2. Token field naming. Nesting under usage avoids collision with the existing compaction-related token fields — is this the preferred convention?
3. Coverage. Main runtime and harness paths are straightforward; compaction sub-runtimes and the chatserver / a2a / acp paths likely need follow-up. Should the initial change scope to the main loop?

I have a working implementation of the payload widening plus an example sidecar ledger, and am happy to open a PR once the schema is agreed.

Alternatives

• Internal Observer + built-in cost ledger (a new pkg/costlog package with its own SQLite store). This works and needs no public API change, but it makes the core own a new subsystem, storage format, and config surface, and is Go-only. The hook approach keeps the core to a primitive and lets the ledger be any language. (I have prototyped both.)
• Persist failed sub-sessions into the session DB. This conflicts with the intended meaning of the session store as resumable state, so it's a poorer fit.

Related Issues

• #504 Token tracking when using subagents (closed)
• #90 Cost tracking for multi-agent runs sometimes resets to zero (closed)
• #1345 Stats on usage + interaction summary (open)

Additional Context

Line references are against main at the time of writing and may drift. The proposal also reconciles the existing doc that says after_llm_call populates the model id, which today it does not. (edit: that was fixed in #2911; model_id is already populated. The remaining scope of this issue is usage + cost.)

[aheritier]

Triage summary

Category	Classification
Type	feature (enhancement)
Area	agent (runtime hooks / loop)
Status	needs design

This is a well-motivated feature request to widen the after_llm_call hook payload with per-turn usage, cost, and model_id fields. The core motivation is a real correctability gap: spend from failed sub-sessions is currently silently dropped from session totals, as the cost recording path is gated on SubSessionCompletedEvent. The author has a working implementation ready and correctly flags that the payload contract is public and irreversible, so the shape (nullable cost, field naming under usage, coverage scope) warrants design review before a PR is merged.

[kimizuka]

Quick correction to my own issue text first: model_id is already populated as of #2911 (merged May 29), so the remaining scope here is just usage and cost, and the "reconciles the doc/impl mismatch" note no longer applies. The upside is that #2911 is the exact template for this — it threaded modelID through executeAfterLLMCallHooks to both call sites and added after_llm_call_test.go, so adding two more fields follows the same path.

Concrete proposal for the shape:

1. Nullable cost

Cost is only computed when pricing exists (pkg/runtime/loop.go, in recordAssistantMessage):

if res.Usage != nil && m != nil && m.Cost != nil {
    messageCost = (...) / 1e6
}

So m.Cost != nil is itself the "priced?" signal. I'd expose it as a pointer:

Cost *float64 `json:"cost"`

• m.Cost == nil (unpriced / no pricing configured) → cost: null
• priced → non-nil, where 0 legitimately means a free call

This removes any need for a "check usage != nil" convention — null vs a number is unambiguous on the wire. I'd keep the key always present as explicit null rather than omitempty so "unpriced" stays a visible signal, but I'm happy to use omitempty if you prefer a lighter payload.

2. Field naming under usage

Reuse chat.Usage directly as a nested object rather than adding flat fields:

Usage *chat.Usage `json:"usage,omitempty"`

chat.Usage is already public and is what chat.Message.Usage serializes today, so the hook inherits the existing token schema verbatim (input_tokens / output_tokens / cached_input_tokens / cached_write_tokens / reasoning_tokens) — one canonical shape. hooks.Input has no existing flat token fields, so there's no collision; nesting is just for a single source of truth.

3. Coverage

Following #2911, I'd scope the initial PR to the two paths it already wired — the main run loop (loop.go:572) and the harness path (harness.go:192) — where res.Usage is in scope at the dispatch site. Compaction sub-runtimes and the chatserver / a2a / acp paths would be a documented follow-up. Since the fields live on the shared Input struct, the schema stays consistent: usage is simply absent and cost is null on paths that aren't wired yet.

Implementation note

The hook fires in runTurn, but cost is computed later in recordAssistantMessage. Same as #2911 threaded modelID, I'd compute the cost once and thread the single value into both the hook payload and recordAssistantMessage, so the hook's cost is guaranteed to equal the session's recorded cost.

Does this shape work for you? If so, I'll open the PR scoped to those two paths and extend after_llm_call_test.go to pin the null-vs-number contract.