Skip to content

feat(analytics): print a session's agent metrics under --experimental-auth - #1023

Merged
Topherhindman merged 4 commits into
mainfrom
devx-798-cli-session-metrics
Oct 10, 2026
Merged

Topherhindman merged 4 commits into
mainfrom
devx-798-cli-session-metrics

Conversation

@Topherhindman

@Topherhindman Topherhindman commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes DEVX-798
Stacked on #1022.
Depends on livekit/public-api-server#53.

lk analytics session metrics SESSION_ID prints a page of the OpenTelemetry metric points a session's agents exported, one line per point. These are the raw metrics agents emit, so they won't exactly match the dashboard's metrics panel, and the help says so. The command runs only under --experimental-auth, since only the Public API serves agent metrics.

What changed

  • Regenerated client (chore(public)): oapi.gen.go is regenerated from livekit/public-api-server@3e3abf0, livekit/public-api-server#53's merge commit on main. It picks up GetSessionMetrics as the server now serves it:

    • agent metrics come back as typed MetricPoints instead of livekit MetricsBatches
    • each point carries its metric's name, unit and kind (gauge, sum, histogram or exponential histogram), its start and end times, and typed attributes with its resource's and scope's merged in
    • a point holds either a gauge's or sum's value, or a histogram's count, sum, min, max and buckets
    • the request gains a names filter (at most 50 names, each 1 to 256 characters), and points come by end time
    • the docs say an attribute integer beyond ±2^53 comes as its decimal string, and on an ACTIVE session a point exported mid-read behind the cursor is missed until a new read

    Nothing hand-written reads the changed types, so only oapi.gen.go changes.

  • Client (feat(public)):

    • GetSessionMetrics returns one page of points, by end time, and the cursor for the next.
    • MetricOptions embeds PageOptions and adds the metric names to keep, matched exactly. Validate rejects a negative limit or a blank name before any request is sent.
    • The generated point is a union that hides whether it holds a value or a histogram. So the body is decoded again into MetricPoints that say, as the transcript's items do. A point marshals back to the API's own JSON.
    • An unknown session is NotFound. An empty unfiltered first page with user data recording off carries the ObservabilityDisabled detail.
  • Command (feat(analytics)):

    • Lines: each point prints its end time to the millisecond, its metric's name, and a gauge's or sum's value or a histogram's count, sum, min and max, in the metric's unit. An annotation such as {token} loses its braces.
    • JSON: --json prints {items, nextCursor} with the points as the API sent them, kind, start time, typed attributes and histogram buckets included.
    • Filter: --name is repeatable and keeps only those metrics' points, matched exactly. A blank one fails before the project lookup.
    • Paging: the next-page hint says to re-run with its --cursor. The user's own command already repeats the names the cursor was made for.
    • Empty page: it says why: no points for the metrics asked for, no agent metrics at all (only agents that export OpenTelemetry metrics to LiveKit Cloud produce them), or no more pages.
    • Errors go through sessionReadError: with user data recording off it says so and links the project's observability settings. An unknown session says there's no such session in the project, and names the project. A permission denial says reading the agent metrics requires being a project admin.
    • Runs only under --experimental-auth, through sessionRead.

Usage

lk --experimental-auth analytics --experimental session metrics SESSION_ID \
    [--name NAME ...] [--limit N] [--json]
  • --limit defaults to 50, the server's page size. --cursor is hidden.
11:00:30.000  lk.agents.usage.llm_input_tokens  1520 token
11:00:30.250  lk.agents.turn.e2e_latency        count 3, sum 2.4 s, min 0.6 s, max 1.1 s
11:01:00.000  gen_ai.client.operation.duration  count 2, sum 1.5 s
More points available — re-run with --cursor c2

Since review

  • a17eb17 fix(analytics): strip terminal escapes from metric names and units

Testing

  • go build ./..., go vet ./pkg/... ./cmd/lk/ and go test ./pkg/... ./cmd/lk/ pass on every commit.
  • New tests:
    • pkg/public: TestGetSessionMetrics (values and histograms decoded), TestMetricPointJSON, TestGetSessionMetricsRejectsBadOptions and TestGetSessionMetricsErrors.
    • pkg/public/render: TestSessionMetricsText, TestSessionMetricsEmpty and TestSessionMetricsJSON.
    • cmd/lk: TestSessionMetricsCommand, TestSessionMetricsRequiresExperimentalAuth, TestMetricOptions, TestFetchSessionMetrics (each empty-page reason), TestFetchSessionMetricsJSON and TestFetchSessionMetricsErrors.
  • The command above was run on staging on 2026-10-08, against public-api-server's stack run locally, with a real sign-in and staging data.
  • Known issue on staging: the aggregator merges metric series across names. That's in the aggregator, not in this command, which prints the points as the API returns them.
  • Compared side by side with the staging dashboard on 2026-10-09 for two sessions: session detail, participants, events, transcript, agent logs and traces match, down to timestamps, turn latencies, redactions and log levels. The dashboard's "Dispatch" trace row is built in the browser from job_entrypoint's span events, which these reads return. Metrics are the known exception: the aggregator files several token and TTS series under one name, which the dashboard avoids by deriving usage from the transcript and traces.

@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch from 12c69e1 to 32f6772 Compare October 9, 2026 02:57
@Topherhindman
Topherhindman force-pushed the devx-797-cli-session-traces branch 2 times, most recently from 168fe22 to 8b6a3c3 Compare October 9, 2026 05:10
@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch from 32f6772 to 279bb7d Compare October 9, 2026 05:11
@Topherhindman
Topherhindman marked this pull request as ready for review October 9, 2026 06:27
@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch from 279bb7d to bbfc89f Compare October 9, 2026 20:26
@Topherhindman
Topherhindman force-pushed the devx-797-cli-session-traces branch from 8b6a3c3 to 48aeddc Compare October 9, 2026 20:26
@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch from bbfc89f to a17eb17 Compare October 9, 2026 20:50
@Topherhindman
Topherhindman force-pushed the devx-797-cli-session-traces branch from 48aeddc to 8c8d0b5 Compare October 9, 2026 20:50
@Topherhindman
Topherhindman force-pushed the devx-797-cli-session-traces branch from 8c8d0b5 to d920038 Compare October 10, 2026 03:11
@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch from a17eb17 to 411b38b Compare October 10, 2026 03:11
@Topherhindman
Topherhindman force-pushed the devx-797-cli-session-traces branch from d920038 to 7fe5029 Compare October 10, 2026 03:12
@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch 2 times, most recently from 79dae47 to 9b007b1 Compare October 10, 2026 03:48
@Topherhindman
Topherhindman force-pushed the devx-797-cli-session-traces branch 2 times, most recently from 8d0101c to fc1094c Compare October 10, 2026 04:21
@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch from 9b007b1 to 39ae30b Compare October 10, 2026 04:21
@Topherhindman
Topherhindman force-pushed the devx-797-cli-session-traces branch from fc1094c to c293aa8 Compare October 10, 2026 04:37
@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch from 39ae30b to 0eb7b58 Compare October 10, 2026 04:37
Base automatically changed from devx-797-cli-session-traces to main October 10, 2026 04:52
Picks up GetSessionMetrics as the server now serves it: the session's
agent metrics come back as typed MetricPoints instead of livekit
MetricsBatches. Each point carries its metric's name, unit and kind
(gauge, sum, histogram or exponential histogram), its start and end
times and its typed attributes with its resource's and scope's merged
in, plus either a gauge's or sum's value or a histogram's count, sum,
min, max and buckets. The request gains a names filter, which takes at
most 50 names, each 1 to 256 characters: an empty name is rejected.
Points come by end time; points that end together by start time, then
by kind and a hash of what they recorded, not by name.

The docs also say an attribute integer beyond ±2^53 comes as its
decimal string, and that on an ACTIVE session a point exported mid-read
with an earlier time than the read has reached is missed until a new
read. They name the upstream aggregator's limits too: several metrics
that share their attributes can come back under one name (an LLM's
input, output and cached token counts all as
lk.agents.usage.llm_input_tokens), points of one series at the same
time with equal values collapse into one, and with two agents in one
session a point can carry the other agent's resource attributes.

Nothing hand-written reads the changed types, so only oapi.gen.go
changes.

Generated from livekit/public-api-server@3e3abf0
GetSessionMetrics returns one page of the OpenTelemetry metric points a
session's agents exported, by end time, and the cursor for the next.
MetricOptions embeds PageOptions and adds the metric names to keep,
matched exactly, and Validate rejects a negative limit or a blank name
before any request is sent.

The generated point is a union that hides whether it holds a value or a
histogram, so the body is decoded again into MetricPoints that say, as
the transcript's items are: each carries its metric's name, unit and
kind, its start and end times and its typed attributes, and either a
gauge's or sum's Value or a histogram's count, sum, min, max and
buckets. A point marshals back to the API's own JSON. An unknown session
is NotFound, and an empty unfiltered first page with user data recording
off carries the ObservabilityDisabled detail.
…-auth

`lk analytics session metrics SESSION_ID` prints a page of the
OpenTelemetry metric points the session's agents exported, one line per
point: its end time to the millisecond, its metric's name, and a gauge's
or sum's value or a histogram's count, sum, min and max, in the metric's
unit (an annotation such as {token} loses its braces). --json prints
{items, nextCursor} with the points as the API sent them, kind, start
time, typed attributes and histogram buckets included.

--name is repeatable and keeps only the points of those metrics, matched
exactly; a blank one fails before the project lookup. --limit (default
50, the server's page size) and a hidden --cursor page through it, and a
next page says to re-run with its --cursor: the user's own command
already repeats the names a cursor was made for. The help says these are
the raw metrics agents emit, so they won't exactly match the dashboard's
metrics panel.

An empty page says why: no points for the metrics asked for, no agent
metrics at all (only agents that export OpenTelemetry metrics to LiveKit
Cloud produce them), or no more pages. With user data recording off it
says so and links the project's observability settings, an unknown
session says there's no such session in the project, and a permission
denial says reading the agent metrics requires being a project admin,
through the shared sessionReadError.

Only the Public API serves agent metrics, so the command runs only under
--experimental-auth, through sessionRead, with the signed-in user's
session token, and refuses to run otherwise before reading its
arguments.
A metric's name and unit are whatever the agent's code declared, and the
metrics read printed them as they came, so in a terminal an escape
sequence in one could set the window title or clear the screen. Both now
go through oneLine, which strips control characters, and the names are
measured after stripping so the values still line up.
@Topherhindman
Topherhindman force-pushed the devx-798-cli-session-metrics branch from 0eb7b58 to d73b173 Compare October 10, 2026 04:53
@Topherhindman
Topherhindman merged commit f4c36b0 into main Oct 10, 2026
26 checks passed
@Topherhindman
Topherhindman deleted the devx-798-cli-session-metrics branch October 10, 2026 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants