Skip to content

fix(soniox): emit PREFLIGHT_TRANSCRIPT for preemptive LLM generation#5553

Merged
davidzhao merged 1 commit into
livekit:mainfrom
octo-patch:fix/issue-5536-soniox-preflight-transcript
Apr 25, 2026
Merged

fix(soniox): emit PREFLIGHT_TRANSCRIPT for preemptive LLM generation#5553
davidzhao merged 1 commit into
livekit:mainfrom
octo-patch:fix/issue-5536-soniox-preflight-transcript

Conversation

@octo-patch
Copy link
Copy Markdown
Contributor

Fixes #5536

Problem

The Soniox STT plugin only emits INTERIM_TRANSCRIPT events, so AgentSession's preemptive LLM generation never fires when using Soniox for STT-driven end-of-turn detection. Users pay full LLM TTFT in wall-clock time after every turn, which is too high for real-time voice use cases.

Solution

When a Soniox token batch contains only final tokens (no non-final/speculative tokens pending), speech has reached a stable state — this indicates a likely pause, analogous to Deepgram v2's EagerEndOfTurn signal. In that case, emit PREFLIGHT_TRANSCRIPT instead of INTERIM_TRANSCRIPT.

This allows AgentSession to start speculative LLM generation. If the user resumes speaking, the next batch will contain non-final tokens, triggering a regular INTERIM_TRANSCRIPT that aborts the preemptive generation — the same abort path used by Deepgram v2's TurnResumed event.

The change is minimal: one conditional in the event-type selection within _recv_messages_task.

Testing

Tested by code review against the Deepgram v2 pattern (EagerEndOfTurnPREFLIGHT_TRANSCRIPT). The logic mirrors that behavior using the token finality heuristic native to Soniox's streaming API.

When a Soniox token batch contains only final tokens (no non-final/speculative
tokens), speech has reached a stable state indicating a likely pause. Emit
PREFLIGHT_TRANSCRIPT instead of INTERIM_TRANSCRIPT in this case to allow
AgentSession to start preemptive LLM generation, matching the latency benefit
already available via Deepgram v2's EagerEndOfTurn event.

Fixes livekit#5536
@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


octo-patch seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 2 additional findings.

Open in Devin Review

@davidzhao davidzhao merged commit 2e96931 into livekit:main Apr 25, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Soniox STT plugin: add PREFLIGHT_TRANSCRIPT support for preemptive LLM generation

3 participants