Skip to content

feat(voice): add client-level realtime assistant - #2585

Merged
bobleer merged 5 commits into
GCWing:1.0.0-explorefrom
bobleer:bob/client-realtime-voice
Aug 28, 2026
Merged

feat(voice): add client-level realtime assistant#2585
bobleer merged 5 commits into
GCWing:1.0.0-explorefrom
bobleer:bob/client-realtime-voice

Conversation

@bobleer

@bobleer bobleer commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add a controller-local Volcengine full-duplex voice session with realtime PCM playback, cumulative-ASR snapshot rendering, and missing-audio fallback.
  • Let the realtime model inspect current client/workspace/session state, switch opened workspaces, start Agent tasks in the intended workspace, and stop the voice-owned task.
  • Convert public Agent text output into queued one-to-two-sentence progress speech with throttling and heartbeat updates.
  • Keep voice state at app scope and place the Voice trigger beside global search, replacing the top new-session shortcut.

Type and Areas

Type: Feature, bug fix, UI/UX, test

Areas: Desktop/Tauri, Rust services and contracts, Web UI, peer/remote policy, settings and i18n

Motivation / Impact

Users can hold a realtime voice conversation with BitFun, delegate work to an Agent in a named opened workspace, hear concise progress while work runs, and cancel the active voice-owned task. Interim speech recognition no longer repeats cumulative partial text, and streamed provider PCM audio is played continuously.

Verification

  • pnpm run check:web
  • pnpm run appearance:contract-audit
  • pnpm --dir src/web-ui run type-check
  • pnpm --dir src/web-ui exec vitest run src/app/components/NavPanel/unifiedSessionCreation.test.ts src/flow_chat/components/voice/realtimeVoiceAudio.test.ts src/flow_chat/components/voice/realtimeVoiceTranscript.test.ts src/flow_chat/components/voice/voiceTaskBridge.test.ts (19 tests passed)
  • cargo test -p bitfun-services-integrations --no-default-features --features speech-realtime --lib speech::realtime::tests:: (4 tests passed)
  • cargo test -p bitfun-core-types realtime_start_request_accepts_legacy_empty_payload (1 test passed)
  • cargo check -p bitfun-desktop
  • pnpm run check:core-boundaries
  • pnpm run check:repo-hygiene

No live Volcengine request was made; protocol and playback behavior were implemented against the supplied official duplex demo, and credentialed acceptance remains a manual follow-up.

Reviewer Notes

  • Realtime speech credentials and the provider session remain controller-local. Agent sessions still use the normal workspace adapters, including remote-capable session configuration.
  • Local desktop paths were compiled and unit-tested. Remote workspace and Peer Device execution were designed through existing adapters but were not runtime-exercised in this change.
  • Persisted voice settings and the realtime start request remain backward-compatible through defaults and legacy empty-payload coverage.

Checklist

  • This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
  • Relevant verification is recorded above, or skipped checks are explained.
  • User-facing strings, docs, and locales are updated where applicable.

@bobleer
bobleer merged commit 44cc4fc into GCWing:1.0.0-explore Aug 28, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant