feat: OpenAI realtime api by vladimir-a-sap · Pull Request #935 · SAP/ai-sdk-java

vladimir-a-sap · 2026-07-07T18:07:18Z

Context

AI/ai-sdk-java-backlog#386.

Adds OpenAI realtime API support

Feature scope:

Definition of Done

Functionality scope stated & covered
Tests cover the scope above
Error handling created / updated & covered by the tests above
~~Aligned changes with the JavaScript SDK~~
Documentation updated
Release notes updated

CharlesDuboisSAP

What do we do about the previous realtime PR? Is it still valid? Why is it abandonned?

#870

vladimir-a-sap · 2026-07-08T08:39:17Z

What do we do about the previous realtime PR? Is it still valid? Why is it abandonned?

feat: PoC OpenAI Realtime API #870

My current understanding is that the mentioned PR cannot be completed to a SDK realtime solution. Here is the context and outcomes of my research why I think so:

There are three ways to implement realtime API with OpenAPI:

WebRTC
Websockets
SIP (telephony)

The previous PR implements SIP - it is a way for client and server to communicate about a connection in order to pass data, but not to establish connection or pass data in fact. The data layer should be implemented separately from the SIP protocol.

OpenAPI SIP protocol by design works with client-side web hooks as a mean to pass data and accept incoming connections (calls). Outgoing calls (client initiated) are not seem to be possible at all using solely SIP API.

We cannot expect SDK environment to have public http endpoints accessible from the internet to expose hooks. AI Core does not implement hook functionality either AFAIK.

Given that WebRTC is difficult to support on the BE and, AFAIK, our backend only supports Websockets at the moment, Websockets seem to be the only option we have.

…ophone api

Jonas-Isr

The design is very cool! Thanks a lot for the detailed investigation and work :) I am reviewing this as if it would be the beta release already because I think without too much effort we turn this into a releasable state.

The two points below (and many of my comments) concern code conventions in the team.

General points:

Please add @Beta to all newly introduced public API. This way we have a grace period to change the API if we overlooked something in the beginning. We periodically re-evaluate and remove these annotations later on.
Please don’t put hand written classes into any of the model/ packages. These are only for generated code. If possible, put the SpeechOutputParam interface and the 2 classes implementing it into the open package.

vladimir-a-sap · 2026-07-17T13:35:03Z

The design is very cool! Thanks a lot for the detailed investigation and work :) I am reviewing this as if it would be the beta release already because I think without too much effort we turn this into a releasable state.

The two points below (and many of my comments) concern code conventions in the team.

General points:

Please add @Beta to all newly introduced public API. This way we have a grace period to change the API if we overlooked something in the beginning. We periodically re-evaluate and remove these annotations later on.

Please don’t put hand written classes into any of the model/ packages. These are only for generated code. If possible, put the SpeechOutputParam interface and the 2 classes implementing it into the open package.

addressed both these points in addition to your comments in the PR

CharlesDuboisSAP · 2026-07-24T07:41:12Z

+    <dependency>
+      <groupId>com.openai</groupId>
+      <artifactId>openai-java</artifactId>
+    </dependency>


Users of the other OpenAI clients should not have to import this dependency.
This should be optional like the existing openai-java-core that is already in this pom.

CharlesDuboisSAP · 2026-07-24T07:42:25Z

+import javax.annotation.Nullable;
+
+/** Allows to configure turn detection (how model responds). */
+public final class RealtimeParamTurnDetection implements RealtimeParam {


Why are realtime classes in the core module.
If they cannot be used outside of the realtime client they belong in the same folder as the realtime client

CharlesDuboisSAP · 2026-07-24T07:43:45Z

@@ -0,0 +1,18 @@
+package com.sap.ai.sdk.foundationmodels.openai;


(Minor) We had a discussion that it could be great to put every class in its own realtime folder to make it easier to maintain and not make the same mistake that we did in the orchestration package that became very crowded.
package com.sap.ai.sdk.foundationmodels.openai.realtime

missed/forgot the part about the dedicated sub-package

thanks, now moved!

…itly used

…java into feat/poc-openai-realtime-api

Jonas-Isr

The ability to try it out in the sample app is really cool!

Besides Charles' points and the smaller comments below my main concern is test coverage. Can we maybe add some unit tests covering WSOpenAiRealtimeClient, ToAudioRealtimeClient, and the URL construction in OpenAiRealtimeClient.getRealtimeEndpoint()?

Jonas-Isr · 2026-07-24T10:49:01Z

+    final var extraHeaders = destination.asHttp().getHeaders();
+    final var headers = new HashMap<String, String>(extraHeaders.size() + 1);
+    for (Header header : extraHeaders) {
+      headers.put(header.getName(), header.getValue());
+    }


(Minor)

Would maybe be a bit cleaner to put this in a private method as it is used also in speechToSpeech().

Jonas-Isr · 2026-07-24T10:53:15Z

+    try {
+      ws = this.ws.join();
+    } catch (final CompletionException e) {
+      throw new ClientException("failed to establish web socket connection", e);


(Very Minor)

Also for other exceptions.

Suggested change

throw new ClientException("failed to establish web socket connection", e);

throw new ClientException("Failed to establish web socket connection", e);

Jonas-Isr · 2026-07-24T10:53:42Z

+      try {
+        ws.sendText(JACKSON.writeValueAsString(ResponseCreateEvent.builder().build()), true);
+      } catch (final JsonProcessingException e) {
+        throw new RuntimeException("Failed to serialize ask for response", e);


(Question)

Shouldn't this be a ClientException like the others?

Jonas-Isr · 2026-07-24T10:59:35Z

+
+  private static final int MAX_DATA_CHUNK_SIZE_BYTES = 8192;
+
+  private static final String TASK = "";


(Question)

I don't understand the role of TASK here. Is it always empty? Or can it be set by a user?

Jonas-Isr · 2026-07-24T11:00:59Z

+      @Nonnull final BiConsumer<WebSocket, CharSequence> onText) {
+    this.onOpen = onOpen;
+    this.onText = onText;
+    this.buffer = new StringBuilder(1024 * 1024);


(Question)

How many are we expecting users to create of these? Will this size be too large?

Context
This is now used for responses. There is at least three levels of data segmentation handled by software (not going too deep into networking):

TCP level - packet partitioning/glueing - handled by operation system

WebSocket level - partitioning made by the server, packet glueing is done by this buffer

User application level data partitioning (server sends sound.delta messages in the web socket channel) - managed by the application using the realtime-client/SDK (either to wait and consume fully or play deltas/increments)

It didn't make sense to expose level 2 glueing logic to the app since partial websocket messages would be broken data-wise (not full json) and wouldn't provide any value, but would complicate the SDK exposed API

Response to the question
Users are not intended to create these buffers directly (since package local), but implicitly these buffers will be created once for every open realtime session. The total number of buffers created will depend on how many open realtime conversations a user would want at the same time. This can't normally be above ~64.5k per network interface and limited by the number of available TCP sockets.

I experimented with different requests/responses and most model audio responses were inside 1m range with short responses being typically 200-300k (20-30% of the allocated hardcoded capacity).

This might be slightly over for text responses, but two our use cases (Text-to-Speech and Speech-to-Speech) now output audio along with supplementary text.

We may later add ways to configure this if it will be needed

UPDATE:
I experimented a bit more and it seems that in practice this buffer size can be significantly reduced indeed. I set 128k instead of the previous 1m for now.

Jonas-Isr · 2026-07-24T11:03:01Z

+ * OpenAI client implementation of Realtime API. Abstracts technical implementation, transport and
+ * threading and exposes business-level operations (high level interface)
+ */
+public class OpenAiRealtimeClient {


(Major)

@Beta missing on a new public class. Same for AudioInputChannel, AudioOutputChannel, TextInputChannel, TextOutputChannel as far as I see.

Jonas-Isr · 2026-07-24T11:07:27Z

+    for (RealtimeParam param : params) {
+      if (param.getParamName() == RealtimeParam.SpeechOutputParamName.TURN_DETECTION) {
+        if (RealtimeParamTurnDetection.EACH_CALL_IS_A_TURN.equals(param)) {
+          turnDetectionEager = true;
+        } else if (RealtimeParamTurnDetection.BY_MODEL_AUTO.equals(param)) {
+          turnDetectionEager = false;
+        }
+      }
+    }
+    this.eagerTurnDetection = turnDetectionEager;


(Minor)

As far as I can see this is the same as is done in the constructor of the SpeechToSpeechRealtimeClient. Maybe it makes sense to put it into ToAudioRealtimeClient as utility method?

Jonas-Isr · 2026-07-24T11:09:46Z

+                            <h2>⏰ Realtime API</h2>
+                        </div>
+                        Realtime API allows for various real time interactions <a
+                            href="https://todo.todo"


(Minor)

Do we have "official" docs somewhere that we can link here?

…java into feat/poc-openai-realtime-api

vladimir-a-sap added 4 commits July 7, 2026 14:53

add realtime text to speech client, WIP

6f303a3

polish minimap text-to-speech realtime poc

ecd953e

add incomplete interim packets buffering and glueing

71d4a4c

remove obsolete debug constants

232725d

vladimir-a-sap requested review from CharlesDuboisSAP and Jonas-Isr July 7, 2026 18:07

Formatting

6b872d4

vladimir-a-sap requested a review from n-o-u-r-h-a-n July 7, 2026 19:22

CharlesDuboisSAP reviewed Jul 8, 2026

View reviewed changes

Comment thread orchestration/src/main/java/com/sap/ai/sdk/orchestration/OrchestrationClient.java Outdated

Comment thread orchestration/src/main/java/com/sap/ai/sdk/orchestration/TextToSpeechRealtimeClient.java Outdated

CharlesDuboisSAP reviewed Jul 8, 2026

View reviewed changes

vladimir-a-sap and others added 8 commits July 10, 2026 14:56

add speech to speech realtime support and text to speech example

54b06a5

merge changes from upstream

df2098a

add speech to speech demo scenario

80a5d3e

add smoke test for realtime api and configurability

d6ad031

rework sample application so it no longer use deprecated browser micr…

2ed77e0

…ophone api

make realtime implementation package local

c212c1e

Merge branch 'main' into feat/poc-openai-realtime-api

d144a3a

Formatting

03d1e70

vladimir-a-sap changed the title ~~DRAFT: Feat/poc OpenAI realtime api~~ Feat/poc OpenAI realtime api Jul 13, 2026

vladimir-a-sap changed the title ~~Feat/poc OpenAI realtime api~~ feat: OpenAI realtime api Jul 13, 2026

vladimir-a-sap requested a review from CharlesDuboisSAP July 13, 2026 16:48

vladimir-a-sap marked this pull request as ready for review July 13, 2026 16:48

Jonas-Isr reviewed Jul 16, 2026

View reviewed changes

vladimir-a-sap mentioned this pull request Jul 17, 2026

feat: PoC OpenAI Realtime API #870

Closed

9 tasks

vladimir-a-sap and others added 2 commits July 17, 2026 15:32

address pr feedback, add tests

5415795

Formatting

a0cd474

Merge branch 'main' into feat/poc-openai-realtime-api

25d673b

Formatting

611c0eb

CharlesDuboisSAP reviewed Jul 24, 2026

View reviewed changes

vladimir-a-sap and others added 7 commits July 24, 2026 12:10

use open-ai-core sub dependency as nothing from open-ai dep is explic…

b90ab43

…itly used

Merge branch 'feat/poc-openai-realtime-api' of github.com:SAP/ai-sdk-…

4ad44ff

…java into feat/poc-openai-realtime-api

address code style pmd issues

dc53160

Formatting

c6a0e94

address code style pmd issues

1400efc

Merge branch 'feat/poc-openai-realtime-api' of github.com:SAP/ai-sdk-…

86ab775

…java into feat/poc-openai-realtime-api

Formatting

51b35a0

Jonas-Isr reviewed Jul 24, 2026

View reviewed changes

vladimir-a-sap and others added 20 commits July 24, 2026 15:35

reduce default response buffer size

f92d327

Merge branch 'feat/poc-openai-realtime-api' of github.com:SAP/ai-sdk-…

a0e5eef

…java into feat/poc-openai-realtime-api

move realtime api classed to a dedicated package

ff3007b

Formatting

2060668

fix pmd issues

1e74df5

Merge branch 'feat/poc-openai-realtime-api' of github.com:SAP/ai-sdk-…

fc6bd75

…java into feat/poc-openai-realtime-api

address spotbugs plugin issues

0fb2646

Formatting

54b151d

address dependency issue

f6698c5

Merge branch 'feat/poc-openai-realtime-api' of github.com:SAP/ai-sdk-…

f36c8c7

…java into feat/poc-openai-realtime-api

fix checkstyle issues

40971d4

Formatting

7f656aa

fix checkstyle issues

7eed889

Merge branch 'feat/poc-openai-realtime-api' of github.com:SAP/ai-sdk-…

f7def36

…java into feat/poc-openai-realtime-api

Formatting

a3d4641

fix checkstyle issues

bd8cb90

Merge branch 'feat/poc-openai-realtime-api' of github.com:SAP/ai-sdk-…

4d7cf2c

…java into feat/poc-openai-realtime-api

Formatting

7e008bb

fix checkstyle issues

b36ed95

Merge branch 'feat/poc-openai-realtime-api' of github.com:SAP/ai-sdk-…

143c01f

…java into feat/poc-openai-realtime-api

		@@ -0,0 +1,18 @@
		package com.sap.ai.sdk.foundationmodels.openai;

	throw new ClientException("failed to establish web socket connection", e);
	throw new ClientException("Failed to establish web socket connection", e);


		private static final int MAX_DATA_CHUNK_SIZE_BYTES = 8192;

		private static final String TASK = "";

Uh oh!

Conversation

vladimir-a-sap commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Feature scope:

Definition of Done

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CharlesDuboisSAP left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vladimir-a-sap commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jonas-Isr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladimir-a-sap commented Jul 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jonas-Isr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vladimir-a-sap Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

vladimir-a-sap commented Jul 7, 2026 •

edited

Loading

CharlesDuboisSAP left a comment •

edited

Loading

vladimir-a-sap commented Jul 8, 2026 •

edited

Loading

vladimir-a-sap Jul 24, 2026 •

edited

Loading