Skip to content

Fix audio lagging video: rebase surface PTS on the shared start, not the first frame #2180

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pedroSG94 merged 1 commit into from
Aug 12, 2026
Merged

Fix audio lagging video: rebase surface PTS on the shared start, not the first frame #2180

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 27 additions & 5 deletions encoder/src/main/java/com/pedro/encoder/video/VideoEncoder.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,6 +444,30 @@ public void formatChanged(@NonNull MediaCodec mediaCodec, @NonNull MediaFormat m
spsPpsSetted = sendSPSandPPS(mediaFormat);
}

/**
* Surface mode: the EGL timestamp is a huge absolute value that would break RTMP, so it has to be
* rebased to something relative. Rebase it against the start the encoders were given, not against
* this encoder's own first frame.
*
* GlStreamInterface stamps every frame through GlTimestamp, which anchors to TimeUtils — the same
* clock StreamBase reads to start both encoders — so subtracting presentTimeUs is meaningful and
* mirrors what AudioEncoder.calculatePts already does.
*
* Rebasing on the first frame instead put the two timelines on different zeros: audio counted from
* the moment the stream started, video from the moment its first frame came out of the encoder.
* Everything in between — opening the camera, warming up GL — turned into a fixed offset with audio
* behind the picture. On a Samsung SM-A065F that offset measured 470 ms (18 clap pairs, 427-517 ms),
* plainly visible as lips moving before the sound. It is the same in both TimestampMode branches,
* which is why switching modes never helped.
*
* Falls back to the old behaviour when the encoder was started without a shared origin.
*/
private long rebaseSurfacePts(long ptsUs) {
if (presentTimeUs > 0) return Math.max(0, ptsUs - presentTimeUs);
if (firstTimestamp == 0) firstTimestamp = ptsUs;
return Math.max(0, ptsUs - firstTimestamp);
}

@Override
protected boolean checkBuffer(@NonNull ByteBuffer byteBuffer, @NonNull MediaCodec.BufferInfo bufferInfo) {
if (forceKey && Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
Expand Down Expand Up @@ -510,12 +534,10 @@ protected boolean checkBuffer(@NonNull ByteBuffer byteBuffer, @NonNull MediaCode
// Buffer mode: synthesize PTS from wall clock.
bufferInfo.presentationTimeUs = TimeUtils.getCurrentTimeMicro() - presentTimeUs;
} else {
// Surface mode: EGL timestamp is camera sensor time (nanoseconds from boot ÷ 1000).
// It has clean, jitter-free intervals — but it's a huge absolute value that breaks RTMP.
// Rebase to relative by subtracting the first frame's PTS → clean intervals, starts at 0.
if (firstTimestamp == 0) firstTimestamp = bufferInfo.presentationTimeUs;
bufferInfo.presentationTimeUs -= firstTimestamp;
bufferInfo.presentationTimeUs = rebaseSurfacePts(bufferInfo.presentationTimeUs);
}
} else if (formatVideoEncoder == FormatVideoEncoder.SURFACE) {
bufferInfo.presentationTimeUs = rebaseSurfacePts(bufferInfo.presentationTimeUs);
} else {
if (firstTimestamp == 0) firstTimestamp = bufferInfo.presentationTimeUs;
bufferInfo.presentationTimeUs -= firstTimestamp;
Expand Down