Skip to content

Fix audio lagging video: rebase surface PTS on the shared start, not the first frame - #2180

Merged
pedroSG94 merged 1 commit into
pedroSG94:masterfrom
x270880x:fix/surface-pts-shares-audio-origin
Aug 12, 2026
Merged

Fix audio lagging video: rebase surface PTS on the shared start, not the first frame#2180
pedroSG94 merged 1 commit into
pedroSG94:masterfrom
x270880x:fix/surface-pts-shares-audio-origin

Conversation

@x270880x

Copy link
Copy Markdown
Contributor

In SURFACE mode VideoEncoder.checkBuffer rebases every frame on firstTimestamp — the PTS of the first frame this encoder produced. AudioEncoder.calculatePts rebases on presentTimeUs, the origin StreamBase hands to every encoder in startSources().

The two timelines therefore start at different moments: audio at the instant the stream started, video at the instant its first frame came out of the encoder. Whatever sits between them — opening the camera, warming up GL — becomes a constant offset, with the picture running ahead of the sound.

Measurement

Samsung SM-A065F, Android 15, camera source, built-in mic, RTMP to a local ffmpeg -listen 1 sink over an adb reverse tunnel. Method: clap into the camera, find the transient in the audio and the burst of motion in the video, compare.

build offset
2.7.2 video 436-540 ms ahead of audio
2.7.5 video 470 ms ahead (18 clap pairs, spread 427-517 ms)
2.7.5 + this fix 490 ms cluster gone; residual within the resolution of a clap test

It is plainly visible as lips moving before the sound is heard, which is how the users reported it.

Both TimestampMode branches perform the same rebase, so switching CLOCK/BUFFER changes nothing — I measured both and got the same figure. Worth noting because the container metadata is misleading here: with CLOCK the track start_time values look perfectly aligned while the actual A/V correspondence is still off by the same 470 ms.

Why rebasing on presentTimeUs is safe

GlStreamInterface stamps frames through GlTimestamp, which anchors the first frame to TimeUtils.getCurrentTimeNano() and clamps later frames to within one frame of that clock. StreamBase.startSources() reads the same TimeUtils to start the encoders. Both sides are CLOCK_BOOTTIME microseconds, so the subtraction is meaningful and mirrors what the audio path already does.

Scope

Only the SURFACE path changes. The buffer path keeps synthesizing from the wall clock, the non-surface BUFFER path keeps its own rebase, and if an encoder is ever started without a shared origin the helper falls back to the previous behaviour.

Happy to adjust the approach if you would rather solve it elsewhere — for instance by resetting firstTimestamp from startSources() — the important part is that the two encoders stop counting from different zeros.

In SURFACE mode VideoEncoder.checkBuffer rebases every frame on firstTimestamp —
the PTS of the first frame this encoder produced. AudioEncoder.calculatePts
rebases on presentTimeUs, the origin StreamBase hands to every encoder in
startSources(). The two timelines therefore start at different moments: audio at
the instant the stream started, video at the instant its first frame came out of
the encoder. Whatever sits between them — opening the camera, warming up GL —
becomes a constant offset, with the picture running ahead of the sound.

Measured on a Samsung SM-A065F (Android 15, camera source, built-in mic, RTMP):
video ran 470 ms ahead of audio, 18 clap pairs, spread 427-517 ms. Lips move,
speech follows. Both TimestampMode branches perform the same rebase, so switching
CLOCK/BUFFER makes no difference — verified by measuring both.

Rebasing on presentTimeUs is safe here: GlStreamInterface stamps frames through
GlTimestamp, which anchors the first frame to TimeUtils and clamps later frames to
within one frame of that clock, and StreamBase reads the same TimeUtils to start
the encoders. Both sides are on CLOCK_BOOTTIME microseconds, so the subtraction is
meaningful and matches what the audio path already does. Non-surface paths are
untouched, and the old behaviour still applies when the encoder was started
without a shared origin.

With the same test the 490 ms cluster disappears; what remains is inside the
resolution of a clap measurement.
@pedroSG94

Copy link
Copy Markdown
Owner

Hello,

Thank you for the PR. I did a fix to it because when you use a stream that no use gl but use surface render the timestamp could be allways 0. You can check it here:
#2181

Test it and let me know if all is working for you and I will merge all to master

@pedroSG94
pedroSG94 merged commit 52b7d14 into pedroSG94:master Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants