Fix audio lagging video: rebase surface PTS on the shared start, not the first frame - #2180
Merged
pedroSG94 merged 1 commit intoAug 12, 2026
Conversation
In SURFACE mode VideoEncoder.checkBuffer rebases every frame on firstTimestamp — the PTS of the first frame this encoder produced. AudioEncoder.calculatePts rebases on presentTimeUs, the origin StreamBase hands to every encoder in startSources(). The two timelines therefore start at different moments: audio at the instant the stream started, video at the instant its first frame came out of the encoder. Whatever sits between them — opening the camera, warming up GL — becomes a constant offset, with the picture running ahead of the sound. Measured on a Samsung SM-A065F (Android 15, camera source, built-in mic, RTMP): video ran 470 ms ahead of audio, 18 clap pairs, spread 427-517 ms. Lips move, speech follows. Both TimestampMode branches perform the same rebase, so switching CLOCK/BUFFER makes no difference — verified by measuring both. Rebasing on presentTimeUs is safe here: GlStreamInterface stamps frames through GlTimestamp, which anchors the first frame to TimeUtils and clamps later frames to within one frame of that clock, and StreamBase reads the same TimeUtils to start the encoders. Both sides are on CLOCK_BOOTTIME microseconds, so the subtraction is meaningful and matches what the audio path already does. Non-surface paths are untouched, and the old behaviour still applies when the encoder was started without a shared origin. With the same test the 490 ms cluster disappears; what remains is inside the resolution of a clap measurement.
Owner
|
Hello, Thank you for the PR. I did a fix to it because when you use a stream that no use gl but use surface render the timestamp could be allways 0. You can check it here: Test it and let me know if all is working for you and I will merge all to master |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In SURFACE mode
VideoEncoder.checkBufferrebases every frame onfirstTimestamp— the PTS of the first frame this encoder produced.AudioEncoder.calculatePtsrebases onpresentTimeUs, the originStreamBasehands to every encoder instartSources().The two timelines therefore start at different moments: audio at the instant the stream started, video at the instant its first frame came out of the encoder. Whatever sits between them — opening the camera, warming up GL — becomes a constant offset, with the picture running ahead of the sound.
Measurement
Samsung SM-A065F, Android 15, camera source, built-in mic, RTMP to a local
ffmpeg -listen 1sink over anadb reversetunnel. Method: clap into the camera, find the transient in the audio and the burst of motion in the video, compare.It is plainly visible as lips moving before the sound is heard, which is how the users reported it.
Both
TimestampModebranches perform the same rebase, so switching CLOCK/BUFFER changes nothing — I measured both and got the same figure. Worth noting because the container metadata is misleading here: with CLOCK the trackstart_timevalues look perfectly aligned while the actual A/V correspondence is still off by the same 470 ms.Why rebasing on
presentTimeUsis safeGlStreamInterfacestamps frames throughGlTimestamp, which anchors the first frame toTimeUtils.getCurrentTimeNano()and clamps later frames to within one frame of that clock.StreamBase.startSources()reads the sameTimeUtilsto start the encoders. Both sides are CLOCK_BOOTTIME microseconds, so the subtraction is meaningful and mirrors what the audio path already does.Scope
Only the SURFACE path changes. The buffer path keeps synthesizing from the wall clock, the non-surface BUFFER path keeps its own rebase, and if an encoder is ever started without a shared origin the helper falls back to the previous behaviour.
Happy to adjust the approach if you would rather solve it elsewhere — for instance by resetting
firstTimestampfromstartSources()— the important part is that the two encoders stop counting from different zeros.