Skip to content

fix(core): forward nextToken when paging log streams in _flush_log_streams - #6447

Open
AkshayShah03 wants to merge 1 commit into
aws:masterfrom
AkshayShah03:fix/flush-log-streams-pagination
Open

AkshayShah03 wants to merge 1 commit into
aws:masterfrom
AkshayShah03:fix/flush-log-streams-pagination

Conversation

@AkshayShah03

Copy link
Copy Markdown

Issue #, if available: N/A

Description of changes:

_flush_log_streams pages through DescribeLogStreams while the response has a nextToken, but never passes the token back:

while "nextToken" in streams:
    streams = client.describe_log_streams(
        logGroupName=log_group, logStreamNamePrefix=job_name + "/",
        orderBy="LogStreamName", limit=50,
    )

Every call returns the first page again. For a job with more log streams than one page holds (more than 50 instances, or more streams than min(instance_count, 50)), the loop never ends and stream_names keeps growing, which hangs wait(logs=True) and log tailing.

This change passes nextToken=streams["nextToken"] in all three copies of this function: common_utils, helper/session_helper and remote_function/job.

Testing:

  • New tests/unit/test_flush_log_streams.py, parametrized over the three modules, uses a paginating fake client with 60 streams. All 60 are picked up in 2 calls. All three fail before this change.
  • The existing log-stream tests (5 files) pass: 470 passed.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…reams

_flush_log_streams pages through DescribeLogStreams while the response has
a nextToken, but never passed that token back. Each call returned the same
first page again, so for a job with more log streams than one page (more
than 50 instances, or more streams than min(instance_count, 50)) the loop
never ended and stream_names kept growing, hanging wait(logs=True) and log
tailing. Pass nextToken in all three copies (common_utils, session_helper
and remote_function.job).

This branch is waiting to be deployed

1 waiting deployment
manual-approval — aa6d5d76 Waiting Oct 12, 2026 by AkshayShah03 via wait-for-approval #2001
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant