What happens
_wait_for_image_numbers in tests/mock_vws/test_database_summary.py polls GET /summary for up to 700 seconds, because, as its comment says, "the database summary endpoint lags behind the real data". Its retry predicate is:
retry=retry_if_exception_type(exception_types=(AssertionError,)),
tests/mock_vws/test_database_summary.py:42
So the loop retries a stale summary, but any transient failure of a poll — a read timeout, a 429, a 5xx — is not retried and propagates, ending the test with the wait budget mostly unspent.
Where it was seen
TestDatabaseSummary::test_active_images[Real Vuforia] failed twice in a row on #3505 with:
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='vws.vuforia.com', port=443): Read timed out. (read timeout=30.0)
The second attempt logged 758 polls, every one of them Actual: {'active_images': 0, 'inactive_images': 0, 'failed_images': 1, 'processing_images': 0} — the summary was serving a stale failed_images: 1 while the target the target_id fixture added was in fact successful. The test then died on a timed-out poll at roughly 415 seconds, so the remaining ~285 seconds of the 700-second allowance were never used.
Suggested resolution
Retry the same transient exceptions the rest of the suite already retries. tests/mock_vws/utils/retries.py:13 defines them:
TRANSIENT_VWS_EXCEPTIONS = (TooManyRequestsError, ServerError, RequestsTimeout)
Adding those to the predicate keeps a single flaky poll from throwing away the wait:
retry=retry_if_exception_type(
exception_types=(AssertionError, *TRANSIENT_VWS_EXCEPTIONS),
),Two things worth saying plainly. This is not guaranteed to turn the observed failures green — if the summary stays stale past 700 seconds the test fails anyway, and how long Vuforia takes to catch up is not something we control. And pytest-retry does already retry the whole test on these exceptions, but each retry restarts the 700-second clock and adds another target, which is both slower and more API traffic than retrying the one poll that failed.
The same pattern exists in a loop in tests/mock_vws/test_query.py (around line 2050), though with a 3-second window there is much less to lose.
Context
This surfaced because adding a ci_pattern entry shifts every later job's SECRET_INDEX (job-index % 100), so test_active_images moved from database 37 to database 38 — which test_failed_images previously used, and whose summary still reported that test's failed_images: 1. The lag is the underlying problem either way; the reshuffle just made it visible.
What happens
_wait_for_image_numbersintests/mock_vws/test_database_summary.pypollsGET /summaryfor up to 700 seconds, because, as its comment says, "the database summary endpoint lags behind the real data". Its retry predicate is:tests/mock_vws/test_database_summary.py:42So the loop retries a stale summary, but any transient failure of a poll — a read timeout, a 429, a 5xx — is not retried and propagates, ending the test with the wait budget mostly unspent.
Where it was seen
TestDatabaseSummary::test_active_images[Real Vuforia]failed twice in a row on #3505 with:The second attempt logged 758 polls, every one of them
Actual: {'active_images': 0, 'inactive_images': 0, 'failed_images': 1, 'processing_images': 0}— the summary was serving a stalefailed_images: 1while the target thetarget_idfixture added was in fact successful. The test then died on a timed-out poll at roughly 415 seconds, so the remaining ~285 seconds of the 700-second allowance were never used.Suggested resolution
Retry the same transient exceptions the rest of the suite already retries.
tests/mock_vws/utils/retries.py:13defines them:Adding those to the predicate keeps a single flaky poll from throwing away the wait:
Two things worth saying plainly. This is not guaranteed to turn the observed failures green — if the summary stays stale past 700 seconds the test fails anyway, and how long Vuforia takes to catch up is not something we control. And
pytest-retrydoes already retry the whole test on these exceptions, but each retry restarts the 700-second clock and adds another target, which is both slower and more API traffic than retrying the one poll that failed.The same pattern exists in a loop in
tests/mock_vws/test_query.py(around line 2050), though with a 3-second window there is much less to lose.Context
This surfaced because adding a
ci_patternentry shifts every later job'sSECRET_INDEX(job-index % 100), sotest_active_imagesmoved from database 37 to database 38 — whichtest_failed_imagespreviously used, and whose summary still reported that test'sfailed_images: 1. The lag is the underlying problem either way; the reshuffle just made it visible.