TEST Validate non-default dataset variants against real upstream artifacts

[romanlutz]

Is your feature request related to a problem? Please describe.

The daily real-upstream dataset sweep in tests/end_to_end/test_all_datasets.py parametrizes over registered SeedDatasetProvider classes and normally instantiates each class with no arguments. This validates each provider's default configuration, but it does not exercise constructor options that select a different upstream URL, Hugging Face split, or source field.

Current unit tests validate much of this wiring with mocks and fixture data, but those tests cannot detect upstream drift. A non-default artifact can disappear or change schema while the default artifact and daily sweep remain green.

The current untested matrix in this scope is:

• _AyaRedteamingDataset: Hindi, French, Spanish, Arabic, Russian, Serbian, and Tagalog. Each language selects a distinct JSONL URL; only English is the default.
• _CategoricalHarmfulQADataset: zh and vi. Each language selects a distinct Hugging Face split; only en is the default.
• _VLGuardDataset: VLGuardSubset.SAFE_UNSAFES and VLGuardSubset.SAFE_SAFES. Each subset selects a different instr-resp field and image/instruction contract; only UNSAFES is the default. VLGuard is gated and must retain the existing Hugging Face credential/terms behavior.

Closed #1789 proposed solving a broader version of this problem by registering 25 sibling provider classes, renaming default dataset names, and adding compatibility aliases/deprecations. That product/catalog change is not required to close this coverage gap and should not be revived here.

Describe the solution you'd like

Add an explicit parameterized real-upstream variant matrix using the current provider constructors. Each case should instantiate its existing provider directly with the non-default argument, fetch with caching disabled, and validate the artifact returned by that exact variant.

A test shape could use named constructor factories or typed case objects, for example:

_VARIANT_CASES = [
    pytest.param(
        lambda: _AyaRedteamingDataset(language="Hindi"),
        id="aya-hindi",
    ),
    pytest.param(
        lambda: _CategoricalHarmfulQADataset(language="zh"),
        id="categorical-harmful-qa-zh",
    ),
    pytest.param(
        lambda: _VLGuardDataset(subset=VLGuardSubset.SAFE_SAFES, token=os.environ.get("HUGGINGFACE_TOKEN")),
        id="vlguard-safe-safes",
    ),
]

Place the matrix in the test tier/job that performs the daily live-provider validation. Reuse the existing retry policy for transient network failures and the existing explicit skip behavior for unavailable gated credentials or unaccepted Hugging Face terms. Do not turn schema/contract failures into skips.

For every case, assert:

• the requested non-default URL, split, or subset is the one exercised;
• fetching produces a nonempty SeedDataset;
• every seed has a value and the provider's expected dataset name;
• language/subset metadata identifies the requested variant where the loader records it;
• expected text/image modalities and seed shapes are present;
• harm categories do not silently fall back because an upstream category changed;
• VLGuard image references resolve for both safe-image subsets and each selected instr-resp field remains valid.

Keep execution bounded using existing production-supported filters or test selection. Do not add public test-only constructor parameters solely to make the live test cheaper. If an image-heavy case cannot fit the existing daily job, document its measured cost and place it in the existing gated integration schedule rather than silently reducing the contract being tested.

This is test-only coverage. It must not:

• register sibling provider classes;
• add or rename public dataset names;
• add legacy-name aliases or deprecation warnings;
• change scenario/catalog selection semantics;
• change provider defaults;
• duplicate variants that already traverse the same upstream artifact and schema.

Describe alternatives you've considered, if relevant

Registering one provider subclass per variant would make the registry-driven sweep discover them automatically, but it would also expand and rename the public dataset catalog. That was the stale approach in closed #1789 and couples test discovery to a product API decision.

Expanding only mocked unit tests would verify argument forwarding but would still miss the intended failure class: upstream URLs, splits, fields, archives, images, and schemas drifting independently from the default variant.

Testing only one representative non-default variant per provider would reduce network cost, but it would leave other distinct URLs/splits/fields unchecked. The matrix should cover all 11 non-default variants listed above because each selects a distinct upstream artifact or contract.

Additional context

Suggested acceptance criteria:

• Seven non-English Aya cases pass against their real JSONL URLs.
• CategoricalHarmfulQA zh and vi pass against their real Hugging Face splits.
• VLGuard SAFE_UNSAFES and SAFE_SAFES pass against the gated real artifact when credentials and accepted terms are available.
• A case fails when its requested upstream artifact is missing, malformed, empty, or lacks required fields/media.
• Transient HTTP failures retain bounded retries; structural failures are not skipped.
• Existing default-provider coverage remains unchanged.
• No provider registration, public dataset name, scenario API, or compatibility surface changes.
• Unit tests remain network-free; live cases run only in the repository's real-upstream test tier.
• Ruff, formatting, relevant unit tests, and the live variant matrix pass.

This issue was identified by the September 2026 daily maintainability and resilience audits. It is intentionally narrower than closed #1789.

[chiruu12]

I'd like to take this.

Ran the three providers against real upstream before asking, to check the variant matrix is actually distinguishable rather than the same artifact under different arguments.

Aya returns a different artifact per language, so the case really is exercising the requested URL: English 987 seeds, Hindi 915, Tagalog 1009. CatQA returns 550 on each of en, zh and vi with different text, and metadata["language"] carries the requested code, so the split assertion you want has something to bind to. Those two I can verify end to end locally.

VLGuard I cannot. Both subsets 403 with GatedRepoError on an account with no token, which is the path _HF_GATED_PROVIDERS already skips in test_all_datasets.py. I can write the SAFE_SAFES and SAFE_UNSAFES cases against the same skip and terms-not-accepted branches, but the only thing I would be able to prove locally is that the skip fires, not that the instr-resp field and image references still resolve. Worth saying up front rather than implying I had tested it.

On the harm-category assertion: since a silent fallback is the failure you want caught, I plan to assert the resolved categories against the mapping the loader declares rather than just asserting the list is nonempty, so an upstream category rename fails instead of quietly landing on the default.

One thing to confirm before I start. Do you want the variant matrix as a second parametrized test inside tests/end_to_end/test_all_datasets.py, sharing _fetch_with_retry and the gated-skip constants, or as its own module in the same daily job? The first keeps the retry and skip policy in one place, which is the part I would not want to duplicate, but it does mean the file stops being one test per registered provider.

[romanlutz]

I can run VLGuard no problem!