Report DUT-related errors in fixtures as fail instead of error?

[Slarag]

Pytest has the really great approach that unhandeled exceptions or a call to pytest.fail in a fixture reports the test as error instead of failed.

We're testing devices (DUT, device under test) with a pytest-based framework. And before each test, we use fixtures to ensure that the device is in a certain state. If somehow the DUT fails to enter the correct state due to a DUT-related problem (which will raise a specific exception in the fixture) the test analysts want to see the according test marked as failed and not as error as the root-cause is DUT-related and not environment-related.

So far, I could not find an easy solution for this. Can I do this via a hook function? If yes, which hook function would be the correct one?

I've tried pytest_runtest_makereport as there I have acces to the call object after the test has run (or failed), but there I cannot alter call.result and calling pytest.fail inside this hook will raise an internal error and caues pytest to exit.

Another approach was to just catch the exception in a try/except block, append some error infomation to the request.node object and continue fixture setup. Then we would evaluate that information in the pytest_runtest_call hook and call pytest.fail from there. This works, but the big downside here is that other fixtures that might depend on the failed fixture are still being setup. And as the device is already in a bad state, those fixture might have undefined behavior. In this case I would need an option to avoid setting up other fixtures (without needing to add checks in each and every other fixture).

[PhantomBugz]

If your analysts mean pytest's terminal status and summary, pytest_report_teststatus can classify the specific DUT exception as a failure without letting fixture setup continue. Keep raising the exception normally; don't call pytest.fail() from a reporting hook.

For example, in conftest.py (replace the import with your actual exception type):

import pytest
from your_framework.errors import DUTSetupError


@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item, call):
    report = yield
    report.dut_setup_failure = (
        report.when == "setup"
        and report.failed
        and call.excinfo is not None
        and isinstance(call.excinfo.value, DUTSetupError)
    )
    return report


@pytest.hookimpl(tryfirst=True)
def pytest_report_teststatus(report):
    if getattr(report, "dut_setup_failure", False):
        return "failed", "F", "FAILED"

pytest_report_teststatus controls the terminal category, short letter and verbose status. The actual report still has when == "setup", so pytest does not run fixtures depending on the failed fixture or enter the test body. Unrelated setup/teardown exceptions keep their normal error classification.

The important limit is JUnit XML and other report consumers: this changes terminal presentation, not the test phase. The built-in JUnit plugin still emits an <error> for this setup failure. If your analysts consume XML, the corresponding classification needs to happen in that reporting layer; the hook above alone does not solve that part.

I checked on Python 3.12.10 / pytest 9.0.3 using a simulated DUT exception, dependent fixture, normal setup error, teardown error, assertion failure and passing test. The terminal summary changed from 1 failed, 2 passed, 3 errors to 2 failed, 2 passed, 2 errors; neither the dependent fixture nor the affected test body ran. JUnit counts stayed at three errors and one failure, and the exit code stayed 1. No real device or xdist run was involved.

References: pytest_report_teststatus, pytest_runtest_makereport.

Prepared with AI assistance; the behavior described above was checked locally.

[RonnyPfannschmidt]

Ai slop

[RonnyPfannschmidt]

Preliminary work for external test blockers is done in pytest-external-blockers

I haven't done the junitxml mapping there