[Proposal] OpenEval Import/Export Support

[adhabnr-ux]

Overview

We've just launched OpenEval (v1.0.0-rc.1), an open standard (Apache 2.0) for portable LLM evaluation datasets.

The problem: Every eval framework uses incompatible formats. You can't share eval datasets across DeepEval, Promptfoo, Inspect AI, LangSmith, etc. without manual conversion.

OpenEval solves this with a standard JSON format + converters + SDKs for TypeScript and Python.

Why Braintrust Should Support It

Your users could import/export eval suites across frameworks. We've built converters for Promptfoo, DeepEval, and Inspect AI. Braintrust already has a generic eval format — OpenEval alignment reduces friction for users who also use other tools.

What We're Proposing

Add to_openeval() and from_openeval() methods using our SDK (~50-100 lines).

Resources

• Spec: https://github.com/adhabnr-ux/openeval/blob/main/spec/SPEC.md
• npm SDK: npm install @openeval/sdk
• PyPI SDK: pip install openeval

We'd love to collaborate. Happy to submit a PR if you'd like.

[adhabnr-ux]

Update: OpenEval packages are now published! npm install openeval-sdk / pip install openeval-sdk. Active responses from Inspect AI, CrewAI, and Arize. Would love to collaborate.

[lforst]

hi, can you share a concrete example on how this would be useful for braintrust users?

[adhabnr-ux]

Hi Luca! Great question — here's a concrete example.

Scenario: A Braintrust user has built a 200-case RAG evaluation dataset for their customer support chatbot. They've been running it in Braintrust for months. Now their colleague wants to try the same eval suite in DeepEval (or Promptfoo, or Inspect AI) to compare results across frameworks.

Without OpenEval: They'd have to manually rewrite all 200 test cases + scoring criteria in the target framework's format. This takes hours and is error-prone.

With OpenEval:

# Export from Braintrust to OpenEval
from braintrust import Eval
from openeval.validate import validate_suite

eval = Eval(...)  # existing Braintrust eval
suite = {
    "version": "1.0.0",
    "id": "braintrust_rag_export",
    "test_cases": [
        {"id": tc.id, "input": tc.input, "expected_output": tc.expected, "graders": ["gr_relevancy"]}
        for tc in eval.dataset
    ],
    "graders": [{"id": "gr_relevancy", "type": "semantic_similarity", "params": {"threshold": 0.75}}],
}
validate_suite(suite)  # verify it's valid OpenEval

# Now this suite.json can be imported into DeepEval, Promptfoo, etc.

Why this matters for Braintrust specifically:

1. Braintrust already has a generic eval format — OpenEval alignment would make it portable to other tools without Braintrust users leaving your platform
2. Import from other tools — users on DeepEval/Promptfoo could bring their eval datasets INTO Braintrust ("I built this in Promptfoo, now I want Braintrust's dashboard")
3. Benchmark sharing — community-published OpenEval benchmark datasets could be imported into Braintrust for evaluation
4. Result comparison — OpenEval ResultSets from different tools could be compared in Braintrust's UI

The implementation would be ~50 lines: two functions (to_openeval() and from_openeval()) using our Python SDK (pip install openeval-sdk).

Does this help clarify? Happy to build a prototype if you're interested.

[lforst]

OpenEval alignment would make it portable to other tools without Braintrust users leaving your platform

Interesting way to sell this 😂

We have no intention of vendor locking folks, though right now this is not a big priority and probably not worth the API surface. We'll think about it if we get more inquiries about this.

[adhabnr-ux]

Thanks for the honest feedback, Luca! Totally understand — not a priority right now. We'll keep the door open. If more users ask about cross-framework eval portability, you know where to find us. 👋