Add --matrix-param argument to eval

[Nathan Selvidge (nselvidge)]

Adds the --matrix-param flag to bt eval

will create all combinations of arguments when multiple flags used.

--matrix-param hashes are appended to eval name and experiment name to ensure uniqueness and make it easier to differentiate between multiple experiments in the CLI and links.

 framework-evals git:(main) ✗ bt eval rewrite.eval.ts --matrix-param model=claude-haiku-4-5,claude-sonnet-4-6 --matrix-param includeSkill=true,false --sample 1
Processing 4 evaluators...
▶ Experiment Rewrite test [model="claude-sonnet-4-6", includeSkill=false] is running at https://www.braintrust.dev/app/braintrustdata.com/p/Rewrite%20Evals/experiments/Rewrite%20test%20%5Bmodel%3D%22claude-sonnet-4-6%22%2C%20includeSkill%3Dfalse%5D
▶ Experiment Rewrite test [model="claude-haiku-4-5", includeSkill=true] is running at https://www.braintrust.dev/app/braintrustdata.com/p/Rewrite%20Evals/experiments/Rewrite%20test%20%5Bmodel%3D%22claude-haiku-4-5%22%2C%20includeSkill%3Dtrue%5D
▶ Experiment Rewrite test [model="claude-haiku-4-5", includeSkill=false] is running at https://www.braintrust.dev/app/braintrustdata.com/p/Rewrite%20Evals/experiments/Rewrite%20test%20%5Bmodel%3D%22claude-haiku-4-5%22%2C%20includeSkill%3Dfalse%5D
▶ Experiment Rewrite test [model="claude-sonnet-4-6", includeSkill=true] is running at https://www.braintrust.dev/app/braintrustdata.com/p/Rewrite%20Evals/experiments/Rewrite%20test%20%5Bmodel%3D%22claude-sonnet-4-6%22%2C%20includeSkill%3Dtrue%5D
⠁ Rewrite Evals [exp...includeSkill=false]
⠁ Rewrite Evals [exp... includeSkill=true]
⠁ Rewrite Evals [exp...includeSkill=false]
⠁ Rewrite Evals [exp... includeSkill=true]  

If the value contains a comma, must use a JSON string containing an array

framework-evals git:(main) ✗ bt eval rewrite.eval.ts --matrix-param model="[\"claude-haiku-4-5\",\"claude-sonnet-4-6\"]"  --sample 1                                    
Processing 2 evaluators...
▶ Experiment Rewrite test [model="claude-sonnet-4-6"] is running at https://www.braintrust.dev/app/braintrustdata.com/p/Rewrite%20Evals/experiments/Rewrite%20test%20%5Bmodel%3D%22claude-sonnet-4-6%22%5D
▶ Experiment Rewrite test [model="claude-haiku-4-5"] is running at https://www.braintrust.dev/app/braintrustdata.com/p/Rewrite%20Evals/experiments/Rewrite%20test%20%5Bmodel%3D%22claude-haiku-4-5%22%5D
⠁ Rewrite Evals [exp...claude-sonnet-4-6"]
⠁ Rewrite Evals [exp..."claude-haiku-4-5"]                 

[github-actions]

Latest downloadable build artifacts for this PR commit a95ddd60198a:

• Workflow run: https://github.com/braintrustdata/bt/actions/runs/25065995534
• Download all artifacts (GitHub CLI): gh run download 25065995534 --repo braintrustdata/bt
• Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

• ``artifacts-build-global
• ``artifacts-build-local-x86_64-apple-darwin
• ``artifacts-build-local-x86_64-pc-windows-msvc
• ``artifacts-build-local-x86_64-unknown-linux-musl
• ``artifacts-build-local-x86_64-unknown-linux-gnu
• ``artifacts-build-local-aarch64-apple-darwin
• ``artifacts-build-local-aarch64-unknown-linux-gnu
• ``artifacts-plan-dist-manifest
• ``cargo-dist-cache

[Abhijeet Prasad (AbhiPrasad)]

I think tests/evals/js/eval-matrix-param-terminate/ needs a fixture.json.

[Nathan Selvidge (nselvidge)]

oops! added

[Abhijeet Prasad (AbhiPrasad)]

don't we need to honour the matrix on this path too? We should alteast print a warning if someone tries to use a matrix and they fall back into this path.

[Nathan Selvidge (nselvidge)]

thanks this is a good callout, I didn't even realize this existed. as I was thinking through how we would support it if felt like it would be a lot of added complexity. I think we can add this sort of support if needed but to keep things simple while nobody is using this feature I think we can ship without support for params on btEvalMain and add support if we get feedback from users. updated the PR to fail and log an error message if using params with btEvalMain