Issue
Summary
I set up the benchmark from a fresh clone of v0.100.12 (commit c92e7dc) following benchmark/README.md, intending to benchmark newer models against the polyglot exercises. I hit a chain of five distinct problems before getting clean end-to-end runs. Reporting them together since they were found in sequence on one setup and may share context; happy to split into separate issues if you prefer, and happy to open PRs for any of these if the direction I took locally is actually the right one — I have not validated my workarounds beyond "the harness then ran," so please treat them as diagnostic hints, not proposed fixes I also totally relied on Claude Fable to do all the fixes and summarize this issue.
Environment: Ubuntu 24.04 host, rootless podman via podman-docker, benchmark image from benchmark/docker_build.sh (Python 3.11 in-container), exercises from ErichBSchulz/cecli-cats (225 cats discovered), models served over an OpenAI-compatible endpoint (Ollama Cloud) via OPENAI_API_BASE/OPENAI_API_KEY.
1. README setup path leads to "Found 0 cats" / a non-functional classic harness
The README's setup section clones Aider-AI/polyglot-benchmark, but that repo contains no cat.yaml files, so benchmark.py reports Found 0 cats / No exercise directories found. Falling back to benchmark_classic.py (which defaults to that exercises dir) fails immediately on every case:
File "benchmark/benchmark_classic.py", line 1055, in run_test_real
dump(coder.ignore_mentions)
AttributeError: 'coroutine' object has no attribute 'ignore_mentions'
RuntimeWarning: coroutine 'Coder.create' was never awaited
which looks like the classic harness predates the async Coder API (I see the roadmap lists updating the test harness for async). I eventually found ErichBSchulz/cecli-cats as a Cats-formatted exercise set and moved on with benchmark.py. Ask: could the README document the intended exercises source for the Cats harness, and/or state whether benchmark_classic.py is deprecated?
2. benchmark.py: ImportError on every test case
File "benchmark/benchmark.py", line 993, in run_test_real
from cecli.main import SwitchCoderSignal
ImportError: cannot import name 'SwitchCoderSignal' from 'cecli.main'
Every case fails at setup, pre-API-call. I noticed the test suite imports this from cecli.commands, and changing the import there let the run proceed — but I don't know whether that's the intended home for it.
3. benchmark.py: args stub missing attributes the Coder reads
With (2) worked around:
File "benchmark/benchmark.py", line 1200, in run_test_real
coder.show_announcements()
File "cecli/cecli/coders/base_coder.py", line 925, in get_announcements
if not self.args.tui:
AttributeError: 'types.SimpleNamespace' object has no attribute 'tui'
The SimpleNamespace built around line 1183 doesn't include some attributes base_coder.py now reads. Locally I added tui/fancy_input/debug/disable_scraping as False, which got runs completing — but I enumerated by trial, so a proper fix probably wants a different approach (defaults from the real parser, or tolerant getattr reads).
4. --stats crashes with ZeroDivisionError on some runs
File "benchmark/benchmark.py", line 517, in load_results
results["test_cases_pass_ratio"] = passed_cases / total_cases
ZeroDivisionError: division by zero
Hit on both an in-progress run and a completed 225-case run: some .cecli.results.json files record zero total test cases, and line ~516 guards is not None but not zero, making --stats unusable for the whole run. Guarding the zero locally let stats print; whether zero-test-case results should exist at all is a separate question I can't judge.
5. Per-case token counts appear cumulative, inflating token/cost stats ~40×
On a full 225-case run (openai/qwen3.5:cloud, diff, --threads 3), the stats summary reported 148,981,213 prompt / 57,319,971 completion tokens ($197.79 estimated) — far beyond plausible for the run, and contradicted by the provider's own usage meter (the whole run consumed a modest fraction of a rate-limit window).
What the per-case .cecli.results.json files show:
Two unrelated cases record byte-identical counts (completion=510,803, prompt=1,340,502) — which shouldn't happen for independent measurements.
Across cases, mean ≈ max/2 (254,755 vs ~510,803) — consistent with each case snapshotting a value that ramps linearly over the run rather than measuring its own usage.
Sum-of-ramp arithmetic: n×max/2 = 225 × 510,803 / 2 ≈ 57.5M, matching the aggregated 57.3M "total" almost exactly.
My read is that a shared token counter isn't being reset (or read as a delta) per test case, and the aggregator then sums N snapshots of the same accumulator — but I haven't traced the code path, so the mechanism may be different. If correct, the impact is that token and cost columns of contributed results are inflated proportionally to run length. Pass rates appear unaffected (outcome-based); for what it's worth, with items 2–4 worked around, that qwen3.5 run itself completed cleanly: pass_rate_1 45.8, pass_rate_2 72.9, 98.2% well-formed, so the harness is very usable once past these.
Note, I'm unable to sign up for Discord, not sure why, so filing here.
Version and model info
cecli. v0.100.12
Issue
Summary
I set up the benchmark from a fresh clone of v0.100.12 (commit c92e7dc) following benchmark/README.md, intending to benchmark newer models against the polyglot exercises. I hit a chain of five distinct problems before getting clean end-to-end runs. Reporting them together since they were found in sequence on one setup and may share context; happy to split into separate issues if you prefer, and happy to open PRs for any of these if the direction I took locally is actually the right one — I have not validated my workarounds beyond "the harness then ran," so please treat them as diagnostic hints, not proposed fixes I also totally relied on Claude Fable to do all the fixes and summarize this issue.
Environment: Ubuntu 24.04 host, rootless podman via podman-docker, benchmark image from benchmark/docker_build.sh (Python 3.11 in-container), exercises from ErichBSchulz/cecli-cats (225 cats discovered), models served over an OpenAI-compatible endpoint (Ollama Cloud) via OPENAI_API_BASE/OPENAI_API_KEY.
1. README setup path leads to "Found 0 cats" / a non-functional classic harness
The README's setup section clones Aider-AI/polyglot-benchmark, but that repo contains no cat.yaml files, so benchmark.py reports Found 0 cats / No exercise directories found. Falling back to benchmark_classic.py (which defaults to that exercises dir) fails immediately on every case:
File "benchmark/benchmark_classic.py", line 1055, in run_test_real
dump(coder.ignore_mentions)
AttributeError: 'coroutine' object has no attribute 'ignore_mentions'
RuntimeWarning: coroutine 'Coder.create' was never awaited
which looks like the classic harness predates the async Coder API (I see the roadmap lists updating the test harness for async). I eventually found ErichBSchulz/cecli-cats as a Cats-formatted exercise set and moved on with benchmark.py. Ask: could the README document the intended exercises source for the Cats harness, and/or state whether benchmark_classic.py is deprecated?
2. benchmark.py: ImportError on every test case
File "benchmark/benchmark.py", line 993, in run_test_real
from cecli.main import SwitchCoderSignal
ImportError: cannot import name 'SwitchCoderSignal' from 'cecli.main'
Every case fails at setup, pre-API-call. I noticed the test suite imports this from cecli.commands, and changing the import there let the run proceed — but I don't know whether that's the intended home for it.
3. benchmark.py: args stub missing attributes the Coder reads
With (2) worked around:
File "benchmark/benchmark.py", line 1200, in run_test_real
coder.show_announcements()
File "cecli/cecli/coders/base_coder.py", line 925, in get_announcements
if not self.args.tui:
AttributeError: 'types.SimpleNamespace' object has no attribute 'tui'
The SimpleNamespace built around line 1183 doesn't include some attributes base_coder.py now reads. Locally I added tui/fancy_input/debug/disable_scraping as False, which got runs completing — but I enumerated by trial, so a proper fix probably wants a different approach (defaults from the real parser, or tolerant getattr reads).
4. --stats crashes with ZeroDivisionError on some runs
File "benchmark/benchmark.py", line 517, in load_results
results["test_cases_pass_ratio"] = passed_cases / total_cases
ZeroDivisionError: division by zero
Hit on both an in-progress run and a completed 225-case run: some .cecli.results.json files record zero total test cases, and line ~516 guards is not None but not zero, making --stats unusable for the whole run. Guarding the zero locally let stats print; whether zero-test-case results should exist at all is a separate question I can't judge.
5. Per-case token counts appear cumulative, inflating token/cost stats ~40×
On a full 225-case run (openai/qwen3.5:cloud, diff, --threads 3), the stats summary reported 148,981,213 prompt / 57,319,971 completion tokens ($197.79 estimated) — far beyond plausible for the run, and contradicted by the provider's own usage meter (the whole run consumed a modest fraction of a rate-limit window).
What the per-case .cecli.results.json files show:
Two unrelated cases record byte-identical counts (completion=510,803, prompt=1,340,502) — which shouldn't happen for independent measurements.
Across cases, mean ≈ max/2 (254,755 vs ~510,803) — consistent with each case snapshotting a value that ramps linearly over the run rather than measuring its own usage.
Sum-of-ramp arithmetic: n×max/2 = 225 × 510,803 / 2 ≈ 57.5M, matching the aggregated 57.3M "total" almost exactly.
My read is that a shared token counter isn't being reset (or read as a delta) per test case, and the aggregator then sums N snapshots of the same accumulator — but I haven't traced the code path, so the mechanism may be different. If correct, the impact is that token and cost columns of contributed results are inflated proportionally to run length. Pass rates appear unaffected (outcome-based); for what it's worth, with items 2–4 worked around, that qwen3.5 run itself completed cleanly: pass_rate_1 45.8, pass_rate_2 72.9, 98.2% well-formed, so the harness is very usable once past these.
Note, I'm unable to sign up for Discord, not sure why, so filing here.
Version and model info
cecli. v0.100.12