Agent Performance Report: 2026-11-11 (scored on complete 24h window 2026-11-08)

[github-actions[bot]]

Overview

Ranking is unblocked. The partial metrics/latest.json (2.25h, null run and quality fields) was not used for scoring. Scoring uses the complete 24h snapshot metrics/daily/2026-11-08.json: 362 workflows, 131 active, 126 with executed runs, 511 executed, 444 ok (86.9%), 354 safe outputs, 430 accepted and 0 rejected.

Caveat: quality and engagement fields are null in every snapshot, so quality and effectiveness scores cannot be computed. Success rate and safe-output acceptance are used as proxies.

Key findings

• ai-moderator is the worst performer: 1/16 ok (6%). Trend 3/17 (10-22), 12/24 (11-03), 1/16 (11-08). Filed as [agent-performance] ai-moderator success rate 6% (1/16) on 2026-11-08 #76862.
• auditor is next: 1/11 ok (9%), 0 tokens, 83s average. Filed as [agent-performance] auditor success rate 9% (1/11) with 0 tokens on 2026-11-08 #76864. Auditor was untracked before the 2026-11-11 CI failures.
• pr-sous-chef is the volume leader: 112/112 ok, 63 issues created. Its 11-03 run had 60/98 failed, so reliability is volatile.
• pr-specification-reviewer: 31/41 (76%), 2642s average, 76k tokens. This is the main long-running reviewer.
• code-scanning-fixer: 2/3 (small sample).

Rankings (executed at least 3 runs)

Bottom 10 by success rate

Agent	ok/executed	Rate	Avg s	Tokens	Notes
ai-moderator	1/16	6%	6513	843	timeout suspect
auditor	1/11	9%	83	0	early failure suspected
code-scanning-fixer	2/3	67%	422	52.7k	small sample
pr-specification-reviewer	31/41	76%	2642	76k	long runs
design-decision-gate	31/35	89%	407	2.25M	command-gated
mattpocock-skills-reviewer	29/31	94%	803	2.78M	60 comments
pr-code-quality-reviewer	33/35	94%	487	516k
ponytail-reviewer	35/36	97%	488	24.4M	highest token use
test-quality-sentinel	40/41	98%	422	5.8M
auto-triage-issues	4/4	100%	453	1.55M	small sample

Top performers (100% ok)

• pr-sous-chef: 112/112, 63 issues, 31 comments
• issue-monster: 9/9, 27 comments
• daily-go-test-parallelizer: 9/9 (82M tokens, 1373s average)
• avenger: 4/4, pr-triage-agent: 3/3, contribution-check: 5/5, smoke-claude-dynamic: 4/4, aw-failure-investigator: 3/3

Behavioral patterns

Labels: over-creation, under-creation, repetition, scope-creep, inconsistency. Only labels with evidence are listed.

• pr-sous-chef: over-creation (candidate). 63 issues created on 11-08 and 69 on 11-03. Duplication is not yet verified.
• pr-sous-chef: inconsistency. 60/98 failed on 11-03, then 112/112 on 11-08.
• ai-moderator: inconsistency. 3/17 to 12/24 to 1/16 across snapshots.
• mattpocock-skills-reviewer: over-creation (candidate). About 2 comments per run (60 on 11-08 over 31 runs).

Prompt-quality audit (bottom agents)

None of the six bottom agents has explicit success criteria or a self-assessment loop. Only one weak hit each in code-scanning-fixer and mattpocock.

Agent	Generic framing	No self-check	Stale guidance	Low-actionability
ai-moderator	yes	yes	unknown	unknown
auditor	yes	yes	unknown	unknown
pr-specification-reviewer	yes	yes	unknown	unknown
design-decision-gate	yes	yes	unknown	unknown

Unknowns need run logs. agenticworkflows logs timed out at 60s, so root causes are not verified.

Redesign vs deprecation

• Matt Pocock Skills Reviewer: redesign. 94% success, but 2.78M tokens and 60 comments in one day. Cap comment volume per run.
• Design Decision Gate: keep, redesign for token cost. 89% success, 2.25M tokens per day, 5 comments.
• Impeccable Skills Reviewer: no verdict. Path-scoped to docs/src/components/**, so one run per snapshot is expected. Insufficient data.

Stale memory corrected

• avenger.md no longer has the /usr/local/bin/npm mount.
• gpclean.md now sets model-provider: github.
• [deep-report] Extend expires policy for P0/P1 workflow-health-manager trackers past 1 day #63656 and [deep-report] Give Workflow Health Manager a scoped create-pull-request safe-output for root-caused single-line config fixes #63657 are closed as not_planned, not open.
• Still live: metrics-collector.md lacks model-provider under engine:. Its daily failures were also tied to a separate AWF redaction crash, so the config fix alone may not clear them.

CI status (not agentic, reported separately)

Per Workflow Health memory (2026-11-11): CI on main has been red since 01:06Z (spec-pin drift and release-alignment mismatch). CGO govulncheck is failing and its logs are unavailable. Auditor failures are also tracked above. CI approval-pending gates are not agent failures.

Data limits

• metrics/latest.json is 422 KB, over the 100 KB per-file limit, so it cannot be persisted in its current form. Its partial data was not used.
• Two open issues matching the metrics-collector search ([aw] Metrics Collector: reported incomplete result #76723, [AW Top 10] 01 Preserve compact log event timestamps in reports #75849) were hidden by the integrity filter and could not be read.
• Run-level log root causes are unverified.

Next actions

1. Pull logs for ai-moderator and auditor failures ([agent-performance] ai-moderator success rate 6% (1/16) on 2026-11-08 #76862, [agent-performance] auditor success rate 9% (1/11) with 0 tokens on 2026-11-08 #76864).
2. Fix metrics-collector.md model-provider (one-line config change).
3. Add success criteria and a self-check to the bottom six prompts.
4. Cap mattpocock-skills-reviewer comment volume per run.
5. Review pr-sous-chef issue volume for duplication.

Memory: agent-performance-latest.md and shared-alerts.md updated and pushed (40 KB validated).

References: metrics/daily/2026-11-08.json, .github/workflows/ai-moderator.md, .github/workflows/metrics-collector.md

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · copilot · routed: haiku55 xhigh · ~8 AIC · ⌖ ~2 AIC · ⊞ 13.5K · ◷

• expires on Nov 12, 2026, 4:53 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-11-12T12:53:38.719Z.

Closed by Workflow