[unbalancedparentheses]

Stores the completed fede backend study under research/backend-experiments/2026-11-06-fede, with all analysis, graphs, original data, exact source bundles, review evidence and reproduction instructions together. Adds a main README backend-selection guide and usage documentation covering supported modes, private pinned park/spin profiles and actors communicating across runtimes. Earlier studies remain linked from the research index.

The archive contains5,744 successful processes in45 separate blocks,51 figures,144 independent profiles and17 five-minute lifecycle configurations. Tokio leads the displayed corrected many-actor graph; pinned shards show a useful eight-core streaming-pipeline advantage. The report includes all smaller-core cases, min/max ranges, idle/active CPU costs, invalid admission schedules and historical correctness limits. Spinning consumes the assigned cores while idle. No runtime/API change or supported-backend promotion is made by this PR.

Validation on fede: all archive hashes and original-row identity;96 actor CPU audits;24 pinned full cells with three fresh processes each; exact completed block counts and lifecycle conservation. A clean second render reproduces all107 report artifacts byte for byte. Representative graph/scale/offered/resource and both pinned charts were visually inspected; title/legend overlap and clipped-data guards pass. Local documentation links resolve and Python syntax checks pass. Source-specific runtime gates are archived; those code changes were already independently validated and merged in PR185–188.

Agy approved the supplied final findings/protocol/profile interpretation with no blockers, without claiming raw-data reproduction. Review qualifications and transcripts are retained. Raw diagnostics/source boundaries distinguish library defects, historical prototypes and invalid harness attempts. The recommendations keep Tokio default and require representative client workload evidence before supporting another configuration.

[github-actions]

🤖 Codex Code Review

⚠️ Warning: Diff was truncated from 832127 to 10000 lines. Some changes were not reviewed.

No actionable findings. This PR changes documentation, research artifacts, and Python tooling; it introduces no Rust actor lifecycle or concurrency changes.

Archive verification passed: 11,889 hashes, 5,744 original rows, and all recorded CPU audits. Report regeneration could not run because matplotlib is unavailable.

---

Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

[github-actions]

🤖 Claude Code Review

⚠️ Warning: Diff was truncated from 832127 to 6700 lines. Some changes were not reviewed.

PR #189 review: backend research archive

Verdict: This PR changes no runtime or library Rust code, so actor lifecycle, concurrency and API concerns don't apply to this diff. The docs keep a clear line between supported and experimental backends, and the hedging is careful. Most of my feedback is about repository hygiene, the reliability of the evidence files, and robustness of the analysis scripts.

Blocking / high priority

1. Repository bloat from raw artifacts

The PR commits many large files directly into the repository:

• about 45 diagnostics-*.tar.gz archives
• validation-and-orchestration.tar.gz
• 144 profiles stored as .stacks and .svg; each .stacks line is several KB of folded frames
• full raw CSVs and time series

Once merged, these stay in git history permanently and every clone pays for them.

Recommendation: move data/ (except metadata.json and sha256.json) and the diagnostics tarballs to Git LFS or a GitHub release asset. Reference them by SHA256 from the repo. The manifest already supports this.

2. Local machine paths and usernames in committed evidence

The logs under data/evidence/*.log embed:

• file:///private/tmp/... paths
• /Users/unbalancedparen/.cargo/registry/... (in spawned-publication-agy.log)

Recommendation: strip or redact these before publishing.

3. Some "evidence" logs are empty, failed or incomplete

These are presented as review evidence but contain no actual review:

• spawned-frontier-agy.log and spawned-frontier-agy2.log hold only jetski: no output produced — ....
• These contain only the preamble ("I will start by reading..."):
  • spawned-native-review-agy.log
  • spawned-root-driver-agy.log
  • spawned-overnight-renderer-agy.log
  • spawned-full-actors-mac-results-agy.log
• spawned-publication-agy.log is unfiltered stream-of-consciousness ("HOLD ON! Look at line 58...", "Let's call read_file").

Recommendation: delete these, or mark them explicitly as failed or aborted runs in an index. Otherwise readers will treat them as supporting approvals. Also, internal tool names ("Agy", "jetski") mean nothing to outside readers. Define them once in reviews/AGY.md, or drop them.

4. Files referenced in the docs are missing from the diff

These are referenced but don't appear in the diff:

• VALIDATION.md links verification.log, reproduction.log, report-sha256.json and reviews/AGY.md.
• README.md links report/REPORT.md and report/REPORT.html.
• analysis-source-sha256.json hashes final-analysis.py, final-input-plan.json and final-repro.sh.

Recommendation: confirm these are committed, or fix the links.

Documentation quality

A systematic text-munging bug has dropped the space between a word and a following number or identifier. It looks like non-breaking spaces were stripped. It appears in rendered output and in source strings. Examples:

• benchmarks/README.md: "their1/2/4/8-core comparison"
• render_fede_overnight.py doc literals:
  • "async-task4.7.1", "Crossbeam0.8.8"
  • "unstarted480-case", "One or16"
  • "or16 pipelines", "same800-step"
  • "There are256", "cores10–13"
• CORRECTNESS.md:
  • "fails on3478e5f", "corrected5c0eca3"
  • "in[PR185]", "on95b4ca6", "Final2cec0a6"
  • "Rust1.93", "nice19", "than256"
  • "a267308 ...,0998deb", "control95b4ca6", "preliminary4b56238"
• FINDINGS.md:
  • "1,920 actors,256 logical producers"
  • "CPUs2–5", "Tokio5"
  • "reaches731,672/s", "baseline508,283/s"
  • "parked shards1,162,851/s"
  • "was943,042/s", "changed622,911→629,457/s"
  • "reached944,187/s", "a50µs turn"
  • "steady500k/s", "was0.011ms", "was410,074"
  • "passed316/432", "All17", "maxima were10.8–12.1MB"
• PROTOCOL.md:
  • "CPUs2–5", "Rust1.93"
  • "Graphs:1,920 actors,256", "turn16", "slice0"
  • "withcores8–15", "consistent300-second", "all14V3/V4/V5"
  • "--seconds300", "source2cec0a6"
  • "V8's482", "passes21", "source20260cd"
• VALIDATION.md: "source2cec0a6", "sourcea8471f7", "all45", "all107"
• PINNED-PROFILES.md: "pinned9.py", "=72 full cases", "Groups1", "groups16"

Recommendation: run a regex pass ([a-z,][0-9] near units and hashes) before merging. These strings are hashed in analysis-source-sha256.json, so regenerate the hashes afterwards.

Other doc notes:

• PROTOCOL.md: much of the content is operator log ("Deadline for new launches is 14:30 UTC", "after the user's explicit ... request", "PR188 merged20260cd only after ... Agy supplied-code approval"). Consider moving it to a changelog and keeping the protocol prescriptive.
• benchmarks/RUNNABLE-DRIVER-PROTOCOL.md: the correction about Arc<W> → Waker not allocating is accurate. Good catch.

Script issues

benchmarks/collect_fede_overnight.py

• Line 34: spec.split("=", 1) raises an unhelpful ValueError when the spec has no =. Validate the spec and report the offending value.
• Lines 33/66: the loop variable directory (for directory in ["series", ...]) shadows the directory parsed from the spec. It isn't a bug today, since root is computed first, but it's a trap for future edits. Rename it.
• Line 37: meta["dirty"] raises KeyError on older metadata. Use meta.get("dirty", True) so that missing provenance fails closed.
• Lines 41 and 98: csv.DictReader((root / "raw.csv").open()) never closes the handle. Use with. Open CSVs with newline="", as the csv docs recommend, in this file and in the other two scripts.
• Line 50: int(r.get("final_queued", 0)) raises ValueError if the column exists but is empty. Use int(r.get(...) or 0).
• Validation via assert: the __debug__ guard works, but explicit raise ValueError(...) gives better messages and needs no guard. The same applies to verify_fede_overnight.py.

benchmarks/render_fede_overnight.py

• Findings insertion: doc[6:6] = [...] relies on a hard-coded index. Any edit to the header silently moves FINDINGS into the wrong place. Insert relative to a marker instead.
• KeyError on missing metrics: in the graph table, the pinned table and bars(), the code reads s['replies_per_s_median'], s['observed_p99_ns_median'], s['idle_cores_median'] and s['active_cpu_us_per_root_median'] directly. stats() returns None for empty metrics, so any cell missing a metric crashes the render. Guard these reads, or assert presence with a clear message.
• bars(): cores = 8 if phase in [...] else 4 hard-codes the core count. Derive it from config["actor-cpus"] instead.
• Time-series suptitle: it hard-codes "1.2M/s steady" while the data selects max(rate). Format the title from the selected rate.
• save(): it checks only y-axis clipping, but bars() uses barh with set_xlim(left=0). If x-clipping matters for those figures, check dataLim.xmax as well.
• Dependencies: matplotlib, numpy and markdown aren't pinned anywhere. Byte-for-byte reproduction (as claimed in VALIDATION.md) needs a lockfile or requirements.txt with versions.
• Repeated parsing: json.loads(s["config_json"]) runs several times per row in the pinned section. Parse it once.

benchmarks/verify_fede_overnight.py

• Hard-coded values: the expected counts (45, 5744, 96, 24, 17, 51, 144) and the CPU base {2 + i % n} make this a single-archive verifier. Either rename it to reflect that, or read the expected values from metadata.json and the config (actor-cpus).
• cpus(): it doesn't strip whitespace or handle empty segments. "2, 3" or a trailing comma raises.

Minor

• benchmarks/README.md: the new paragraph is prepended before the existing content, which has no heading. Add a top-level heading.
• docs/BACKENDS.md and the top-level README describe pinned parked shards as "promising" and "encouraging". Given the stated mixed results at 1–4 cores, consider adding the smaller-core caveat to the README summary as well, not only in BACKENDS.md and FINDINGS.md.

---

Automated review by Claude · claude-opus-5-5 · custom prompt

[github-actions]

🤖 Grok Code Review

⚠️ Warning: Diff was truncated from 832127 to 6093 lines. Some changes were not reviewed.

This PR does not change actor runtime code. It archives research and adds offline Python tooling. The lifecycle, wake, and shutdown claims are documentation only and are not re-verified against Rust sources here.

Blocking / correctness

1. Broken research index links. README.md and research/README.md point at research/backend-experiments/README.md, which this diff does not add. The study index that does exist is research/backend-experiments/2026-11-06-fede/README.md.

2. Advertised archive is not in the diff. Docs and verify_fede_overnight.py require data/raw.csv, data/sha256.json, data/runs/*/raw.csv, data/series/, and report/ (51 PNG/SVG, SUMMARY.csv, RESOURCES.csv). Only metadata.json, diagnostics tarballs, evidence logs, and one .stacks file are present. A checkout cannot run the documented verify/render commands, and the “5,744 processes / 51 figures” claim is not reproducible from this patch.

3. Collector drain check fails open (benchmarks/collect_fede_overnight.py ~45–54). Missing final_queued / final_outstanding become 0 via .get(..., 0), so a row without those columns still passes. Require the keys, and do not default them.

4. Resource table claims an equality it does not show (benchmarks/render_fede_overnight.py resource markdown). Header is Starts = stops = drops, but the cell prints only started. Assert started == stopped == dropped at render time and print all three. The collector only checks that equality on rows with an empty succeeded field, so a soak row that also has succeeded skips the lifecycle check.

Scripts

• Profile rendering assumes files that collection does not create. The profile loop reads profiles/{stem}.stacks with no exists() check, and links to ../data/profiles/{stem}.svg. collect_fede_overnight.py only copies a profiles/ tree if the run directory already has one; it never builds SVGs. Missing stacks abort the report; missing SVGs are dead links.
• Hardcoded fede CPU base (verify_fede_overnight.py pinned branch): non-async audits must be {2 + i % n}, not cpus(config["actor-cpus"]). That matches this host’s masks, but it will reject a faithful rerun on any other topology and will not catch a mask that does not start at CPU 2 if the hardcoded cores still match.
• Report links are path-coupled. ../data/metadata.json and ../data/profiles/... assume --output is exactly research/backend-experiments/2026-11-06-fede/report. Any other --output writes a report whose links are wrong.
• label() omits settings that the group key includes (slice_us, and Z/I when the variant is not in the hard-coded list). Unlike groups then hit assert len(g) <= 1 instead of being pooled, which is safe, but worker labels are also sorted as strings (N16 < N4). Coerce workers to int before display sort.
• CSV inputs are opened without a context manager ((root / "raw.csv").open()). Fine for a one-shot, but the handles are leaked if a later assert fires.

Docs / actor contract

The supported-backend guidance in docs/BACKENDS.md is the right contract: Tokio default, blocking actors hold a worker until stop, Backend::Thread is one OS thread plus a runtime and does not preempt a running handler, pinned park/spin stays private and process-wide, and cross-runtime ActorRef traffic does not migrate I/O or the actor. Keep that boundary. Do not let the eight-core pipeline ratios in the root README.md read like a supported speedup; the same paragraph already says they are private experiments, which is the correct status.

Typos that will ship into the generated report and benchmarks/README.md: their1/2/4/8-core, One or16, CPUs2–9, with1,024, withothercores, on3478e5f, in[PR185].

No new race, deadlock, or cancellation defect is introduced in Rust by this diff. I would not merge until the missing index, the drain-column check, the resource-table equality, and the actual data/report tree (or an explicit “data not vendored” note plus dropped verify assertions) are fixed.

---

Automated review by Grok (xAI) · grok-4.7 · custom prompt

[github-actions]

🤖 DeepSeek Code Review

⚠️ Warning: Diff was truncated from 832127 to 7035 lines. Some changes were not reviewed.

Overall this PR changes documentation and research/benchmark artifacts only; there are no Rust runtime/actor-code changes, so no new actor lifecycle, cancellation, timer, or message-ordering bugs are introduced in this diff.

A few actionable issues in the Python tooling/docs:

• Missing counters in archived artifacts — benchmarks/collect_fede_overnight.py, artifact-copy loop around lines 65-69:

  for directory in ["series", "threads", "profiles"]:

  The file’s own docstring says it copies series/profiles/counters, but "counters" is not included. If counters are expected for provenance, the archive is incomplete. Add "counters" to the list or correct the docstring/protocol.

• Renderer can silently skip assertions under python -O — benchmarks/render_fede_overnight.py relies heavily on assert for validation (save, grouping uniqueness, factorial completeness), but unlike collect and verify, it does not guard against -O. A user can run python -O bench... render... and disable every validation assertion. Add the same if not __debug__: raise RuntimeError(...) guard at the top of main().

• label() can hide experiment dimensions — benchmarks/render_fede_overnight.py, label() near lines 39-53, does not include slice_us. That is used intentionally in the clock figure, but graph/bar helpers also call label() and assert len(g) <= 1; if a graph phase ever includes multiple slice_us values, those cells would either crash or be mislabeled. Either include slice_us in label() or explicitly assert it is constant for phases reaching bars().

• Hardcoded verification constants — benchmarks/verify_fede_overnight.py, around lines 80-95, pins exact counts (45, 5744, 96, etc.). This is fine for a frozen archive, but it couples verification to one specific dataset version. Any legitimate data correction/regeneration will require coordinated script changes.

• Repository size/hygiene — many large binary/generated artifacts are committed directly (data/diagnostics-*.tar.gz, large .svg/.stacks profiles, base64-embedded HTML). Consider Git LFS or storing these outside the main repository and keeping only the manifest/hash files.

• Minor typo — benchmarks/README.md: their1/2/4/8-core comparison should be their 1/2/4/8-core comparison.

Overall, the provenance/hash/grouping approach is good; the script issues above are mostly robustness/reproducibility concerns rather than runtime concurrency defects.

---

Automated review by DeepSeek · deepseek-v4-pro · custom prompt

[github-actions]

🤖 Gemini Code Review

⚠️ Warning: Diff was truncated from 832127 to 7035 lines. Some changes were not reviewed.

Here is the review of PR #189. Since this PR consists of documentation, benchmark analysis scripts, and archived research data (with no changes to the Rust crates), the feedback focuses on repository hygiene, script correctness, and documentation quality.

1. Repository Size & Artifact Storage

• research/backend-experiments/2026-11-06-fede/data/: This PR commits ~45 .tar.gz archives, git source bundles, and thousands of raw CSV, .stacks, and .svg flamegraph files (~11,889 files per VALIDATION.md) directly into the repository's git tree. Committing large binary blobs and thousands of generated artifacts permanently bloats git clone and CI checkout times. Consider hosting the raw data/bundles as a GitHub Release archive, in a separate benchmark-data repository, or via Git LFS, while keeping the scripts, markdown reports, and summary tables in the main repo.

2. Benchmark Scripts (benchmarks/)

• research/backend-experiments/2026-11-06-fede/analysis-source-sha256.json (lines 4–6):
  The recorded SHA-256 digests for collect_fede_overnight.py, render_fede_overnight.py, and verify_fede_overnight.py do not match the files committed in benchmarks/:

  • benchmarks/collect_fede_overnight.py actual SHA-256: e353f4cb664ef615c36efac2c742ba289552be5d209bd255f5abfc8743bf9e35 (manifest has d8d975fc...)
  • benchmarks/render_fede_overnight.py actual SHA-256: 4a04b68617a5338804bc8da30daeb53d5f0abdb091d0f95958a1053b733557d4 (manifest has ff26364e...)
  • benchmarks/verify_fede_overnight.py actual SHA-256: 9f96810d7cf82297600a737a0b74857dd28c6821576c6e91418deac7093910fd (manifest has 5f2cd70e...)

  It looks like the scripts were formatted (e.g., with ruff format / black) after analysis-source-sha256.json was generated.

• benchmarks/collect_fede_overnight.py (line 42):

  original = list(csv.DictReader((root / "raw.csv").open()))

  The file handle opened by .open() is not closed deterministically with a with statement (unlike (a.output / "raw.csv").open("w") on line 94). Prefer:

  with (root / "raw.csv").open() as f:
      original = list(csv.DictReader(f))

• benchmarks/render_fede_overnight.py (line 730):

  and int(r["repeat"]) == 0

  If raw.csv contains merged lifecycle soak rows (which don't have a repeat column, so DictWriter writes ""), int(r["repeat"]) will raise ValueError if r["phase"] ever matches offered_phase or if evaluated before filtering. Using int(r.get("repeat") or 0) == 0 is safer.

3. Incomplete / Unredacted Evidence Logs

• research/backend-experiments/2026-11-06-fede/data/evidence/:
  • spawned-frontier-agy.log and spawned-frontier-agy2.log only contain a 1-line permission error (jetski: no output produced — a tool required the "read_file" permission...).
  • spawned-full-actors-mac-results-agy.log, spawned-native-review-agy.log, spawned-overnight-renderer-agy.log, and spawned-root-driver-agy.log are truncated 1–3 line logs containing only initial tool-call preambles without review content.
  • spawned-publication-agy.log (lines 20–27) contains raw internal scratchpad text and local user paths (/Users/unbalancedparen/.cargo/...).

4. Documentation Typos (Missing Spaces)

Several markdown files and report strings have missing spaces around numbers and commit hashes:

• benchmarks/README.md (line 196): their1/2/4/8-core -> their 1/2/4/8-core
• benchmarks/render_fede_overnight.py:
  • Line 371: async-task4.7.1 -> async-task 4.7.1, Crossbeam0.8.8 -> Crossbeam 0.8.8
  • Line 385: unstarted480-case -> unstarted 480-case
  • Line 390: One or16 actors/core -> One or 16 actors/core
  • Line 393: old95b4ca6 -> old 95b4ca6
  • Line 409: CPUs2–9, producers10–13 and collectors14–17, with1,024 ... includes512 ... Those16CPU ... CPUs2–7 share96MiB L3, CPUs8–9 share32MiB L3 withothercores ... the96MiB
  • Line 765: or16 pipelines, same800-step, are256, cores10–13, CPUs2..N+1
  • Line 831: nice10, nice19
• research/backend-experiments/2026-11-06-fede/FINDINGS.md:
  • Lines 18–40: actors,256, CPUs2–5, Tokio5, actors,1,024, reaches731,672/s, baseline508,283/s, CPUs2–9 span96MiB
  • Lines 47–68: same800, cores10–13,1,048,576, shards1,162,851/s, shards1,211,613/s
  • Lines 83–119: was943,042/s, changed622,911→629,457/s, rate689,489→679,968/s, reached944,187/s, a50µs, reached863,051/s, steady500k/s, was0.011ms, was410,074, passed316/432, All17, were10.8–12.1MB
• research/backend-experiments/2026-11-06-fede/CORRECTNESS.md, PINNED-PROFILES.md, PROTOCOL.md, and VALIDATION.md: Similar missing spaces between words and numbers/SHAs throughout (e.g., on3478e5f, in[PR185], All45, verifies11,889).

---

Automated review by Gemini · gemini-4-argon · custom prompt

[github-actions]

🤖 Kimi Code Review

⚠️ Warning: Diff was truncated from 832127 to 7035 lines. Some changes were not reviewed.

No production Rust/actor code changes are included; the issues are in the research tooling and documentation.

Findings

1. benchmarks/render_fede_overnight.py:96 — validation assertions can be disabled

   Unlike the collector and verifier, the renderer does not reject python -O. Its assertions protect against pooling unlike groups, incomplete factorials, duplicate schedules, and malformed pinned/resource cells. With optimization enabled, those checks disappear and an invalid report can be produced silently.

   Add the same guard used by the other scripts:

   if not __debug__:
       raise RuntimeError("Report rendering requires assertions enabled")

2. benchmarks/verify_fede_overnight.py:67-69 — CPU audit hardcodes fede CPU IDs

   assert cpus(record["allowed_cpus"]) == {2 + i % n}

   PINNED-PROFILES.md explicitly tells users to substitute and record masks for other machines. Any valid mask not starting at CPU 2 will fail verification. Parse the ordered CPU list from config["actor-cpus"] and compare against {core_ids[i % n]}. cpus() returns a set, so a separate ordered parser is needed to preserve the configured shard-to-core mapping.

   The hardcoded archive expectations at lines 41, 55, and 72-79 (45, 5744, 96, 24, 17, 51, 144) also make the verifier brittle to any legitimate additions. Either document that this verifier only accepts the immutable 2026-11-06 archive or derive those expectations from the archive metadata.

3. Several linked documentation/report artifacts are missing from the diff

   • research/README.md:4 and README.md:189,255 link to research/backend-experiments/README.md, but that file is not added.
   • research/backend-experiments/2026-11-06-fede/README.md:10 links to report/REPORT.md and report/REPORT.html, which are not present.
   • VALIDATION.md:23 references verification.log, reproduction.log, and report-sha256.json, also absent from the diff.

   Add the artifacts or change the links to point at the actual entry points and generation commands. As written, repository documentation navigation and the documented verification flow are incomplete.

4. benchmarks/render_fede_overnight.py:99-100 — stale output can contaminate later reports

   The output and graph directories are reused without cleaning. A subsequent render with fewer phases/figures leaves old PNG/SVG files behind, and verify_fede_overnight.py counts every file in those directories. This can produce misleading artifact counts and stale figures in the HTML report. Require an empty output directory or remove previously generated artifacts before rendering.

5. benchmarks/collect_fede_overnight.py:12 — whole-file hashing has avoidable memory overhead

   sha() calls path.read_bytes() for every artifact, including large source bundles and archives. Stream files in chunks instead to avoid large transient allocations when archiving many multi-megabyte artifacts.

---

Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt