Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
e532776
bench: preserve versioned overnight experiment analysis and protocol
unbalancedparentheses Nov 6, 2026
7eadabf
docs: clarify scheduler resources and I/O engine boundaries
unbalancedparentheses Nov 6, 2026
b081c5a
fix: preserve valid report paragraph boundaries
unbalancedparentheses Nov 6, 2026
378eb89
docs: preserve reproduced defects and harness failure boundaries
unbalancedparentheses Nov 6, 2026
6e80774
docs: tie backend research to the next production priorities
unbalancedparentheses Nov 6, 2026
c2b4fb9
docs: record the next bounded scheduling hypothesis
unbalancedparentheses Nov 6, 2026
85c68ac
docs: define resource sampling and shutdown boundaries
unbalancedparentheses Nov 6, 2026
2d83e23
bench: reject overlapping report titles and legends
unbalancedparentheses Nov 6, 2026
d626883
docs: pin the final measured scheduler source
unbalancedparentheses Nov 6, 2026
3570bbe
docs: distinguish waker refcounts from heap allocation
unbalancedparentheses Nov 6, 2026
e1e2137
docs: record final scheduler gate and review provenance
unbalancedparentheses Nov 6, 2026
8c41dbc
docs: separate supported backends from research candidates
unbalancedparentheses Nov 6, 2026
f128111
docs: specify lifecycle counter and warmup scope
unbalancedparentheses Nov 6, 2026
ca351d2
style: make the report analysis pipeline easier to review
unbalancedparentheses Nov 6, 2026
2754e6b
bench: separate valid offered-load curves from diagnostics
unbalancedparentheses Nov 6, 2026
0aafc5c
docs: publish fede backend research with graphs and usage guidance
unbalancedparentheses Nov 6, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,22 @@ async fn handle_find(&mut self, msg: Find, _ctx: &Context<Self>) -> FindResult {

Both provide the same `Actor`, `Handler<M>`, `ActorRef<A>`, and `Context<A>` types. Switching between them requires changing imports and adding/removing `async`/`.await`.

## Choosing a backend

Use Tokio (`Backend::Async`) for general async applications and dynamic actor
populations. Use `Backend::Blocking` for persistent blocking work, or
`Backend::Thread` when an actor needs a dedicated thread. Each blocking actor
occupies a worker until it stops; each dedicated actor has its own thread/runtime.

The experimental pinned profiles target fixed streaming pipelines. Parked shards
showed promising eight-core results; spinning can improve some tails but consumes
the assigned cores while idle. These remain private experiments, and Tokio stays
the supported default. Actor APIs, ordinary messages and request/reply are unchanged.

See [backend selection and usage](docs/BACKENDS.md), including actors communicating
across runtimes, and the [backend research folder](research/backend-experiments/README.md)
for graphs, analysis, raw results and reproduction instructions.

## Project Structure

```
Expand Down Expand Up @@ -235,6 +251,8 @@ removing Tokio is not a committed direction.
See the [reproducible baseline harness](benchmarks/README.md) and
[initial local analysis](benchmarks/REPORT-2026-11-05.md). These microbenchmarks
measure existing backends and document limits; they are not client-level results.
The [backend research index](research/backend-experiments/README.md) brings together
the studies, including the completed fede scheduler and pinned-pipeline analysis.

## Testing

Expand Down
34 changes: 34 additions & 0 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
The completed [backend research study](../research/backend-experiments/2026-11-06-fede/README.md)
contains 5,744 successful processes, 51 figures, the pinned pipeline comparison,
source bundles and review evidence. See [backend usage](../docs/BACKENDS.md) for
supported choices and experimental status. New graphs and analysis live in the
research folder; benchmark programs and runners stay here.

The [native Linux graph, overload and profiling report](reports/fede-research-2026-11-06/REPORT.md)
adds 1,002 fresh processes, longer graph confirmations and 66 CPU profiles.
[HTML with embedded charts](reports/fede-research-2026-11-06/REPORT.html),
Expand All @@ -24,6 +30,28 @@ graphs for sharing. Start with the [findings](DESIGN-FINDINGS.md) and
[implementation/test guide](EXPERIMENTS.md). [Raw samples and source metadata](results/designs-2026-11-05/manifest.json)
retain the nine failed resource-limit samples alongside successful measurements.

## Supported backends and research status

The supported execution choices remain the existing Tokio async, blocking-pool
and dedicated-thread task backends, plus synchronous thread actors. Tokio is the
general-purpose default. Blocking/dedicated modes have distinct use cases and
resource costs; a persistent actor population must fit the chosen execution
resources.

Private selectors behind backend-experiments are research interfaces. Shared
runtime ownership, static shards and Root/Runnable/Deque candidates are retained
for defined workload studies. Global/local/split-queue variants also serve as
comparison controls. Their presence does not make them supported public backend
choices or recommended defaults.

Generic inline-future storage remains on its research branch and in the exact
source bundle; current measurements do not justify its added complexity.
Benchmark-only transports and io_uring/epoll/kqueue engine controls remain
experiments too. Supported additions need a named use case, a repeatable measured
benefit, acceptable latency/CPU/memory/fairness tradeoffs, lifecycle/I/O/failure
coverage, and a maintainable documented interface. The supported set should stay
small.

## Initial baseline

The [initial local report](REPORT-2026-11-05.md) includes raw results and a
Expand Down Expand Up @@ -163,3 +191,9 @@ Review archives are retained as requested provenance, with explicit editorial
limits. Suggested numeric adoption thresholds are reviewer proposals, not
approved production criteria. Baseline observer asymmetry remains disclosed;
the later matched design study adds a direct observer and full-actor controls.

The [fixed-core pipeline study](../research/backend-experiments/2026-11-06-fede/PINNED-PROFILES.md) explains the private
Silver-inspired pinned park/spin profiles and their1/2/4/8-core comparison.
It retains Tokio and addressed messages; CPU affinity is not a supported public
backend option. Results distinguish dynamic many-actor graphs from small fixed
pipelines and include the idle CPU cost of spinning.
5 changes: 4 additions & 1 deletion benchmarks/RUNNABLE-DRIVER-PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ private native-runnable, not removal of Tokio or a supported runtime replacement

The ready queue owns a unique Runnable ticket. Its integrated state machine owns
poll/wake exclusion, so this path does not need our custom Gate, Arc<Job> clone
per poll, a separately allocated Waker, or the mutex protecting each future.
per poll, or the mutex protecting each future. The custom path's conversion
from an existing Arc to Waker is an inline handle conversion, not another heap
allocation; its preceding Arc clone still changes the reference count.
[Standard-library implementation](https://doc.rust-lang.org/src/alloc/task.rs.html).
Metadata keeps a weak pool pointer and preferred-worker atomic. Before each run,
worker TLS routes internal wakes locally; external wakes prefer the last worker.
Runnable.run consumes the ticket; wake schedules a new ticket. Detach Task before
Expand Down
123 changes: 123 additions & 0 deletions benchmarks/collect_fede_overnight.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
#!/usr/bin/env python3
"""Archive completed, version-separated experiment blocks without changing inputs."""

import argparse
import csv
import hashlib
import json
from pathlib import Path
import shutil
import tarfile


def sha(path):
return hashlib.sha256(path.read_bytes()).hexdigest()


def main():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument(
"--run", action="append", required=True, help="unique name=directory"
)
p.add_argument("--bundle", action="append", default=[], type=Path)
p.add_argument("--evidence", action="append", default=[], type=Path)
p.add_argument("--output", type=Path, required=True)
a = p.parse_args()
if not __debug__:
raise RuntimeError("Run without -O: validation assertions are required")
a.output.mkdir(parents=True, exist_ok=False)
records = []
inputs = {}
revisions = {}
for spec in a.run:
name, directory = spec.split("=", 1)
root = Path(directory)
assert name not in inputs and "/" not in name and name not in [".", ".."]
meta = json.loads((root / "metadata.json").read_text())
assert meta.get("finished_utc") and not meta["dirty"], (
name,
"unfinished or dirty",
)
assert not meta.get("stopped_at_deadline"), (name, "truncated block")
original = list(csv.DictReader((root / "raw.csv").open()))
assert original and all(r["status"] == "ok" for r in original)
if meta.get("planned_samples"):
assert len(original) == meta["planned_samples"]
for r in original:
if r.get("succeeded") not in [None, ""]:
assert int(r["succeeded"]) + int(r["timeouts"]) == int(r["messages"])
assert (
int(r.get("final_queued", 0))
== int(r.get("final_outstanding", 0))
== 0
)
else:
assert int(r["started"]) == int(r["stopped"]) == int(r["dropped"])
records.append(
dict(
r,
dataset=name,
revision=meta["revision"],
original_stem=r["stem"],
stem=name + "/" + r["stem"],
)
)
dest = a.output / "runs" / name
dest.mkdir(parents=True)
for file in ["metadata.json", "raw.csv", "build.log"]:
if (root / file).exists():
shutil.copyfile(root / file, dest / file)
for directory in ["series", "threads", "profiles"]:
if (root / directory).exists():
shutil.copytree(root / directory, a.output / directory / name)
if (root / "logs").exists():
with tarfile.open(
a.output / ("diagnostics-" + name + ".tar.gz"), "w:gz"
) as archive:
archive.add(root / "logs", arcname="logs")
binaries = meta.get("binaries", meta.get("binary_sha256"))
inputs[name] = dict(
original_directory=str(root.resolve()),
samples=len(original),
revision=meta["revision"],
started_utc=meta["started_utc"],
finished_utc=meta["finished_utc"],
actor_cpus=meta["actor_cpus"],
driver_cpus=meta["driver_cpus"],
metadata_sha256=sha(root / "metadata.json"),
raw_sha256=sha(root / "raw.csv"),
binaries=binaries,
)
# Exact source and binaries remain properties of each block; never infer cross-version identity.
revisions.setdefault(meta["revision"], []).append(name)
fields = list(dict.fromkeys(k for r in records for k in r))
with (a.output / "raw.csv").open("w") as f:
writer = csv.DictWriter(f, fields, lineterminator="\n")
writer.writeheader()
writer.writerows(records)
for category, files in [("source-bundles", a.bundle), ("evidence", a.evidence)]:
(a.output / category).mkdir()
for file in files:
target = a.output / category / file.name
assert not target.exists(), file.name
shutil.copyfile(file, target)
metadata = dict(
inputs=inputs,
revisions=revisions,
samples=len(records),
derivation="Original CSVs/metadata retained in runs/. Derived rows add dataset, revision and original_stem, prefix stem with dataset; all original numeric and config/experiment fields are unchanged. Blocks and source revisions are never pooled.",
)
(a.output / "metadata.json").write_text(json.dumps(metadata, indent=2) + "\n")
manifest = {
str(f.relative_to(a.output)): sha(f)
for f in sorted(a.output.rglob("*"))
if f.is_file()
}
(a.output / "sha256.json").write_text(json.dumps(manifest, indent=2) + "\n")
print(
f"Archived {len(records)} successful processes in {len(inputs)} separate blocks; {len(manifest)} hashed artifacts"
)


if __name__ == "__main__":
main()
Loading
Loading