Replace jwalk with a work-stealing directory walker

[Byron]

Tasks

• refackiew
• review clippy

---

Created by Codex on behalf of Byron. Byron will review before this is ready to merge.

Summary

• Replace jwalk and Rayon with an in-tree directory walker backed by crossbeam-deque work stealing.
• Preserve deterministic depth-first, optionally name-sorted iteration while directory reads and metadata collection proceed in parallel.
• Reuse the walker for aggregation, interactive traversal, tests, and recursive deletion; delete files concurrently and directories deepest-first.
• Raise the default macOS walker thread count from 3 to 8 after local release benchmarks showed the best stable improvement near that setting.

Context

The iterator facade keeps ordering out of the worker scheduler: workers discover and prepare directory batches in parallel, while consumers only observe a directory after its complete batch is available and sorted. This avoids the previous handbrake where traversal order constrained discovery.

Advantages include removing two dependencies, sharing one traversal implementation across commands, retaining deterministic output, and improving measured scan time on /Users/byron/dev/github.com. The main tradeoffs are more in-tree concurrency code to maintain, one buffered result per discovered directory until consumed, and a platform-specific default chosen from local hardware measurements rather than dynamic tuning.

An attempted explicit parker/unparker wake-up path did not improve the benchmark and was omitted.

Validation

• cargo fmt --all
• cargo clippy --all-targets --all-features
• cargo test --all-targets --all-features (85 tests)
• git diff --check
• Release benchmark on /Users/byron/dev/github.com with --count-hard-links: about 29.9s at 3 threads, 18.9s at 6, 18.0s at 8, 17.6s at 10, 20.9s at 12, and 22.9s at 16.

User Prompts

Replace jwalk with an own implementation of a directory walk. Use crossbeam-deque to implement simple workstealing, and do it without rayon as well. Sorting is not so trivial, but maybe you can find a way to use the iterator facade to your advantage, as everything that is returned must be known to be sorted.

Can it be that the sorting prevents all parallelism?

Can you remove the handbrakes?

Also, look at the parallelism of /Users/byron/dev/github.com/KSXGitHub/parallel-disk-usage and see why it can do in 19s what dua can to in 25 or more?

Create a new stg commit with the current state and details on the changes, with advantages and maybe disadvantages. Then try making it faster. I always thought that crossbeam-deque will solve all problems.

$pr-from-session

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1b6987f63c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".