Skip to content

find: stream -files0-from input instead of reading it all into memory - #831

Open
MsfPablo wants to merge 1 commit into
uutils:mainfrom
MsfPablo:files0-stream
Open

find: stream -files0-from input instead of reading it all into memory#831
MsfPablo wants to merge 1 commit into
uutils:mainfrom
MsfPablo:files0-stream

Conversation

@MsfPablo

Copy link
Copy Markdown
Contributor

Fixes #779.

Problem

parse_files0_args read the entire -files0-from source with read_to_end before splitting it on NUL, so find -files0-from /dev/urandom (or /dev/zero, or any very large list) grew the buffer without bound and never walked a single starting point.

Fix

The starting points are now produced lazily by a Files0Paths iterator built on BufRead::read_until(0, ..), and do_find walks each one as soon as it is read. The source is still opened during argument parsing, so cannot open ... for reading is still reported before any traversal starts.

Behaviour is otherwise unchanged: a trailing NUL still terminates the last name rather than producing an empty one, zero-length names still warn invalid zero-length file name and are skipped, and invalid UTF-8 is still an error — the difference is that it now surfaces when that entry is reached, after the earlier valid entries have been walked, which is closer to the GNU behaviour described in the issue.

Measured on this branch (macOS, /usr/bin/time -l):

$ /usr/bin/time -l ./target/debug/find -files0-from /dev/urandom
             1884520  peak memory footprint

$ /usr/bin/time -l timeout 3 ./target/debug/find -files0-from /dev/zero > /dev/null
             1048864  peak memory footprint

I also checked src/xargs: it already reads its input incrementally and shares no code with this path.

Test

Bounded memory is awkward to assert in the test suite, so the new test covers the observable consequence of streaming: files0_streams_before_invalid_utf8 feeds ./test_data/simple\0\xff\0 and asserts the valid starting point is printed before the UTF-8 error. That fails on the previous slurp-everything implementation.

Verification

  • cargo test — 226 + 60 + 27 + 25 + 15 + 6 passed, 0 failed
  • cargo fmt --check — clean
  • cargo clippy --all-targets -- -D warnings — clean

Disclosure: this change was prepared with AI assistance (Claude). I reviewed the diff and ran the checks above myself.

-files0-from read the whole input into memory with read_to_end before
splitting on NUL, so an endless source such as /dev/urandom or /dev/zero
exhausted memory and produced no output. Read the starting points
incrementally with read_until(0) and walk each one as it is parsed.

Fixes uutils#779
@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.14286% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.92%. Comparing base (1f19cdd) to head (57ed3a4).
⚠️ Report is 7 commits behind head on main.

Files with missing lines Patch % Lines
src/find/mod.rs 97.14% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #831      +/-   ##
==========================================
- Coverage   91.93%   91.92%   -0.01%     
==========================================
  Files          35       35              
  Lines        7251     7255       +4     
  Branches      378      378              
==========================================
+ Hits         6666     6669       +3     
- Misses        443      444       +1     
  Partials      142      142              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown

Commit 57ed3a4 has test result changes:

bfs testsuite:

Test results comparison:
  Current:   TOTAL: 315 / PASSED: 267 / FAILED: 42 / SKIPPED: 6
  Reference: TOTAL: 313 / PASSED: 267 / FAILED: 40 / SKIPPED: 6

Changes from main branch:
  TOTAL: +2
  PASSED: +0
  FAILED: +2

New test failures (2):
  - gnu/okdir_path_empty
  - gnu/okdir_path_relative

@codspeed-hq

codspeed-hq Bot commented Aug 12, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 10.05%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 19 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
type_f 20.5 ms 18.7 ms +10.05%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing MsfPablo:files0-stream (57ed3a4) with main (5aa8184)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(find): when using -files0-from /dev/urandom or /dev/full, it consumes massive large memory

1 participant