Skip to content

ipparse: review fixes, malformed-input robustness, hot-path work and test coverage - #3

Merged
jperon merged 78 commits into
ipparse-perf-compatfrom
ipparse-review
Nov 10, 2026
Merged

jperon merged 78 commits into
ipparse-perf-compatfrom
ipparse-review

Conversation

@jperon

@jperon jperon commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #2 (base branch ipparse-perf-compat). Result of a full read-through of the ipparse library code: confirmed bugs fixed (each with a regression test), malformed-input robustness, hot-path work, dead-code cleanup, and test coverage tooling.

Bug fixes (each commit carries its regression test)

  • IPv6: pack was not exported (ip.pack on IPv6 raised); s2net6 without mask raised; vtf was frozen at new; UDP zero checksum now emitted as 0xFFFF.
  • IPv4 / TCP: pack raised when options was omitted; raw flag keys given to new shadowed __newindex (t.syn = false was ignored); ihl < 5 and TCP data offset < 20 are rejected.
  • QUIC: long-header version packed on 16 bits instead of 32; __newindex silently dropped new fields; remove_header_protection raised on short packets; version_negotiation rewritten (32-bit versions, pack always raised).
  • DNS: rcode/opcode/z getters return the numeric value (API change, they returned booleans); ra setter added; unknown header fields are stored; unknown EDNS options no longer repack as "nil"; client-subnet repack fixed (global unpack, lost address); classes.NONE is 254, ANY added.
  • Core: lbin2hex dropped the tail of its input; bin2hex overflowed the Lua stack in the kernel on 128-byte chunks; hex2bin returned two values; protected always printed a traceback (leak_debug was always truthy); hexdump aligns the ASCII column on incomplete lines.
  • Misc: bit53.arshift was a logical shift; five wrong TLS handshake type codes; HKDF expand unusable with the crypto.shash backend; pcapng parser could loop forever on malformed block lengths, wrong timestamps on LuaJIT, crash on short-header QUIC packets.
  • fragmented_ip4 rewritten on plain Lua strings (API change: collect(ip, pkt); no data module dependency), with gap detection, bounded eviction queue and per-flow limits. It did not match ip4.parse output and could not reassemble.

Robustness on untrusted input

  • pack_compat_lib.unpack (LuaJIT / Lua 5.1 fallback) now raises on truncated input like native string.unpack (it returned nil or short strings).
  • TLS record / handshake / ClientHello parsers and the QUIC short-header parser return nil instead of raising; a single robust sni_from_tls is shared.

Hot path (measured on this branch vs #2, best of 3)

Benchmark LuaJIT Lua 5.4
QUIC frames: 3 CRYPTO + 1000 padding bytes 120 µs → 2.2 µs 254 µs → 36 µs
protected call 834 ns → not measurable (loop compiled away) 821 ns → 198 ns
TCP segment tostring 383 ns → 314 ns 742 ns → 398 ns
contains_ip 273 ns → 95 ns 894 ns → 593 ns
checksum6 1.79 µs → 1.59 µs 2.73 µs → 2.48 µs
bit53 band / bor (2 args) n/a 47 ns → 32 ns
DNS parse / pack unchanged unchanged
  • Behaviour change: consecutive PADDING bytes yield one frame with a length field instead of one frame per byte.
  • Behaviour change: TCP/IPv4 flag accessors recognise all-lower and all-upper names only (t.syn, t.SYN), no longer mixed case.
  • Lunatik backend: no per-packet closures, HKDF/shash transforms reused; binary hkdf_expand_bin / hkdf_expand_label_bin added (hex API unchanged). No measurable gain on the pure-Lua key derivation path.
  • No more per-length dynamic pack formats (c#{n}), which filled unbounded caches in pack_compat_lib.

Cleanup

Broken pcapng writer, ffi_stub.lua, stale tests/pack.moon and other dead code removed; need_bytes deduplicated; forced package.loaded reloads dropped; French comments translated.

Tests and coverage

  • make coverage (LuaJIT + Lua 5.4 cumulated, OpenSSL and mbedTLS backends exercised); cumulated line coverage 74% → 89.6%.
  • New non-blocking coverage CI job: prints the per-file table in the run summary and emits a warning (never a failure) below ipparse/tests/coverage_baseline.
  • The LuaJIT userspace job now loads OpenSSL and mbedTLS so FFI backend and QUIC session tests run instead of being skipped.

Verification (local)

  • All 78 commits compile and pass the LuaJIT suite.
  • Head: LuaJIT 553/553, Lua 5.1 553/553, Lua 5.4 577/577, Lua 5.5 577/577.
  • VM (Lunatik v5.1): kernel portable suite 484/484, All tests passed in VM.

🤖 Generated with Claude Code

jperon and others added 30 commits November 10, 2026 22:23
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
The loop bound #@-128 skipped the final chunk for lengths not a multiple
of 128, yielding truncated (or empty) output. Use #@-1 instead.

Co-Authored-By: Claude <noreply@anthropic.com>
string.gsub returns both the string and the substitution count; wrap the
call in parentheses so callers do not receive the extra value.

Co-Authored-By: Claude <noreply@anthropic.com>
The module table's __index fabricates a function for any missing key, so
_.leak_debug was always truthy. Use rawget to read the actual flag.

Co-Authored-By: Claude <noreply@anthropic.com>
State that memo/memoN caches are intentionally weak, and that
zero_indexed shifts elements down and clears the last slot rather than
preserving the original entries. Add a test freezing that behaviour.

Co-Authored-By: Claude <noreply@anthropic.com>
The LuaJIT/Lua 5.1 fallback was more permissive than native string.unpack:
it silently returned nil or shortened strings instead of raising
'data string too short'. Add the same bounds check used by read_uint to
the b, B, c, s (body) and x options.

Co-Authored-By: Claude <noreply@anthropic.com>
Guard the three parse entry points so a short or hostile input yields
nil, off instead of raising 'data string too short'. ClientHello wraps
its unpack call in pcall; the record and handshake parsers check their
fixed header width with need_bytes.

Co-Authored-By: Claude <noreply@anthropic.com>
l7.quic carried a less robust copy of sni_from_tls without an extension
progress guard nor a message-size cap. Export session.sni_from_tls and
re-export it from l7.quic, dropping the local copy and the now-unused
requires.

Co-Authored-By: Claude <noreply@anthropic.com>
parse_short_header unpacked a fixed-width c<N> string, which raised on a
packet shorter than the expected DCID and built the format from the
length. Check the available bytes with need_bytes and slice the DCID
directly instead.

Co-Authored-By: Claude <noreply@anthropic.com>
Validate block_len before dispatching (zero, misaligned, or past EOF
now stops parsing) and bound captured_len to the EPB. Extract
parse_pcapng_data so the parser is testable without a file.

Co-Authored-By: Claude <noreply@anthropic.com>
LuaJIT and bit_pure shift by 32 modulo 32, so lshift(high, 32) ==
high. Multiply the high word by 2^32 instead.

Co-Authored-By: Claude <noreply@anthropic.com>
pack_epb wrote a 24-byte header for six fields, so any file it
produced was corrupt. Nothing in the tree calls the pack/write
functions; drop them along with the unused imports and the
wrong RFC 2838 reference.

Co-Authored-By: Claude <noreply@anthropic.com>
No caller uses write_uint or ffi.C.memcpy/memset; cast_uint stays
because make_uint uses it.

Co-Authored-By: Claude <noreply@anthropic.com>
Nothing calls close_tfm; transforms are cached deliberately.

Co-Authored-By: Claude <noreply@anthropic.com>
pack_compat_lib works without FFI, so the fake ffi module and
its preload in parse_real_quic are dead weight.

Co-Authored-By: Claude <noreply@anthropic.com>
jperon and others added 28 commits November 10, 2026 23:11
Measured with make -C ipparse coverage-nobuild under LuaJIT + Lua 5.4:
COVERAGE_TOTAL=74.15, rounded down to 74.

Co-Authored-By: Claude <noreply@anthropic.com>
Runs make -C ipparse coverage-nobuild under LuaJIT + Lua 5.4 with
luacov, publishes the per-file summary to the step summary, and warns
(without failing) when the total is below ipparse/tests/coverage_baseline.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
On Lunatik's 16 KiB kernel stack, ipparse.fun (a large module parsed
recursively) fails to load when first required too deep in a require
chain. Require ipparse before any other dependency in l7/quic init and
session, as other modules already do.

Co-Authored-By: Claude <noreply@anthropic.com>
Load libcrypto and libmbedcrypto via a small ffi_libs.sh helper so the
LuaJIT coverage run reaches 460/460 instead of skipping the OpenSSL and
mbedTLS backends. Remeasured cumulative coverage baseline: 81%.

Co-Authored-By: Claude <noreply@anthropic.com>
Provide openssl and mbedtls in the coverage job and in the userspace
LuaJIT matrix entry, exporting LD_LIBRARY_PATH through
ipparse/tests/ffi_libs.sh so the backends are no longer skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…size

Exercise all integer and string options, both byte orders, widths 1-8,
signed/unsigned limits, packsize (fixed and variable formats), spaces,
alignment modifiers, >8 arguments, large buffers and inject. Tests run on
both runtimes, so they also check the fallback against native string.pack.

Co-Authored-By: Claude <noreply@anthropic.com>
Add big-endian 32-bit float bytes, n/d equivalence, floats embedded in a
mixed format, and packsize for floating-point options.

Co-Authored-By: Claude <noreply@anthropic.com>
The FFI bytes_to_double/bytes_to_float helpers fed string.byte results
straight into an FFI buffer. On a truncated string string.byte returns nil
and the assignment failed with "cannot convert 'nil' to 'unsigned char'"
instead of the native "data string too short" error. Add an explicit
bounds check, matching the pure-Lua ieee_decode path.

Co-Authored-By: Claude <noreply@anthropic.com>
Reload bit_compat with simulated bit/bit32/bit53/bit_pure backends via
package.preload, saving and restoring every touched entry even when loading
raises. Covers fallback_lshift/fallback_rshift/normalize and each selection
branch, plus the no-backend error. Guarded to userspace runtimes with
package.preload and math.floor.

Co-Authored-By: Claude <noreply@anthropic.com>
… sizes

Co-Authored-By: Claude <noreply@anthropic.com>
…backends

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
filter_quic_packets called bin2hex on a nil dst_connection_id when the
QUIC packet used a short header (the DCID length is not on the wire, so
parse() cannot populate it). Guard the value with an empty string.

Co-Authored-By: Claude <noreply@anthropic.com>
…tcuts

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Incomplete rows now pad the hex section to the full row width, so the
ASCII column starts at the same index on every line. ASCII groups are
concatenated without separators (as on complete lines) and no trailing
spaces are emitted. Complete-row output is unchanged and now frozen by
tests for 16, 32 and non-default (len=4, cols=3) inputs.

The expected values of the two incomplete-row tests ("hexdump pads an
incomplete final line", "hexdump honours offset, length, cols and
format") are updated to the corrected output: the old values froze the
misalignment and the stray ASCII separators.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@jperon
jperon merged commit 65fc75d into ipparse-perf-compat Nov 10, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant