Skip to content

ipparse: review fixes, robustness, hot path, coverage and kernel stack budget - #4

Merged
jperon merged 91 commits into
masterfrom
ipparse-kernel-stack
Nov 11, 2026
Merged

jperon merged 91 commits into
masterfrom
ipparse-kernel-stack

Conversation

@jperon

@jperon jperon commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Targets master. Carries the content of #3 (which was merged into its stacked base branch ipparse-perf-compat, not into master) plus the kernel stack-budget work below. Result of a full read-through of the ipparse library code: confirmed bugs fixed (each with a regression test), malformed-input robustness, hot-path work, dead-code cleanup, and test coverage tooling.

Bug fixes (each commit carries its regression test)

  • IPv6: pack was not exported (ip.pack on IPv6 raised); s2net6 without mask raised; vtf was frozen at new; UDP zero checksum now emitted as 0xFFFF.
  • IPv4 / TCP: pack raised when options was omitted; raw flag keys given to new shadowed __newindex (t.syn = false was ignored); ihl < 5 and TCP data offset < 20 are rejected.
  • QUIC: long-header version packed on 16 bits instead of 32; __newindex silently dropped new fields; remove_header_protection raised on short packets; version_negotiation rewritten (32-bit versions, pack always raised).
  • DNS: rcode/opcode/z getters return the numeric value (API change, they returned booleans); ra setter added; unknown header fields are stored; unknown EDNS options no longer repack as "nil"; client-subnet repack fixed (global unpack, lost address); classes.NONE is 254, ANY added.
  • Core: lbin2hex dropped the tail of its input; bin2hex overflowed the Lua stack in the kernel on 128-byte chunks; hex2bin returned two values; protected always printed a traceback (leak_debug was always truthy); hexdump aligns the ASCII column on incomplete lines.
  • Misc: bit53.arshift was a logical shift; five wrong TLS handshake type codes; HKDF expand unusable with the crypto.shash backend; pcapng parser could loop forever on malformed block lengths, wrong timestamps on LuaJIT, crash on short-header QUIC packets.
  • fragmented_ip4 rewritten on plain Lua strings (API change: collect(ip, pkt); no data module dependency), with gap detection, bounded eviction queue and per-flow limits. It did not match ip4.parse output and could not reassemble.

Robustness on untrusted input

  • pack_compat_lib.unpack (LuaJIT / Lua 5.1 fallback) now raises on truncated input like native string.unpack (it returned nil or short strings).
  • TLS record / handshake / ClientHello parsers and the QUIC short-header parser return nil instead of raising; a single robust sni_from_tls is shared.

Hot path (measured on this branch vs #2, best of 3)

Benchmark LuaJIT Lua 5.4
QUIC frames: 3 CRYPTO + 1000 padding bytes 120 µs → 2.2 µs 254 µs → 36 µs
protected call 834 ns → not measurable (loop compiled away) 821 ns → 198 ns
TCP segment tostring 383 ns → 314 ns 742 ns → 398 ns
contains_ip 273 ns → 95 ns 894 ns → 593 ns
checksum6 1.79 µs → 1.59 µs 2.73 µs → 2.48 µs
bit53 band / bor (2 args) n/a 47 ns → 32 ns
DNS parse / pack unchanged unchanged
  • Behaviour change: consecutive PADDING bytes yield one frame with a length field instead of one frame per byte.
  • Behaviour change: TCP/IPv4 flag accessors recognise all-lower and all-upper names only (t.syn, t.SYN), no longer mixed case.
  • Lunatik backend: no per-packet closures, HKDF/shash transforms reused; binary hkdf_expand_bin / hkdf_expand_label_bin added (hex API unchanged). No measurable gain on the pure-Lua key derivation path.
  • No more per-length dynamic pack formats (c#{n}), which filled unbounded caches in pack_compat_lib.

Cleanup

Broken pcapng writer, ffi_stub.lua, stale tests/pack.moon and other dead code removed; need_bytes deduplicated; forced package.loaded reloads dropped; French comments translated.

Tests and coverage

  • make coverage (LuaJIT + Lua 5.4 cumulated, OpenSSL and mbedTLS backends exercised); cumulated line coverage 74% → 89.6%.
  • New non-blocking coverage CI job: prints the per-file table in the run summary and emits a warning (never a failure) below ipparse/tests/coverage_baseline.
  • The LuaJIT userspace job now loads OpenSSL and mbedTLS so FFI backend and QUIC session tests run instead of being skipped.

Kernel stack budget (new since #3)

Lunatik gives Lua 5/8 of the 16 KiB kernel stack; a large module first parsed at the bottom of a require chain raises C stack overflow.

  • Stack canary (tests/test_lunatik_kernel_stack_canary): for each entry module, measures the deepest nested-require chain it can be first loaded from, fails below REQUIRED_DEPTH, and checks the rule below mechanically.
  • ipparse.fun split into fun.tables, fun.memo, fun.iter, fun.protected; ipparse.fun stays as a facade with the same API. Protocol modules only load ipparse.fun.tables.
  • Rule (documented in AGENTS.md, applied to the 16 entry modules): require "ipparse" is the first require.
  • Bytecode: make install BYTECODE=1 (or BYTECODE=strip) compiles the installed files with Lunatik's own lunatic; the VM harness now runs the kernel suite twice, source then bytecode (VMTEST_BYTECODE=0 to skip).
  • Fix found by the canary: bit_compat swallowed the real load error behind "no bitwise compatibility backend available" and could prefer a third-party 32-bit bit module on Lua 5.3+; the backend is now chosen by capability, and bit53 is loaded without pcall.

Measured in the VM (Lunatik v5.1; probes run under pcall, so about one level pessimistic):

Module Before Source Bytecode
ipparse (shared base) 0 8 ≥ 12
l2.ethernet, l4.tcp, l7.dns, l7.tls 0 7 ≥ 9
l3.ip, l3.ip4, l3.ip6, l4.udp 0 6 11
l4.quic 0 4 11
l7.quic.session 0 3 9
l7.quic 0 2 8
ipparse.fun facade 1 0 ≥ 12

Verification (local)

  • Tree identical to the branch validated below; 91 linear commits on master.
  • Head: LuaJIT 558/558, Lua 5.1 558/558, Lua 5.4 578/578, Lua 5.5 578/578.
  • VM (Lunatik v5.1): kernel portable suite 489/489 and stack canary 24/24 in both source and bytecode passes, All tests passed in VM.

🤖 Generated with Claude Code

jperon and others added 30 commits November 11, 2026 04:41
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
The loop bound #@-128 skipped the final chunk for lengths not a multiple
of 128, yielding truncated (or empty) output. Use #@-1 instead.

Co-Authored-By: Claude <noreply@anthropic.com>
string.gsub returns both the string and the substitution count; wrap the
call in parentheses so callers do not receive the extra value.

Co-Authored-By: Claude <noreply@anthropic.com>
The module table's __index fabricates a function for any missing key, so
_.leak_debug was always truthy. Use rawget to read the actual flag.

Co-Authored-By: Claude <noreply@anthropic.com>
State that memo/memoN caches are intentionally weak, and that
zero_indexed shifts elements down and clears the last slot rather than
preserving the original entries. Add a test freezing that behaviour.

Co-Authored-By: Claude <noreply@anthropic.com>
The LuaJIT/Lua 5.1 fallback was more permissive than native string.unpack:
it silently returned nil or shortened strings instead of raising
'data string too short'. Add the same bounds check used by read_uint to
the b, B, c, s (body) and x options.

Co-Authored-By: Claude <noreply@anthropic.com>
Guard the three parse entry points so a short or hostile input yields
nil, off instead of raising 'data string too short'. ClientHello wraps
its unpack call in pcall; the record and handshake parsers check their
fixed header width with need_bytes.

Co-Authored-By: Claude <noreply@anthropic.com>
l7.quic carried a less robust copy of sni_from_tls without an extension
progress guard nor a message-size cap. Export session.sni_from_tls and
re-export it from l7.quic, dropping the local copy and the now-unused
requires.

Co-Authored-By: Claude <noreply@anthropic.com>
parse_short_header unpacked a fixed-width c<N> string, which raised on a
packet shorter than the expected DCID and built the format from the
length. Check the available bytes with need_bytes and slice the DCID
directly instead.

Co-Authored-By: Claude <noreply@anthropic.com>
Validate block_len before dispatching (zero, misaligned, or past EOF
now stops parsing) and bound captured_len to the EPB. Extract
parse_pcapng_data so the parser is testable without a file.

Co-Authored-By: Claude <noreply@anthropic.com>
LuaJIT and bit_pure shift by 32 modulo 32, so lshift(high, 32) ==
high. Multiply the high word by 2^32 instead.

Co-Authored-By: Claude <noreply@anthropic.com>
pack_epb wrote a 24-byte header for six fields, so any file it
produced was corrupt. Nothing in the tree calls the pack/write
functions; drop them along with the unused imports and the
wrong RFC 2838 reference.

Co-Authored-By: Claude <noreply@anthropic.com>
No caller uses write_uint or ffi.C.memcpy/memset; cast_uint stays
because make_uint uses it.

Co-Authored-By: Claude <noreply@anthropic.com>
Nothing calls close_tfm; transforms are cached deliberately.

Co-Authored-By: Claude <noreply@anthropic.com>
pack_compat_lib works without FFI, so the fake ffi module and
its preload in parse_real_quic are dead weight.

Co-Authored-By: Claude <noreply@anthropic.com>
jperon and others added 29 commits November 11, 2026 04:41
Co-Authored-By: Claude <noreply@anthropic.com>
…size

Exercise all integer and string options, both byte orders, widths 1-8,
signed/unsigned limits, packsize (fixed and variable formats), spaces,
alignment modifiers, >8 arguments, large buffers and inject. Tests run on
both runtimes, so they also check the fallback against native string.pack.

Co-Authored-By: Claude <noreply@anthropic.com>
Add big-endian 32-bit float bytes, n/d equivalence, floats embedded in a
mixed format, and packsize for floating-point options.

Co-Authored-By: Claude <noreply@anthropic.com>
The FFI bytes_to_double/bytes_to_float helpers fed string.byte results
straight into an FFI buffer. On a truncated string string.byte returns nil
and the assignment failed with "cannot convert 'nil' to 'unsigned char'"
instead of the native "data string too short" error. Add an explicit
bounds check, matching the pure-Lua ieee_decode path.

Co-Authored-By: Claude <noreply@anthropic.com>
Reload bit_compat with simulated bit/bit32/bit53/bit_pure backends via
package.preload, saving and restoring every touched entry even when loading
raises. Covers fallback_lshift/fallback_rshift/normalize and each selection
branch, plus the no-backend error. Guarded to userspace runtimes with
package.preload and math.floor.

Co-Authored-By: Claude <noreply@anthropic.com>
… sizes

Co-Authored-By: Claude <noreply@anthropic.com>
…backends

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
filter_quic_packets called bin2hex on a nil dst_connection_id when the
QUIC packet used a short header (the DCID length is not on the wire, so
parse() cannot populate it). Guard the value with an empty string.

Co-Authored-By: Claude <noreply@anthropic.com>
…tcuts

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Incomplete rows now pad the hex section to the full row width, so the
ASCII column starts at the same index on every line. ASCII groups are
concatenated without separators (as on complete lines) and no trailing
spaces are emitted. Complete-row output is unchanged and now frozen by
tests for 16, 32 and non-default (len=4, cols=3) inputs.

The expected values of the two incomplete-row tests ("hexdump pads an
incomplete final line", "hexdump honours offset, length, cols and
format") are updated to the corrected output: the old values froze the
misalignment and the stray ASCII separators.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Probe every ipparse entry module through a simulated chain of nested
package.preload requires and require a minimum first-load depth margin,
guarding against C stack overflow when a big module is parsed at the
bottom of a require chain.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Make require "ipparse" the first require of each entry module so the
shared base is parsed at the shallowest require depth under Lunatik's
16 KiB kernel stack.

Co-Authored-By: Claude <noreply@anthropic.com>
Report the first failing probe's error and classify it as a stack or
non-stack stop, and check that each entry module loads ipparse first
without pulling in the heavy fun facade.

Co-Authored-By: Claude <noreply@anthropic.com>
Add the four ipparse.fun submodules to the canary entry modules, exempt
them from the base-first guards (protocol modules never load them) and
allow per-module depth overrides for the facade and heavy submodules.

Co-Authored-By: Claude <noreply@anthropic.com>
Add BYTECODE and LUNATIC make variables. When BYTECODE=1 (or strip),
compile each installed .lua in place with Lunatik's lunatic, keeping
debug info or stripping it. Files that lunatic rejects as kernel
incompatible are kept as source with a warning; a missing lunatic is
fatal.

Co-Authored-By: Claude <noreply@anthropic.com>
After the source kernel pass, reinstall ipparse with BYTECODE=1 using
Lunatik's lunatic and rerun the suite, unless VMTEST_BYTECODE=0. Skip
the second pass when lunatic is absent. Transmit VMTEST_BYTECODE to the
guest and document it.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…call

Require ipparse.lib.bit53 without pcall whenever math.type is available
(or math.floor is absent, as in the Lunatik kernel), so a load failure
such as a C stack overflow propagates with its real message instead of
being swallowed and replaced by the generic no-backend error. A fake
third-party 'bit' module can no longer shadow the native 64-bit
operators on Lua 5.3+.

Tests: under a native-integer runtime the native backend must win over a
preloaded fake 'bit' (identity with ipparse.lib.bit53, 64-bit band) and a
bit53 load error must surface 'boom-stack'. The bit/bit32/bit_pure chain
tests are now restricted to non-native runtimes, and bit53 must be
ignored there.

Co-Authored-By: Claude <noreply@anthropic.com>
Record the stack canary measurements for source and bytecode installs in
the Kernel stack budget section, and note the facade/protected depths in
the canary's REQUIRED_DEPTH_OVERRIDES comment.

Co-Authored-By: Claude <noreply@anthropic.com>
@jperon
jperon merged commit 265876b into master Nov 11, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant