Repository navigation
ipparse: review fixes, robustness, hot path, coverage and kernel stack budget - #4
Merged
Merged
Conversation
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
The loop bound #@-128 skipped the final chunk for lengths not a multiple of 128, yielding truncated (or empty) output. Use #@-1 instead. Co-Authored-By: Claude <noreply@anthropic.com>
string.gsub returns both the string and the substitution count; wrap the call in parentheses so callers do not receive the extra value. Co-Authored-By: Claude <noreply@anthropic.com>
The module table's __index fabricates a function for any missing key, so _.leak_debug was always truthy. Use rawget to read the actual flag. Co-Authored-By: Claude <noreply@anthropic.com>
State that memo/memoN caches are intentionally weak, and that zero_indexed shifts elements down and clears the last slot rather than preserving the original entries. Add a test freezing that behaviour. Co-Authored-By: Claude <noreply@anthropic.com>
The LuaJIT/Lua 5.1 fallback was more permissive than native string.unpack: it silently returned nil or shortened strings instead of raising 'data string too short'. Add the same bounds check used by read_uint to the b, B, c, s (body) and x options. Co-Authored-By: Claude <noreply@anthropic.com>
Guard the three parse entry points so a short or hostile input yields nil, off instead of raising 'data string too short'. ClientHello wraps its unpack call in pcall; the record and handshake parsers check their fixed header width with need_bytes. Co-Authored-By: Claude <noreply@anthropic.com>
l7.quic carried a less robust copy of sni_from_tls without an extension progress guard nor a message-size cap. Export session.sni_from_tls and re-export it from l7.quic, dropping the local copy and the now-unused requires. Co-Authored-By: Claude <noreply@anthropic.com>
parse_short_header unpacked a fixed-width c<N> string, which raised on a packet shorter than the expected DCID and built the format from the length. Check the available bytes with need_bytes and slice the DCID directly instead. Co-Authored-By: Claude <noreply@anthropic.com>
Validate block_len before dispatching (zero, misaligned, or past EOF now stops parsing) and bound captured_len to the EPB. Extract parse_pcapng_data so the parser is testable without a file. Co-Authored-By: Claude <noreply@anthropic.com>
LuaJIT and bit_pure shift by 32 modulo 32, so lshift(high, 32) == high. Multiply the high word by 2^32 instead. Co-Authored-By: Claude <noreply@anthropic.com>
pack_epb wrote a 24-byte header for six fields, so any file it produced was corrupt. Nothing in the tree calls the pack/write functions; drop them along with the unused imports and the wrong RFC 2838 reference. Co-Authored-By: Claude <noreply@anthropic.com>
No caller uses write_uint or ffi.C.memcpy/memset; cast_uint stays because make_uint uses it. Co-Authored-By: Claude <noreply@anthropic.com>
Nothing calls close_tfm; transforms are cached deliberately. Co-Authored-By: Claude <noreply@anthropic.com>
pack_compat_lib works without FFI, so the fake ffi module and its preload in parse_real_quic are dead weight. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…size Exercise all integer and string options, both byte orders, widths 1-8, signed/unsigned limits, packsize (fixed and variable formats), spaces, alignment modifiers, >8 arguments, large buffers and inject. Tests run on both runtimes, so they also check the fallback against native string.pack. Co-Authored-By: Claude <noreply@anthropic.com>
Add big-endian 32-bit float bytes, n/d equivalence, floats embedded in a mixed format, and packsize for floating-point options. Co-Authored-By: Claude <noreply@anthropic.com>
The FFI bytes_to_double/bytes_to_float helpers fed string.byte results straight into an FFI buffer. On a truncated string string.byte returns nil and the assignment failed with "cannot convert 'nil' to 'unsigned char'" instead of the native "data string too short" error. Add an explicit bounds check, matching the pure-Lua ieee_decode path. Co-Authored-By: Claude <noreply@anthropic.com>
Reload bit_compat with simulated bit/bit32/bit53/bit_pure backends via package.preload, saving and restoring every touched entry even when loading raises. Covers fallback_lshift/fallback_rshift/normalize and each selection branch, plus the no-backend error. Guarded to userspace runtimes with package.preload and math.floor. Co-Authored-By: Claude <noreply@anthropic.com>
… sizes Co-Authored-By: Claude <noreply@anthropic.com>
…backends Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
filter_quic_packets called bin2hex on a nil dst_connection_id when the QUIC packet used a short header (the DCID length is not on the wire, so parse() cannot populate it). Guard the value with an empty string. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…tcuts Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Incomplete rows now pad the hex section to the full row width, so the
ASCII column starts at the same index on every line. ASCII groups are
concatenated without separators (as on complete lines) and no trailing
spaces are emitted. Complete-row output is unchanged and now frozen by
tests for 16, 32 and non-default (len=4, cols=3) inputs.
The expected values of the two incomplete-row tests ("hexdump pads an
incomplete final line", "hexdump honours offset, length, cols and
format") are updated to the corrected output: the old values froze the
misalignment and the stray ASCII separators.
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Probe every ipparse entry module through a simulated chain of nested package.preload requires and require a minimum first-load depth margin, guarding against C stack overflow when a big module is parsed at the bottom of a require chain. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Make require "ipparse" the first require of each entry module so the shared base is parsed at the shallowest require depth under Lunatik's 16 KiB kernel stack. Co-Authored-By: Claude <noreply@anthropic.com>
Report the first failing probe's error and classify it as a stack or non-stack stop, and check that each entry module loads ipparse first without pulling in the heavy fun facade. Co-Authored-By: Claude <noreply@anthropic.com>
Add the four ipparse.fun submodules to the canary entry modules, exempt them from the base-first guards (protocol modules never load them) and allow per-module depth overrides for the facade and heavy submodules. Co-Authored-By: Claude <noreply@anthropic.com>
Add BYTECODE and LUNATIC make variables. When BYTECODE=1 (or strip), compile each installed .lua in place with Lunatik's lunatic, keeping debug info or stripping it. Files that lunatic rejects as kernel incompatible are kept as source with a warning; a missing lunatic is fatal. Co-Authored-By: Claude <noreply@anthropic.com>
After the source kernel pass, reinstall ipparse with BYTECODE=1 using Lunatik's lunatic and rerun the suite, unless VMTEST_BYTECODE=0. Skip the second pass when lunatic is absent. Transmit VMTEST_BYTECODE to the guest and document it. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…call Require ipparse.lib.bit53 without pcall whenever math.type is available (or math.floor is absent, as in the Lunatik kernel), so a load failure such as a C stack overflow propagates with its real message instead of being swallowed and replaced by the generic no-backend error. A fake third-party 'bit' module can no longer shadow the native 64-bit operators on Lua 5.3+. Tests: under a native-integer runtime the native backend must win over a preloaded fake 'bit' (identity with ipparse.lib.bit53, 64-bit band) and a bit53 load error must surface 'boom-stack'. The bit/bit32/bit_pure chain tests are now restricted to non-native runtimes, and bit53 must be ignored there. Co-Authored-By: Claude <noreply@anthropic.com>
Record the stack canary measurements for source and bytecode installs in the Kernel stack budget section, and note the facade/protected depths in the canary's REQUIRED_DEPTH_OVERRIDES comment. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Targets
master. Carries the content of #3 (which was merged into its stacked base branchipparse-perf-compat, not intomaster) plus the kernel stack-budget work below. Result of a full read-through of the ipparse library code: confirmed bugs fixed (each with a regression test), malformed-input robustness, hot-path work, dead-code cleanup, and test coverage tooling.Bug fixes (each commit carries its regression test)
packwas not exported (ip.packon IPv6 raised);s2net6without mask raised;vtfwas frozen atnew; UDP zero checksum now emitted as 0xFFFF.packraised whenoptionswas omitted; raw flag keys given tonewshadowed__newindex(t.syn = falsewas ignored);ihl < 5and TCP data offset < 20 are rejected.__newindexsilently dropped new fields;remove_header_protectionraised on short packets;version_negotiationrewritten (32-bit versions,packalways raised).rcode/opcode/zgetters return the numeric value (API change, they returned booleans);rasetter added; unknown header fields are stored; unknown EDNS options no longer repack as"nil"; client-subnet repack fixed (globalunpack, lost address);classes.NONEis 254,ANYadded.lbin2hexdropped the tail of its input;bin2hexoverflowed the Lua stack in the kernel on 128-byte chunks;hex2binreturned two values;protectedalways printed a traceback (leak_debugwas always truthy);hexdumpaligns the ASCII column on incomplete lines.bit53.arshiftwas a logical shift; five wrong TLS handshake type codes; HKDF expand unusable with thecrypto.shashbackend; pcapng parser could loop forever on malformed block lengths, wrong timestamps on LuaJIT, crash on short-header QUIC packets.fragmented_ip4rewritten on plain Lua strings (API change:collect(ip, pkt); nodatamodule dependency), with gap detection, bounded eviction queue and per-flow limits. It did not matchip4.parseoutput and could not reassemble.Robustness on untrusted input
pack_compat_lib.unpack(LuaJIT / Lua 5.1 fallback) now raises on truncated input like nativestring.unpack(it returnednilor short strings).nilinstead of raising; a single robustsni_from_tlsis shared.Hot path (measured on this branch vs #2, best of 3)
protectedcalltostringcontains_ipchecksum6bit53band/bor(2 args)lengthfield instead of one frame per byte.t.syn,t.SYN), no longer mixed case.hkdf_expand_bin/hkdf_expand_label_binadded (hex API unchanged). No measurable gain on the pure-Lua key derivation path.c#{n}), which filled unbounded caches inpack_compat_lib.Cleanup
Broken pcapng writer,
ffi_stub.lua, staletests/pack.moonand other dead code removed;need_bytesdeduplicated; forcedpackage.loadedreloads dropped; French comments translated.Tests and coverage
make coverage(LuaJIT + Lua 5.4 cumulated, OpenSSL and mbedTLS backends exercised); cumulated line coverage 74% → 89.6%.coverageCI job: prints the per-file table in the run summary and emits a warning (never a failure) belowipparse/tests/coverage_baseline.Kernel stack budget (new since #3)
Lunatik gives Lua 5/8 of the 16 KiB kernel stack; a large module first parsed at the bottom of a
requirechain raisesC stack overflow.tests/test_lunatik_kernel_stack_canary): for each entry module, measures the deepest nested-requirechain it can be first loaded from, fails belowREQUIRED_DEPTH, and checks the rule below mechanically.ipparse.funsplit intofun.tables,fun.memo,fun.iter,fun.protected;ipparse.funstays as a facade with the same API. Protocol modules only loadipparse.fun.tables.AGENTS.md, applied to the 16 entry modules):require "ipparse"is the firstrequire.make install BYTECODE=1(orBYTECODE=strip) compiles the installed files with Lunatik's ownlunatic; the VM harness now runs the kernel suite twice, source then bytecode (VMTEST_BYTECODE=0to skip).bit_compatswallowed the real load error behind "no bitwise compatibility backend available" and could prefer a third-party 32-bitbitmodule on Lua 5.3+; the backend is now chosen by capability, andbit53is loaded withoutpcall.Measured in the VM (Lunatik v5.1; probes run under
pcall, so about one level pessimistic):ipparse(shared base)l2.ethernet,l4.tcp,l7.dns,l7.tlsl3.ip,l3.ip4,l3.ip6,l4.udpl4.quicl7.quic.sessionl7.quicipparse.funfacadeVerification (local)
master.All tests passed in VM.🤖 Generated with Claude Code