Repository navigation
ipparse: LuaJIT pack perf, Lua 5.1/5.4/5.5 compat, Lunatik v5.1 fixes, kernel test suite and CI - #2
Merged
Merged
Conversation
LuaJIT's bit.band/bor/bxor accept any number of arguments, but the Lua 5.3+ compatibility layer only handled two. This broke e.g. TCP flag construction (bor with seven arguments), failing 6 test_tcp cases under Lua 5.4/5.5. Fold left to right, with a direct path for the two-argument case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- sha256_K had three wrong constants (K[17], K[18], K[54]). - The XOR helper was binary-only but Sum0/Sum1 call it with three terms, so the third term was silently dropped. - hmac() fed the message twice before finalizing the inner hash; feed it once from the returned chunker instead. Also fixes the streaming path, where the message was finalized prematurely. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Save and restore package.preload/package.loaded entries for crypto, crypto.ecb and the lunatik backend so the crypto stubs cannot leak into modules loaded later in the same test run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ECB stub invented a crypto.ecb module with encryptblock(); Lunatik
only exposes require("crypto").skcipher (lib/luacrypto_core.c registers
"skcipher") whose objects provide encrypt(iv, data)
(lib/luacrypto_skcipher.c luacrypto_skcipher_encrypt). Stub skcipher and
use an empty IV, matching ECB ivsize == 0 in luacrypto_checkiv.
Non-auth errors: luacrypto_aead_finish() throws every failing crypt
return code via lunatik_throw (lib/luacrypto_aead.c:144), and a too-short
input throws -EBADMSG (line 186). The backend wraps decrypt in pcall and
maps EBADMSG to a nil,error authentication failure while other failures
become nil,"aead(gcm(aes)) decrypt failed: ...", consistent with the
documented nil,error backend contract. Expect nil,error in the test
instead of a propagated exception.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the per-argument switch dispatch and ffi buffers with a per-format plan of small emitter closures built once and cached. Integers up to 4 bytes are emitted with string.char arithmetic, so LuaJIT no longer abandons traces; byte output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace per-section comprehension closures in pack with plain for loops filling one parts table, concatenated once; the body length is accumulated so header.size stays identical. Read label length and pointer bytes with string.byte after need_bytes has already validated the offsets. Bench (200k msgs, median of 4 interleaved runs, µs/msg): LuaJIT parse 3.93 -> 2.94, pack 3.30 -> 2.96 Lua 5.4 parse 8.01 -> 7.04, pack 2.58 -> 1.87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…able Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it bounds Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ield strings
Replace the per-field string encoders with writers that fill a module-level
byte buffer, then emit one string.char per run of integers. String options
(c, s, z) and floats flush the buffer first; arguments a1..a8 avoid select
and vararg in the fast path. This lets LuaJIT compile pack instead of
falling back to the interpreter (no FNEW/VARG aborts).
LuaJIT 2.1, us per call (before -> after):
pack(">H H") 0.259 -> 0.122
pack(">H B B H H H H") 0.847 -> 0.187
pack(">H H I4 s2") 0.664 -> 0.285
full DNS message pack 3.814 -> 1.904
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guest re-sources lib.sh, which recomputed BUILD_DIR from VMTEST_BUILD_DIR. That variable was never exported into cmd.sh, so a cache outside the repo was silently replaced by the default path inside the read-only repo, and guest-test.sh failed on kernelrelease. Resolve BUILD_DIR and LUNATIK_SRC to absolute paths in lib.sh and export both from the host into the guest wrapper. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A trailing-slash pattern only matches directories, so a natural `tools/vmtest/build -> /shared/cache` symlink showed up as untracked. Anchor the patterns with a leading slash, which also covers symlinks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All invocations wrote cmd.sh/out/rc/console.log under the same fixed /tmp/vmtest-rw. Two concurrent runs removed each other's directory and one failed with "VM did not run the guest command". Use mktemp -d per run (removed on success, kept on failure for the log), while keeping VMTEST_RW_DIR as an explicit opt-in override. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Use 64-bit shifts/masks when values are native integers (Lua 5.3+ and the Lunatik kernel, where math.floor and floats are unavailable); keep the high/low division split on LuaJIT doubles. Byte output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y module require ipparse.lib.pack_compat returns string.pack on Lua 5.3+ and the fallback on LuaJIT, so the test module also parses under the Lunatik kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read the current time through a configurable clock returning an integer in fixed units instead of os.time, which is absent in Lunatik. Defaults to os.time (1 Hz) when available, then linux.time (1 GHz) when the kernel module is loaded, and errors at creation when no clock exists. purge(max_age) now compares against max_age * clock_hz, keeping the comparison integer-only (no division, no floats). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mirror frames.moon's native_int check (math.floor == nil or math.type ~= nil) so the maximum-value case also runs in the Lunatik kernel, where math.type is absent but native 64-bit integers are available. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LuaJIT aborted the traces exiting pack's loop with "NYI: return to lower frame" because `return table.concat(parts)` / `return buf_string(...)` are tail calls, whose return target frame is not recordable. Assigning the result to a local and returning it removes every abort for b4.lua. LuaJIT 2.1, bench b4.lua (median of 5, µs): msg pack 1.76 (was 1.79-3.39, noisy) rr pack 0.28 sp >H H I4 s2 0.24 sp >H B B H H H H 0.19 sp >H H 0.12 -jv: no abort/NYI/blacklist on pack_compat_lib.lua (was 3 kinds). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When every option but the last only writes integer bytes and the last one emits a string (s/c/z/f/d as the final field), skip the parts list and table.concat entirely: write the length prefix (if any), flush the byte buffer, and concatenate once with the emitted string. This covers pack_rr's ">H H I4 s2", EDNS's ">Hs2" and the zero/lib string cases. LuaJIT 2.1, bench b4.lua (median of 5, µs): msg pack 1.41 (was 1.76) rr pack 0.20 (was 0.28) sp >H H I4 s2 0.17 (was 0.24) sp >H B B H H H H 0.18 (was 0.19) sp >H H 0.12 (was 0.12) -jv: still no abort/NYI/blacklist on pack_compat_lib.lua. Tests: LuaJIT 296/296, Lua 5.4 309/309. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pack allocated a fresh 8-slot table on every call, which dominated small formats. Reuse a module-level _args array instead (like _buf, pack never yields or calls user callbacks, so it is not reentrant). Formats consuming more than 8 values keep the one-off vararg table. LuaJIT 2.1, bench b4.lua (median of 5, µs): msg pack 1.19 (was 1.41) rr pack 0.18 (was 0.20) sp >H H I4 s2 0.137 (was 0.17) sp >H B B H H H H 0.146 (was 0.18) sp >H H 0.088 (was 0.12) -jv: still no abort/NYI/blacklist on pack_compat_lib.lua. Tests: LuaJIT 296/296, Lua 5.4 309/309. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lunatik_pusherrname emits "EBADMSG" on kernels >= 6.7, and on older kernels "%pE" of ERR_PTR(-err): "-EBADMSG" with CONFIG_SYMBOLIC_ERRNAME, or the signed decimal "-74" otherwise. Match these exact whole strings (after stripping Lua's optional "chunk:line: " prefix), so lookalike errnos such as -174 are not mistaken for a tag mismatch. decrypt always raises via lunatik_throw on failure and returns only the plaintext on success, so drop the bogus plaintext == "EBADMSG" check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lua 5.1 (PUC) has no ffi module, so requiring it failed before any pack operation could run: ./ipparse/lib/pack_compat_lib.lua:5: module 'ffi' not found Load FFI with pcall and keep the LuaJIT fast paths unchanged. Without FFI: - native endianness defaults to little-endian (no reliable probe); - integers up to 8 bytes use double arithmetic (2^53 precision), with signed l/j/i8 decoded by byte-wise two's complement so small negatives such as -2 stay exact; - f/d/n use a pure-Lua IEEE 754 codec handling zero, -0, infinities, NaN, normals, subnormals and round-to-nearest-even. Also fix the FFI float/double byte extraction, which used a table comprehension that produced an empty table (pack of a double returned an empty string), and decode i8 through int64_t instead of the lossy uint64 path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lua 5.1 does not support hexadecimal escapes in string literals, so test
data such as "\x01hello\xff" was read literally as x01helloxff. This
produced failures across most modules under Lua 5.1, for example:
FAIL init: hex2bin converts hex to bytes
FAIL l2/ethernet: mac2s converts binary to string
expected 'aa:bb:cc:dd:ee:ff', got '78:61:61:78:62:62'
FAIL l3/ip4: ip42s converts 4 bytes to dotted decimal
expected '192.168.1.1', got '120.99.48.120'
FAIL quic/varint: parse 1-byte value 0 ... got 14384
Rewrite every \xNN escape in the test sources as the equivalent \DDD
decimal escape, which is valid in Lua 5.1 through 5.5 and LuaJIT.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Track the official v5.1 release instead of a rewriting fork branch, and verify the tag still resolves to the expected commit so a moved tag fails loudly. An absent LUNATIK_SRC is cloned even when explicitly named; a stale harness-managed cache is re-cloned, while a user-provided checkout is only reported. Build code generation with the Lua 5.4 toolchain already used by CI (the module itself embeds Lua 5.5). Also key the kernel CI cache on KERNEL_VERSION and kernel.fragment only, so changing the Lunatik pin does not trigger a needless kernel rebuild. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run the VM suite with a fixed VMTEST_RW_DIR and upload the guest's console.log and out from it, instead of the useless host dmesg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Install the interpreters through Nix (Lua 5.5 is not in apt on ubuntu-latest), compile MoonScript once, then reuse the generated Lua files. Lua 5.1 is marked continue-on-error until its support lands. The Makefile gains LUA ?= luajit, used by test-nobuild and example, keeping the default behaviour unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add kernel-lunatik-head, which runs the same VM suite against luainkernel/lunatik at a workflow_dispatch-selected ref (default master), with its own LUNATIK_SRC and artefact. LUNATIK_REF/LUNATIK_COMMIT are overridable from the environment so versions.env only sets defaults. Document the upgrade procedure at the top of the workflow. Provide libbpf for master's eBPF loader, and make the guest install use the writable 5.4 Lua module paths instead of the read-only Nix store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lib/luacrypto_aead.c in v5.1 replaces lunatik_throw(L, -EBADMSG) with lunatik_pushfail(L, -EBADMSG), so a bad tag yields nil, "EBADMSG" instead of raising. aes_128_gcm_decrypt only inspected the pcall status slot and lost the error, returning nil, nil. Check both slots: the pcall error (old API) and, when the plaintext is nil, the second return value (v5.1+). Other failures still raise, as luacrypto_aead_finish and luacrypto_skcipher_crypt keep using lunatik_throw; encrypt and ECB therefore need no change. Co-Authored-By: Claude Opus 5.5 <noreply\@anthropic.com>
bit_pure is only the last-resort fallback of bit_compat (Lua 5.1 without `bit`) and relies on math.floor, which the Lua 5.5 kernel runtime does not provide. Guard the bit_pure cases behind math.floor so the kernel run keeps only the bit_compat cases instead of failing 16 assertions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Lua 5.5 kernel runtime has no floats: a single float literal makes the
whole chunk fail to compile ("malformed number near '.5'"), which also
dropped the integer cases of the module. Move every float/double/IEEE test
into ipparse/tests/lib/test_pack_compat_float.moon, registered in run_all
but deliberately absent from test_lunatik_kernel_portable. The remaining
module has no float literal, no `^` and no `/` in code, so it loads and runs
in the kernel.
The native kernel string.pack is correct for negative >i8: the pinned
Lunatik v5.1 lua/ sources built with -D_KERNEL (Lua 5.5.1, 64-bit
lua_Integer, SZINT=8) return fffffffffffffffe for pack(">i8", -2) and
round-trip -2; both cases PASS in the VM now that the module compiles.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Lunatik v5.1 computes LUAI_MAXCCALLS from the kernel stack actually consumed, and the Lua 5.5 parser is recursive in C. Loading ipparse.fun through a second require level (e.g. script -> ipparse.l7.dns -> fun) overflowed the 16 KiB kernel stack because `protected` nested two inline function expressions inside the xpcall call. Define the error handler and the returned function as block-level locals and build the per-call closure as a statement inside it. Parser C recursion drops enough that ipparse.fun now loads up to four require levels deep (measured: depth 4 OK, depth 5 overflows), covering real usage (script -> ipparse.l7.quic.session -> ipparse.l4.quic -> fun, or script -> ipparse.l7.dns -> fun). API and semantics are unchanged: same 26 arguments/return values, same xpcall error handling, same optional `op` callback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most modules exercised by the portable kernel suite require ipparse.fun two or three `require` levels down, inside a pcall(require …) chain that already consumes kernel stack. Load it once at the shallowest C-stack point before the loop so the suite no longer depends on that chain depth and stays clear of Lunatik's stack-derived C-call guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The double -> uint64_t conversion is undefined for negative values, and older LuaJIT releases saturate it to 0x8000000000000000, corrupting 64-bit signed writes. Route negatives through int64_t first, then reinterpret the bits as unsigned, keeping the direct cast for non-negative values (so values >= 2^63 are unaffected). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reverse-lookup cache kept returning a stale key after the forward entry was modified or deleted. Validate the cached key with rawget(self, key) == val before reuse; rawget avoids re-entering the __index metamethod, which would recurse on an absent key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ipparse performance work for LuaJIT, Lua 5.1/5.4/5.5 compatibility, fixes for real Lunatik usage (upstream
v5.1, Lua 5.5 in the kernel), a kernel-side portable test suite, VM harness fixes and a multi-runtime CI.Performance (LuaJIT)
pack_compat_lib: formats are compiled once into byte writers that fill a shared buffer; singlestring.char; no varargs/closures in the hot loop; no tail-call returns (they aborted traces withNYI: return to lower frame); trailing-string fast path; reused argument array. No trace aborts left under-jv.l7/dns: singleconcatin messagepack,string.byteinlabel.Compatibility
band/bor/bxorinbit53(TCP flags were dropped bytcp.new).lib/sha.lua: three wrong SHA-256 constants, 2-argXORcalled with 3 args, HMAC double-feed (broke HKDF/QUIC keys on Lua 5.3+).bit_pure), FFI made optional inpack_compat_lib(pure-Lua 64-bit ints and IEEE 754),\xNNtest escapes.pack("<d", x)returned an empty string; negative 64-bit writes were undefined (double →uint64_t), wrong on older LuaJIT.Lunatik (kernel)
ipparse.funoverflowed the C-stack guard when loaded onerequirelevel down under v5.1 (real usage, e.g.l7.quic.session):protectedflattened. Margin: loading still fails at depth 5.nil, "EBADMSG") and every errno-name form (EBADMSG,-EBADMSG,-74); a plaintext equal to"EBADMSG"is no longer rejected.osin kernel paths: 8-byte QUIC varints,tcp_streamclock (injectable,linux.timefallback).bidirectionalreverse cache revalidated withrawget(stale entries after live updates).Tests, harness, CI
test_lunatik_kernel_portable(parse/pack modules run inside the kernel).v5.1(commit checked); non-blockingkernel-lunatik-headjob againstmaster(or any ref viaworkflow_dispatch); VM console/output uploaded on failure.Verification (local)
All tests passed in VM.actionlintclean.🤖 Generated with Claude Code