Milestones 1–2: PEG parser engine port and the accept/reject conformance gate - #2
Merged
Conversation
Scaffold the module (go.mod, MIT LICENSE + LICENSE.DUCKDB, CI, CLAUDE.md)
and implement the engine layer described in PLAN.md:
- internal/grammar: vendor the .gram/.list files verbatim from DuckDB
commit 044a04a7cd39e6e8235f756597ae42dde084e5e5 (provenance + SHA-256 in
internal/grammar/README.md) and port the grammar loader
(peg_parser.cpp/parsed_grammar.cpp) plus the assembly step from
inline_grammar.py (keyword lists become ordered-choice rules).
- token: token kinds, Token{Kind, Text, Span}, and DuckDB's keyword
categories. The vendored lists overlap outside reserved/unreserved
(32 keywords sit in two lists at this pin), so category membership is a
bitmask, matching the pinned build's effective behavior rather than the
exactly-one-list claim in upstream docs.
- lexer: port of base_tokenizer.cpp + parser_tokenizer.cpp, including
dollar-quoted strings with the $[0-9] parameter disambiguation and
quote-rewrite, E'' escape strings, nested block comments, the
PostgreSQL operator '+' trimming rule, and the parser behavior's
tokenize errors.
- internal/matcher: matcher tree compiled from the grammar
(matcher_factory.cpp), packrat memoization over upstream's designated
rule list (parser_packrat.cpp), the transcribed rule-override table
(identifier variants, literals, EndOfInput), ParseResult tree with
upstream-style dumping, and furthest-failure syntax errors. Upstream
quirks are ported and pinned by tests: negative lookahead drops only
the '!' operator (its operand still consumes), and '/' binds only the
immediately preceding element.
- cmd/debug-parse: dump tokens or the raw ParseResult tree, peeling one
TopLevelStatement at a time like upstream's parse loop.
The 19-unmatched-parens pathological case from upstream's blog post runs
in ~1ms as a benchmark; go test -race is clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CTQBC9AzfdFPFsZtwn1mHY
Advance the pin to 8cbdaba6acb50db7780d22a65dd131584d472262, the commit of
the DuckDB nightly CLI v2.0.0-alpha38195 (the oracle): grammar, corpus
expectations, and oracle must come from the same commit, and the nightly
is the pin we can verify against. Re-vendor the grammar (describe.gram,
select.gram, create_view.gram, one unreserved keyword removed) and update
the provenance README and keyword-count tests.
New corpus machinery, per PLAN.md § Conformance oracle and corpus:
- internal/sqltest: sqllogictest .test reader, extraction only. Records
end at empty-or-comment lines exactly as upstream's SQLLogicParser does;
template substitutions (${...}) are skipped, never expanded.
- internal/testfile: the consolidated corpus format (cases separated by
==, SQL/expectation by --) plus *.metadata.json todo/skip sidecars
keyed by case content hash so entries survive regeneration.
- internal/duckdbsrc: locate and verify the pinned CLI, run statements
against :memory: with extension autoloading disabled, classify Parser
Error (must-reject) vs success/post-parse errors (must-accept).
- cmd/regenerate: extract test/sql/**/*.test + test/fuzzer/**/*.test,
dedup, run the oracle concurrently, write parser/testdata. Expectations
are never edited by hand.
- cmd/next-test: print the next todo case from corpus metadata.
- parser/parser_test.go: the corpus harness and milestone-2 gate, with
-check-parse case dumps and -record-todos triage (auto-records only
transformer-raised rejects; engine disagreements always fail).
Corpus: 4,071 files, 45,226 unique statements (43,819 must-accept, 1,407
must-reject) from 4,178 upstream .test files. The gate is green: every
grammar-level accept/reject decision agrees with the pinned binary. 290
cases are todo (upstream rejects them in its transformer - positional
reference bounds, chained comparisons, recursive-CTE modifier checks,
VALUES arity, and similar - which land with milestones 3-5), and 11 are
skipped with notes (parser-extension records and nested bind-time parses
the grammar never sees).
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CTQBC9AzfdFPFsZtwn1mHY
Regeneration pipeline correctness:
- cmd/regenerate now removes corpus files (and their sidecars) that a
full run did not produce, so tests deleted or renamed upstream stop
gating darkwing against an older pin's verdicts; orphaned sidecars are
cleaned up the same way. Partial runs (-only/-limit) skip both the
cleanup and the corpus-wide README rewrite instead of clobbering state
for files they did not regenerate.
- Oracle timeouts and crashes are no longer recorded as must-accept: the
CLI produced no verdict (it may have died inside its own parser), so
those statements are dropped as unverified and counted in the README.
The 12 timeout statements are gone from the corpus; one file whose
only case was unverified is deleted by the new stale cleanup.
- The bind-time nested-parse imprecision (query('...') raising Parser
Error for its inner string) is now documented on duckdbsrc.classify
and in the corpus README: triage stays manual because such a
disagreement is indistinguishable from a genuine engine bug.
Format and code hygiene:
- testfile.CanStore also rejects empty statements, statements whose
first line is empty (Read drops blank lines between cases), and
statements whose first line starts with '#' (header absorption), with
tests - future corpus sources hit the skippedUnstorable path instead
of silent round-trip corruption.
- The TopLevelStatement peel loop now lives in one place,
matcher.Engine.MatchAll (returning partial results alongside errors);
the harness classify/dumpCase, cmd/debug-parse, and the matcher tests
all use it.
- duckdbsrc uses errors.As instead of the hand-rolled isExit helper;
the dead token import and its keep-alive are gone from parser_test.
The gate stays green over the regenerated corpus (43,807 must-accept,
1,407 must-reject); go test -race ./... is clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CTQBC9AzfdFPFsZtwn1mHY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the first two milestones of PLAN.md: the engine (tokenizer, grammar loader, matcher) and the corpus conformance gate — darkwing accepts a statement iff the pinned DuckDB binary parses it.
Milestone 1 — the engine
internal/grammar/— the.gram/.listfiles vendored verbatim from the pinned DuckDB commit (provenance + SHA-256 checksums ininternal/grammar/README.md), embedded viago:embed, plus the grammar loader: a port ofpeg_parser.cpp/parsed_grammar.cppand the assembly step frominline_grammar.py(keyword lists become ordered-choice rules).token/— token kinds,Token{Kind, Text, Span}, and DuckDB's keyword categories. The vendored lists overlap outside reserved/unreserved (32 keywords sit in two lists at this pin, e.g.GENERATEDis both column-name and func-name), so category membership is a bitmask matching the pinned build's effective behavior, not the exactly-one-list claim in upstream docs.lexer/— port ofbase_tokenizer.cpp+parser_tokenizer.cpp: dollar-quoted strings (with the$[0-9]parameter disambiguation and upstream's rewrite to a plain quoted literal),E''escape strings, nested block comments, the PostgreSQL operator+-trimming rule, and the parser behavior's tokenize errors.internal/matcher/— matcher tree compiled from the grammar (matcher_factory.cpp), packrat memoization over upstream's designated rule list (parser_packrat.cpp), the transcribed rule-override table (identifier variants, literals,EndOfInput),ParseResulttree with upstream-style dumping, and furthest-failuresyntax error at or near "X"reporting. Upstream quirks are ported and pinned by tests: negative lookahead drops only the!operator (its operand still consumes), and/binds only the immediately preceding element.cmd/debug-parse— dump tokens or the rawParseResulttree for SQL on the command line.The 19-unmatched-parens pathological case from upstream's parser blog post runs in ~1 ms as a benchmark (packrat working as advertised).
Milestone 2 — corpus + accept/reject green
The pin advanced to
8cbdaba6acb50db7780d22a65dd131584d472262, the commit of the DuckDB nightly CLIv2.0.0-alpha38195used as the oracle — grammar, corpus expectations, and oracle all come from the same commit.internal/sqltest/— sqllogictest.testreader (extraction only; records end at empty-or-comment lines exactly as upstream'sSQLLogicParserdoes;${...}template substitutions are skipped, never expanded).internal/testfile/— the consolidated corpus format (cases separated by==, SQL/expectation by--) plus*.metadata.jsontodo/skip sidecars keyed by case content hash so entries survive regeneration.internal/duckdbsrc/— locate/verify the pinned CLI and run statements against:memory:with extension autoloading disabled;Parser Error⇒ must-reject, success or any post-parse error ⇒ must-accept.cmd/regenerate— rebuildparser/testdata/from a DuckDB source tree + the pinned binary (the only way expectations change);cmd/next-test— print the next todo case.parser/parser_test.go— the corpus harness and gate, with-check-parsecase dumps and-record-todostriage (auto-records only transformer-raised rejects; engine disagreements always fail).Corpus results
.testfiles scannedEvery accept/reject decision that depends on the grammar + matcher agrees with the pinned binary; the 290 todos are cases upstream rejects inside its transformer (positional-reference bounds, chained-comparison bans, VALUES arity, recursive-CTE modifier checks, …), which the matcher alone cannot see.
go build,go vet,gofmt, andgo test -race ./...are clean; no binaries are committed (the oracle CLI is located viaDARKWING_DUCKDB).🤖 Generated with Claude Code
https://claude.ai/code/session_01CTQBC9AzfdFPFsZtwn1mHY
Generated by Claude Code