Adherence to Agent Friendly Docs Specification - #3576
Open
spmallette wants to merge 12 commits into
Open
Conversation
…pec.com) TinkerPop's docs published only single, multi-MB HTML pages with no Markdown and no llms.txt, so AI coding agents truncated on the reference guide or fell back to stale training data. This adds a Markdown mirror of the docs — split into agent-sized (<=50KB) cross-linked pages — plus an llms.txt discovery index, rendered alongside the existing HTML from a single Gremlin execution. Architecture (the "neutral tree"): the executed Gremlin examples are captured in a format-neutral tab model and rendered to both backends, so HTML/Markdown parity of the executed ==> output is guaranteed by construction. HTML output is unchanged except for one added agent-discovery directive. Extension (docs/tinkeradoc-extension): - GremlinHighlighter: CodeRay highlighting + conum injection extracted from the treeprocessor into an HTML-only rendering seam. - NeutralTab / NeutralTabCodec: format-neutral tab model (label/language/kind/ plain text) with a dependency-free JSON codec. - HtmlTabRenderer / MarkdownTabRenderer: per-backend renderers over the neutral tabs; HTML reproduces the existing radio-tab widget byte-for-byte. - GremlinExecutionCache: keyed store so live Gremlin runs once across the two per-backend render passes (AsciidoctorJ re-runs the treeprocessor per convert()). - MarkdownConverter (tpmarkdown backend) + GremlinDocsConverterRegistry (SPI): renders headings, prose, lists, admonitions, tables, images, and inline formatting to GitHub-flavored Markdown; emits explicit <a id> anchors so xrefs resolve; substitutes the x.y.z version (postprocessor output is discarded for custom StringConverter backends); resolves images against imagesdir; supports curated [llms-summary] section descriptions via a hidden comment that never renders into HTML/Markdown body. - MarkdownSplitter: size-driven, heading-aligned page splitting with cross-page xref rewriting and per-page llms.txt pointer; UTF-8 byte budgeting. - LlmsTxtGenerator: llms.txt index (H1 + summary + per-book sections), curated descriptions preferred over a first-prose fallback, absolute-URL links. - GremlinTreeprocessor / GremlinPostprocessor: emit neutral tabs, render by backend, cache-aware execution; CodeRay cleanup made HTML-only, dead callout patterns removed. Build & publish: - pom.xml: a tpmarkdown <execution> per book mirroring each html5 book; raised docs-build heap. - bin/process-docs.sh: portable server-readiness probe (nc-optional), splits Markdown and generates llms.txt after render. - bin/publish-docs.sh: stages the .md tree beside the HTML in SVN and writes a site-root llms.txt with absolute URLs. - docs/src/docinfo-footer.html: agent-discovery directive in the HTML DOM. - bin/validate-llms-txt.sh: run `npx afdocs check` against the published site or a local build. Tests: 171 extension tests covering the neutral model, both renderers, the codec, execution cache, splitter, llms.txt generator, and dual-backend parity. Assisted-by: Claude Code:claude-opus-4-8
The llms.txt index previously derived each page's description by auto-extracting its first prose line, which produced truncated, mid-sentence, or low-context blurbs. This adds author-curated [llms-summary="..."] attributes to the ~71 sections that currently lead a split Markdown page, so llms.txt shows clean, specific one-line descriptions. The attribute is read only for llms.txt (emitted as a hidden Markdown comment) and never renders into the HTML or Markdown page body. Sections without a curated summary continue to fall back to the first-prose heuristic. Coverage: 70 of 71 page-leading sections annotated across reference, dev/io, dev/provider, dev/developer, dev/future, recipes, upgrade, and tutorials. The sole exception (tutorials/gremlin-language-variants) has no heading to attach to and falls back cleanly. Assisted-by: Claude Code:claude-opus-4-8
The size-driven splitter packed the ~117-step traversal reference into 7 arbitrarily-named grab-bag pages (combine-step.md held 26 steps, etc.), so an agent scanning llms.txt for a step like fold() found no entry and no way to guess which file held it. The page names and llms.txt descriptions effectively lied about their contents. Add an [llms-explode] section attribute that marks a catalog section whose direct subsections should each become their own page. The Markdown converter emits it as a hidden <!-- llms-explode --> marker (never rendered into HTML or the page body); MarkdownSplitter gives each direct child of a marked section its own anchor-named page regardless of size. Applied to "Graph Traversal Steps", this yields one page per step (fold-step.md, group-step.md, ...), each with its own llms.txt entry. Also add a curated [llms-summary] to every step in the catalog (first-sentence extraction, with ~18 hand-written where the opening prose was not self-describing) so each step's llms.txt line is a clean one-liner. Result: reference grows from 24 to 140 pages, llms.txt from ~71 to 187 entries (30.8KB, under the 50KB cap), every page under budget, 408 cross-page links resolve, no marker leakage into HTML. 172 extension tests pass. Assisted-by: Claude Code:claude-opus-4-8
Two more splitting refinements driven by content shape: - Add an [llms-keep] section attribute (hidden <!-- llms-keep --> marker) that keeps a section's whole subtree on one page even when it exceeds the size budget, and never descends into it. Applied to each GraphSON version (1.0, 2.0, 3.0) so a version is a single coherent file rather than being fragmented mid-version into arbitrarily-named tail pages (_requestmessage_2, _extended_2 are gone; their content now sits with its version). Versions 2.0/3.0 are intentionally over the 50KB budget as a result. - Mark the provider "Steps" (Gremlin semantics) catalog with [llms-explode] so each step's formal semantics becomes its own page, matching the reference step catalog. 176 extension tests pass; 408 cross-page links resolve; the only pages over 50KB are the deliberately-kept GraphSON versions. Assisted-by: Claude Code:claude-opus-4-8
Replace the size-driven splitter (and its llms-explode / llms-keep overrides) with a single rule: a section becomes its own Markdown page if and only if it carries an llms-summary. Unsummarized sections attach to their nearest summarized ancestor's page. This makes page boundaries deterministic and author-controlled, keeps every page described in llms.txt by construction, and deletes the size-packing / descend / explode / keep machinery. Size is now an author concern surfaced by a lint rather than enforced by splitting: MarkdownSplitter reports any page over the 50KB budget, and bin/process-docs.sh runs it with --strict so the docs build fails on an over-budget page. A section may opt out with allow-oversize="true" (emitted as a hidden marker) when it is intentionally kept whole. Curation to make the first summary-driven build clean: - explode/keep attributes converted: traversal + provider step catalogs now split per-step purely because each step has a summary; GraphSON versions stay whole because their per-type children have none; oversized-on-purpose pages (GraphSON 2.0, the 3.2.x release notes, the committer guide) get allow-oversize="true". - add llms-summary to the 31 provider step-semantics sections, the 15 traversal recipes, and the reference Introduction and Gremlin Compilers sections. Result: 246 pages, all described in llms.txt (40KB); the only over-budget pages are the three explicitly flagged allow-oversize; 412 cross-page links resolve; 173 extension tests pass. Assisted-by: Claude Code:claude-opus-4-8
The summary-driven split produced wrong llms.txt entries for every book's landing page: the document title broke off to its own page, leaving index.md owning the first chapter, so the index line showed that chapter's title and a scraped description (an image macro, the version line, or a mid-sentence fragment). - Splitter: fold the leading document-title section into the index page so its own llms-summary describes index.md and its content renders there, instead of breaking off to a separate _<doctitle>.md page. - Converter: for article-doctype pages (the tutorials), the title-bearing section is promoted to the document title, so read its llms-summary / allow-oversize from the promoted leading section (not just the Document) and emit the hidden markers on the landing page. - Curation: give the headingless gremlin-language-variants tutorial a titled summary, and replace four recipe summaries that had fallen back to generic or mid-sentence text (collections, shortest-path, edge-move, if-then-based-grouping) with proper descriptions. Result: all 240 pages, including every book/tutorial landing page, have a clean curated llms.txt description; build passes; 412 links resolve; 174 tests pass. Assisted-by: Claude Code:claude-opus-4-8
Add an Agent-Friendly Documentation subsection to the developer docs covering the Markdown mirror, llms.txt indexes, the llms-summary page model, the size budget with allow-oversize, and validation via bin/validate-llms-txt.sh. Note the required 'mvn clean install -f docs/tinkeradoc-extension/pom.xml' step, which a root build does not cover and whose absence silently skips the Markdown split. Extend the tinkeradoc README to cover MarkdownSplitter/LlmsTxtGenerator, and point doc authors to the llms-summary rules from the contributing guide. Assisted-by: Claude Code:claude-opus-4-8
When a summarized section breaks off to its own .md page, render a table-of-contents entry in its place on the parent page -- the child's heading (at its original level), its llms-summary as prose, and a link to the page -- instead of silently dropping the content. The parent stayed readable only for the sections that happened not to break off, so a book like reference/index.md jumped from its intro straight to the conclusion. The reference cascades at every level (index -> chapter -> subsection), so the whole book is followable page to page. Assisted-by: Claude Code:claude-opus-4-8
Both summaries had the Python variant's text (gremlinpython / Pythonic) pasted in. Replace with accurate descriptions of each section. Assisted-by: Claude Code:claude-opus-4-8
The footer discovery directive linked to the site-root /llms.txt (the last-published version) on every page, so an older-version page pointed agents at the wrong index. Use an absolute docs/x.y.z/llms.txt URL, which GremlinPostprocessor rewrites to the concrete version, so each page points to its own version's index. Also note the Markdown availability for humans. Assisted-by: Claude Code:claude-opus-4-8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Modified doc generation to provide support for https://agentdocsspec.com/ to allow agents to better navigate our documentation. It essentially adds a markdown output target in addition to HTML. The markdown output breaks the docs into smaller specific blocks and includes an index-like
llms.txtfile. Tested bin/publish-docs.sh - works:https://tinkerpop.apache.org/docs/3.7.7-SNAPSHOT/llms.txt
VOTE +1