Skip to content

Generated llms.txt for AI agents - #226

Draft
Zethson wants to merge 4 commits into
mainfrom
llms-txt
Draft

Generated llms.txt for AI agents#226
Zethson wants to merge 4 commits into
mainfrom
llms-txt

Conversation

@Zethson

@Zethson Zethson commented Jul 7, 2026

Copy link
Copy Markdown
Member

Fixes #216. Adds a generated /llms.txt giving AI agents a curated map of scverse — core packages come from the ecosystem registry, so they stay in sync, while the ## More packages list of other scverse-org repos is hand-maintained.

Add a Hugo LLMS output format that renders /llms.txt from the homepage
package front matter, plus curated links to other major scverse packages
and project resources. Also fix mudata's dead docs URL.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@netlify

netlify Bot commented Jul 7, 2026

Copy link
Copy Markdown

Deploy Preview for jade-cajeta-1bcca0 ready!

Name Link
🔨 Latest commit ef6930b
🔍 Latest deploy log https://app.netlify.com/projects/jade-cajeta-1bcca0/deploys/6a4d34d598f6da0008210f07
😎 Deploy Preview https://deploy-preview-226--jade-cajeta-1bcca0.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Comment thread layouts/index.llms.txt

@PauBadiaM PauBadiaM Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the core package list depends on code (I guess javascript of similar) execution to get rendered in the txt file. Do we know if an agent will see the rendered list at runtime, or just the:

Foundational packages maintained by the scverse core team.
{{ range .Params.packages }}
- [{{ .name }}]({{ .url }}): {{ .description }}
{{- end }}

This is important because if it does not render properly the agent will be quite lost.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I guess agents that look at the code will see this. Only if they access the rendered website, they'll see what we expect. So hardcoding this would be the right thing to do? Or will agents simply attempt to look at these parameters to figure out what should be there? Sure - less efficient but then we don't need to hardcode things.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I've seen that it depends a lot on the agent capability. High-end models (such as opus or fable) they will go through the trouble of jumping to other pages and following links if they are provided, while lower-end models (haiku, sonnet) they'll "speed-run" and not bother visiting other pages, so I would say it is safer to hardcode this just in case.

@grst

grst commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Please see also #200 where the plan was to replace the manual list of core packages with information from the package registry as the single point of truth.

For agents, we could just point them at this json which contains all core and ecosystem packages: https://scverse.org/ecosystem-packages/packages.json

@Zethson
Zethson marked this pull request as draft August 6, 2026 09:15
@Zethson Zethson changed the title feat: generated llms.txt for AI agents Generated llms.txt for AI agents Aug 14, 2026
Three things had gone stale in the month this sat open.

The core package loop read .Params.packages, which main deleted when it
moved core packages to the ecosystem registry; the section would have
rendered empty. It reads the registry now, the same way the packages page
does, so it covers all twelve rather than the eight that were listed by
hand.

The outputs config collided with site search: this branch set home to
HTML, RSS and LLMS, main had set it to html, rss and entities. Taking
either side alone drops an output — llms.txt or the search index. Both
are listed now.

The mudata documentation URL fix is already upstream in the registry, and
the front matter it patched is gone, so those hunks drop.
@netlify

netlify Bot commented Aug 25, 2026

Copy link
Copy Markdown

Deploy Preview for jade-cajeta-1bcca0 ready!

Name Link
🔨 Latest commit 1cef6d9
🔍 Latest deploy log https://app.netlify.com/projects/jade-cajeta-1bcca0/deploys/6a8d59e2e84700000848ef5b
😎 Deploy Preview https://deploy-preview-226--jade-cajeta-1bcca0.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

The section was hand-maintained with a note that new packages need a
manual line. It now comes from registry entries whose project_home is in
the scverse organisation, minus the core packages listed above it and
minus core-infrastructure, which is the org's own repos rather than
packages.

No package name is written by hand in the template any more. What is left
is site pages, the forums and the YouTube channel, none of which have a
registry to read.

This shrinks the section from eleven entries to two, because eight of the
eleven are absent from the ecosystem registry: gget, spatialdata-io,
napari-spatialdata, spatialdata-plot, annbatch, pytometry, rustar-aligner
and anndata-rs. They are missing from the package index and from site
search too, so the hand-written list was hiding that rather than fixing
it. Adding them to the registry restores them here automatically.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a llms.txt

3 participants