Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,27 @@
<!-- lore:019c8f4f-67c8-7cf4-b93b-c5ec46ed94b6 -->
* **Lore DB uses incremental auto\_vacuum to prevent free-page bloat**: Lore's SQLite DB uses incremental auto\_vacuum (schema version 3 migration) to prevent free-page bloat from deletions. The migration sets PRAGMA auto\_vacuum = INCREMENTAL then VACUUM outside a transaction. temporal\_messages is the primary storage consumer (~51MB); knowledge table is tiny.

<!-- lore:019d9ac2-0bdb-753a-832d-5c5445692349 -->
* **Lore npm publishing under @loreai scope**: The \`@lore\` npm scope was unavailable; \`@loreai\` was acquired instead. Publishing plan: mirror \`opencode-lore\` to \`@loreai/opencode\` (OpenCode plugin), publish \`@loreai/core\` (shared core), and \`@loreai/pi\` (Pi extension). The existing \`opencode-lore\` package name stays working for backwards compatibility — existing users' \`"plugin": \["opencode-lore"]\` config keeps functioning.

<!-- lore:019d15de-e2d6-7ff2-ab86-b78ca39688a7 -->
* **Lore search pipeline: FTS5 with AND-then-OR fallback and RRF fusion**: Lore's search pipeline (\`src/search.ts\`) uses FTS5 with AND-then-OR fallback and RRF fusion. \`ftsQuery()\` builds AND queries (primary), \`ftsQueryOr()\` builds OR fallback (only when AND returns zero results). Conservative stopword list excludes domain terms like 'handle', 'state', 'type'. FTS5 rank is negative (more negative = better). \`bm25()\` column weights: title=6, content=2, category=3. \`extractTopTerms()\` extracts top-40 frequency-ranked terms with stopword filtering. Recall tool uses \`reciprocalRankFusion\<T>(lists, k=60)\` across knowledge, temporal, and distillation sources into a single ranked list with source-type annotations. \`forSession()\` uses OR-based FTS5 BM25 scoring (not AND-then-OR) because it ranks all candidates — BM25 naturally weights multi-term matches higher. Safety net: top-5 project entries by confidence always included.

<!-- lore:019c8f8c-47c3-71a2-b5fd-248a2cfeba78 -->
* **Lore temporal pruning runs after distillation and curation on session.idle**: In src/index.ts, session.idle awaits backgroundDistill and backgroundCurate sequentially before running temporal.prune(). Ordering is critical: pruning must not delete unprocessed messages. Pruning defaults: 120-day retention, 1GB max storage (in .lore.json under pruning.retention and pruning.maxStorage). These generous defaults were chosen because the system was new — earlier proposals of 7d/200MB were based on insufficient data.

<!-- lore:019d92cd-67e4-75fa-bd8c-eb67a6b80871 -->
* **Lore's 5 background LLM calls are all single-turn prompt-response**: All Lore background LLM work (5 call types via \`promptWorker()\` + 1 auto-recovery) is single-turn: one user message in, one text response out, session rotated after each call. No multi-turn, no tool calling, no agentic loops. Call types: distillSegment (XML observations), metaDistill (XML consolidation), curator run (JSON ops), consolidate (JSON ops), expandQuery (JSON strings, 3s timeout). All parse text responses via regex/JSON.parse. This means decoupling from OpenCode SDK is straightforward — replace \`session.prompt()\` with direct provider API \`fetch()\` calls. Embedding calls (\`src/embedding.ts\`) already bypass the SDK with direct \`fetch()\` to Voyage/OpenAI APIs.

<!-- lore:019c94bd-042b-7215-b0a0-05719fcd39b2 -->
* **LTM injection pipeline: system transform → forSession → formatKnowledge → gradient deduction**: LTM injected via experimental.chat.system.transform hook. getLtmBudget() computes ceiling as (contextLimit - outputReserved - overhead) \* ltmFraction (default 10%, configurable 2-30%). forSession() loads project-specific entries unconditionally + cross-project entries scored by term overlap, greedy-packs into budget. formatKnowledge() renders as markdown. setLtmTokens() records consumption so gradient deducts it. Key: LTM goes into output.system (system prompt) — invisible to tryFit(), counts against overhead budget.

<!-- lore:019d15f7-4d00-781e-9512-a4f3e3109f18 -->
* **OpenCode plugin SDK has no embedding API — vector search blocked**: The OpenCode plugin SDK (\`@opencode-ai/plugin\`, \`@opencode-ai/sdk\`) exposes only session/chat/tool operations. There is no \`client.embed()\`, embeddings endpoint, or raw model inference API. The only LLM access is \`client.session.prompt()\` which creates full chat roundtrips through the agentic loop. This means Lore cannot do vector/embedding search without either: (1) OpenCode adding an embedding API, or (2) direct \`fetch()\` to provider APIs bypassing the SDK (fragile — requires key extraction from \`client.config.providers()\`). The FTS5 + RRF search infrastructure is designed to be additive — vector search would layer on top as another RRF input list, not replace BM25.

<!-- lore:019d9ac2-0bdf-711e-b563-9ca6c851604d -->
* **OpenCode repo moved to anomalyco/opencode with Node conditional imports**: OpenCode repo moved from \`sst/opencode\` to \`anomalyco/opencode\`. Still uses Bun as default runtime (\`packageManager: bun@1.3.11\`) but added Node support via conditional imports pattern in \`packages/opencode/package.json\`: \`"#db"\`, \`"#pty"\`, \`"#hono"\` each have \`bun\`/\`node\`/\`default\` variants. Uses Drizzle ORM over both \`bun:sqlite\` and \`node:sqlite\` (via \`drizzle-orm/bun-sqlite\` and \`drizzle-orm/node-sqlite\`). Dependencies include \`@hono/node-server\`, \`@hono/node-ws\`, \`@agentclientprotocol/sdk@0.16.1\`. This dual-runtime pattern validates Lore's planned conditional imports approach.

<!-- lore:019d8c54-e51c-7fe8-87ba-273269c39b7a -->
* **Worker session prompt helper with agent-not-found retry**: src/worker.ts owns workerSessionIDs Set, isWorkerSession(), and promptWorker(). promptWorker() calls session.prompt() and uses the return value directly (no redundant session.messages() call). On 'agent not found' errors (detected via regex on JSON.stringify(result.error)), it retries once without the agent parameter on a fresh session. All callers (distillation×2, curator×2, search×1) use this shared helper. Session rotation (deleting from the caller's Map) happens after every call. The retry creates a new child session via client.session.create() and registers its ID in workerSessionIDs.

Expand All @@ -33,7 +42,10 @@
* **Curator prompt scoped to code-relevant knowledge only**: CURATOR\_SYSTEM in src/prompt.ts now explicitly excludes: general ecosystem knowledge available online, business strategy and marketing positioning, product pricing models, third-party tool details not needed for development, and personal contact information. This was added after the curator extracted entries about OpenWork integration strategy (including an email address), Lore Cloud pricing tiers, and AGENTS.md ecosystem facts — none of which help an agent write code. The curatorUser() function also appends guidance to prefer updating existing entries over creating new ones for the same concept, reducing duplicate creation.

<!-- lore:019d15de-e2e4-777f-8e00-fe21198117ad -->
* **Lore plugin cannot use native Node addons — pure bun:sqlite only**: Lore is a Bun plugin library (\`main: 'src/index.ts'\`, Plugin type) running inside OpenCode's compiled Bun binary. It has no build step and cannot use native Node addons (no better-sqlite3, no node-llama-cpp, no sqlite-vec). Dependencies must be pure JS/TS or Bun built-ins. This rules out QMD as a library dependency (requires better-sqlite3 + node-llama-cpp + sqlite-vec). QMD's search patterns (BM25 + vector + RRF + reranking) are adapted for pure FTS5 instead. Vector/embedding search would need to use OpenCode's existing chat providers rather than local GGUF models.
* **Lore plugin cannot use native Node addons — pure bun:sqlite only**: OpenCode uses conditional imports (\`#db\` in package.json) to swap \`bun:sqlite\` ↔ \`node:sqlite\` at runtime — two 6-line files, zero build step, Drizzle ORM on top. Lore can use the same pattern for its core package: \`"#db": { "bun": "./src/db.bun.ts", "node": "./src/db.node.ts" }\`. The \`bun:sqlite\` and \`node:sqlite\` APIs are nearly identical (both inspired by better-sqlite3). Key differences: \`db.query()\` vs \`db.prepare()\`, \`Bun.CryptoHasher\` vs \`node:crypto\`. FTS5, transactions, pragmas all work identically. \`node:sqlite\` is stable without flags in Node 24+. Drizzle adoption is orthogonal — Lore's custom FTS5/BM25 queries wouldn't benefit from an ORM layer. No native addons needed in either path.

<!-- lore:019d92cd-67d5-705b-b60d-4537fcf4f054 -->
* **Lore standalone ACP server using Pi as agentic engine**: Lore is planned to become a standalone ACP (Agent Client Protocol) server, independent of OpenCode. Architecture: Lore speaks ACP to editors (Zed, JetBrains), uses Pi (\`@mariozechner/pi-coding-agent\`) internally as the agentic loop engine, and layers its memory system via Pi extensions. ACP proxy approach was rejected because proxies cannot modify the downstream agent's internal message array or system prompt — losing gradient context management and LTM injection, Lore's most valuable features. As a full ACP agent, Lore owns the LLM interaction with full control. Pi was chosen for its extension hooks (message injection, history filtering, custom compaction, custom tools) that map to Lore's existing OpenCode hooks. Requires a research spike first to verify Pi's extension API compatibility.

### Gotcha

Expand Down
Loading