Feedback classifier leaks raw LLM completion (think tags, special tokens) into persisted feedback rationale

Summary

Feedback classification responses can leak the raw LLM completion — including closing <think> tags and DeepSeek special tokens — into the persisted feedback rationale. The existing thinking-disable only covers capture.summarize; every other op (e.g. feedback.classify) can receive thinking-enabled model output, and nothing strips think-tags from message.content before it is stored.

This is related to #2308 (the op parameter was dropped at the LLM boundary): even with op correctly propagated, thinking is only disabled for capture.summarize.

Reproduction / evidence

Two feedback rows stored on 2026-09-02 in our deployment contain a rationale that is clearly a raw model completion, not classified output:

rationale: "</think>\n<｜end▁of▁sentence｜>\n<｜end▁of▁session｜>\n\n---\n\n[Writing Rule]..."
polarity: negative

Code path (v2.0.17)

dist/core/feedback/llm-classifier.js → classifyByLlm() calls llm.complete(..., { op: "feedback.classify" })
dist/core/llm/providers/openai.js only disables thinking for one op:

if (opts.op === "capture.summarize") {
    body.thinking = { type: "disabled" };
}

For thinking-enabled models served through gateways that keep <think> blocks inside message.content, the raw completion (with </think>, <｜end▁of▁sentence｜>, …) flows back unmodified.
dist/core/experience/feedback-builder.js then persists feedback.rationale (and raw verifier text) verbatim into the store — polluted text becomes retrieval-visible memory.

Suggested fix

Either (ideally both):

Sanitize at the provider choke point: strip <think>…</think> blocks, orphan closing </think> fragments, and DeepSeek special tokens (<｜begin▁of▁sentence｜>, <｜end▁of▁sentence｜>, <｜end▁of▁session｜>) from choice.message.content before returning from the provider — one place, fixes all ops.
Extend thinking-disable to deterministic/analytical ops (feedback.classify, l3.abstraction, verifier, retrieval filter), not just capture.summarize.

Environment

memos-local plugin 2.0.17 (npm memtensor-memos-local-plugin-2.0.17)
Adapter: Hermes (adapters/hermes/memos_provider)
Windows 11, Node v24, LLM provider openai_compatible (deepseek-family model)

🤖 AutoDev has picked up this issue and started working on it.

Task ID: 372e4cf70b86ec48
Working branch: bugfix/autodev-2336-20260903031435996
Target branch: latest dev* branch
Workflow: opsp (analysis → coding → testing → PR)

I will post the PR link here once done. If I need more information, I will ask in the comments.

fix(plugin): sanitize <think> tags from LLM completions (MemTensor#2336)

@whipser030

✅ AutoDev task 372e4cf70b86ec48 completed.

Summary: Fixes issue #2336 — feedback classifier and other ops were leaking raw LLM completion tokens (<think>...</think> blocks and DeepSeek gateway tokens like <｜end▁of▁sentence｜>) into persisted FeedbackRow.rationale, which then polluted retrieval-visible memory on subsequent turns.

Root cause: apps/memos-local-plugin/core/llm/providers/openai.ts::complete returned choice.message.content verbatim, and no downstream sanitizer knew about <think> tags or DeepSeek special tokens. Thinking-enabled DeepSeek-family models served through gateways that keep the reasoning block inside content (rather than a separate reasoning_content field) leaked it end-to-end into storage. Adopts the reporter's preferred fix (option 1: sanitize at the provider choke point) — one transform covers all current and future ops (feedback.classify, feedback.refine, l3.abstraction, retrieval.filter, …) without requiring per-op thinking-disable coverage.

Changes: new core/llm/sanitize.ts module (strips matched <think>...</think> blocks, orphan closing </think>, orphan opening <think> + trailing text, DeepSeek special tokens, and collapses excess blank lines), applied to openai_compatible, anthropic, gemini, bedrock complete() return values. Streaming path is deliberately out of scope (chunk boundaries make safe chunk-level sanitization stateful — deferred; the reported leak is on the storage path which uses complete()).

Tests: added tests/unit/llm/sanitize.test.ts (17 unit tests including the exact artifact soup captured in the #2336 report, an ASCII-pipe safety guard, and idempotency) plus 2 provider-level regression tests in tests/unit/llm/providers.test.ts pinning end-to-end behavior. Verification: npm run test — 1576 passed / 2 skipped across 184 files; npm run lint (tsc --noEmit) — clean.

Base branch: main
Branch: bugfix/autodev-2336-20260903031435996
Commit: f334037
PR: #2339
Assigned to: @whipser030
Reviewers: @whipser030, @hijzy

fix(plugin): sanitize <think> tags in openai/gemini streaming path (M…

This issue has been automatically marked as stale due to inactivity.

This issue has been automatically closed due to inactivity.

Navigation Menu

Repository navigation

Feedback classifier leaks raw LLM completion (think tags, special tokens) into persisted feedback rationale #2336

Summary

Reproduction / evidence

Code path (v2.0.17)

Suggested fix

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feedback classifier leaks raw LLM completion (think tags, special tokens) into persisted feedback rationale #2336

Description

Summary

Reproduction / evidence

Code path (v2.0.17)

Suggested fix

Environment

Activity

Memtensor-AI commented on Sep 3, 2026

Memtensor-AI commented on Sep 3, 2026

github-actions commented on Oct 4, 2026

github-actions commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions