Add search_id, explain hints, and search relevance telemetry (ENT-1528)

[evisdren]

Summary

Agent-driven search (~80% of expected search traffic) has no web-style click signal, so we can't tell whether the results we return are relevant. This closes the loop statelessly — no local correlation files (ENT-1528, full plan + updates in the ticket comments).

• search_id in every search JSON envelope (--json and --compact): a client-minted ULID — the server response carries no request id and the CLI merges per-cell responses, so it can't come from the server. Also the future seam for server-side impression logging in query-serve.
• Per-hit explain hint in --compact output: "explain": "entire checkpoint explain <id> [--repo <org>/<name>] --search-id <search_id>:<rank>". The --repo form covers cross-repo checkpoint hits (rides feat(explain): cross-repo checkpoint explain via the entire-api cell (ENT-1523) #1942); repo/pr/session rows get no hint since explain can't resolve them. The trailing token is the deterministic link: agents copy the hint verbatim, like a web SERP link carrying its tracking params.
• cli_search_performed event: search_id, query_hash (truncated SHA-256 of the normalized query — raw text never sent), mode, result_count, total, page, limit.
• cli_checkpoint_explained event: opaque checkpoint id + which resolution path fired (prose / export / cross_repo), hooked once at each of explain's three id-resolution success points. When the invocation carried --search-id (hidden flag, threaded via context — same pattern as withCrossRepoRead), the event also carries search_id + rank.

Analysis unlocked in PostHog: exact search→explain funnel joined on search_id, CTR-by-rank, reformulation/abandonment chains, and hint compliance (share of explain events carrying search_id; token-less explains fall back to time-proximity correlation).

Privacy

No query text, prompts, file paths, repo names, or session ids in any event; nothing written locally. search_id/rank are opaque CLI-minted values, and the --search-id flag value is strictly validated (26-char ULID, positive rank) with malformed tokens silently dropped — attribution can never fail an explain or leak free-form flag text into telemetry. Both events follow the existing Build*Payload/Track* pattern through the detached __send_analytics spawn and are double-gated: settings.Telemetry == true at the call site + ENTIRE_TELEMETRY_OPTOUT inside Track*. Documented in docs/security-and-privacy.md.

Review feedback addressed

• Trail Review: moved the export-path emit out of resolveExplainCheckpointID (shared with entire checkpoint tokens) into the explain-only callers, so token lookups don't pollute the funnel.

Test plan

• mise run fmt && mise run lint — clean
• mise run test:ci (unit + integration + Vogon/roger-roger E2E canary) — all passed, rerun after every change
• New tests: payload shapes for both events (raw query/impressions/session_id asserted absent; search_id/rank present only with attribution), query-hash normalization, ULID search_id, search_id in both envelopes, same-repo vs cross-repo hints with rank tokens, absolute rank across pages, no hint on repo/pr rows, --search-id token validation (junk/short/bad-rank cases), context round-trip

🤖 Generated with Claude Code

---

Design update (2026-08-16) — v2 amendments incoming

Following the trail 1019 review findings and an adversarial pass on the wider feedback-loop design (see ENT-1528 comments), this PR will be amended. The mechanism (client-minted search_id, stateless hint tokens, search→explain funnel) is unchanged; the following correct privacy and soundness defects:

Privacy (trail high finding):

• cli_checkpoint_explained will carry doc_ref — a truncated SHA-256 of the checkpoint id — instead of the raw checkpoint_id. A checkpoint id resolves via our own API to repo/session/author/prompt, contradicting the security doc's "no repository identifiers" guarantee. The hash is deterministic so server-side analysis can still join it against candidate ids from search_event records (the search service is gaining that record in a sibling PR); nothing in PostHog alone can resolve it to content.
• query_hash is removed entirely (not HMAC'd): a truncated unsalted SHA-256 of low-entropy natural language is dictionary-confirmable, and no v1 analysis consumes it — reformulation chains group by machine + time window; query-content analysis lives server-side. The event keeps query_length only.

Soundness (trail medium/low findings):

• search_id is minted before the request and sent as X-Entire-Search-Id (plus X-Entire-Client: <surface>/<version>), making it the real server-side seam — each cell will log it in its search_event record.
• --search-id is un-hidden so agent-help advertises it (hidden flags are stripped by agents that validate commands against agent-help, which would silently zero hint compliance).
• cli_checkpoint_explained fires after a successful render, not at id resolution — failures no longer count as clicks.
• The event emits normalized page/limit (matching the envelope and hint ranks) and both fetched_count and the server's true total.
• cli_search_performed fires on the error path too (outcome: error_*) and gains zero_results, reranked, total_ms, all_repos, and filter-presence booleans — all already on the wire, previously discarded.
• Explain hints are added to plain --json output as well (a large share of agent traffic doesn't use --compact; those explains could otherwise only ever join by time proximity).

Tests (trail finding): a hint→explain round-trip contract test (generated hint parsed by the real command's flag set) and settings-gate tests for both emit helpers, including the nil-settings case.

Analysis-layer rules that go with this (documented in the design): funnel joins enforce a ≤1h validity window on the ULID timestamp (stale hints replayed from transcripts must not count), and server-minted-id records are excluded from funnel denominators.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6393e08. Configure here.}

[cursor]

False explain telemetry from tokens

Medium Severity

emitCheckpointExplained is hooked inside resolveExplainCheckpointID, which entire checkpoint tokens also calls. A successful tokens lookup therefore emits cli_checkpoint_explained with source export, inflating the search→explain funnel with non-explain traffic (and twice when --compare resolves two IDs).

 

^{Reviewed by Cursor Bugbot for commit 6393e08. Configure here.}

[cursor]

Export commit path skips telemetry

Medium Severity

In the export resolver, emitCheckpointExplained runs only on checkpoint-prefix matches. Successful --commit resolution and the positional commit-ref fallback return without emitting, so JSON/transcript/raw-transcript explains resolved via commit never produce cli_checkpoint_explained.

Additional Locations (1)

• cmd/entire/cli/explain_export.go#L129-L133

 

^{Reviewed by Cursor Bugbot for commit 6393e08. Configure here.}

[Copilot]

Pull request overview

This PR adds end-to-end correlation for the agent-driven search → explain funnel by minting a client-side search_id, enriching compact search hits with copy/pasteable explain hints (including cross-repo --repo when needed), and emitting two privacy-preserving telemetry events to measure result relevance without storing any local correlation state.

Changes:

• Add search_id to search JSON envelopes (--json and --compact) and return served-count from JSON writers for telemetry.
• Add per-hit explain hints to --compact results (including cross-repo --repo owner/name for checkpoint hits).
• Add new telemetry events cli_search_performed and cli_checkpoint_explained, with corresponding tests and documentation updates.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/security-and-privacy.md	Documents the two new relevance telemetry events and their privacy constraints.
cmd/entire/cli/telemetry/search.go	Implements payload building + detached tracking for cli_search_performed and cli_checkpoint_explained.
cmd/entire/cli/telemetry/search_test.go	Adds payload-shape/privacy tests for the new telemetry events.
cmd/entire/cli/search_telemetry.go	Adds search_id minting, query hashing, and CLI-side gated emit helpers.
cmd/entire/cli/search_telemetry_test.go	Tests query hash normalization/truncation and ULID-shaped search_id.
cmd/entire/cli/search_cmd.go	Threads search_id into JSON envelopes, adds compact per-hit explain hints, and emits search telemetry.
cmd/entire/cli/search_cmd_test.go	Expands JSON writer tests for search_id, served-count, and explain hints (same-repo/cross-repo/no-hint types).
cmd/entire/cli/explain.go	Emits cli_checkpoint_explained on successful prose-mode checkpoint ID resolution.
cmd/entire/cli/explain_repo.go	Emits cli_checkpoint_explained on successful cross-repo explain resolution.
cmd/entire/cli/explain_export.go	Emits cli_checkpoint_explained on successful export-mode checkpoint ID resolution.

Suppressed comments (1)

cmd/entire/cli/search_cmd.go:1136

• Same issue as the full JSON envelope: search_id is tagged with omitempty, so the key can disappear if the ID can't be minted. If agents depend on the field existing, prefer always emitting it and letting the value be empty only in the extreme failure case.

	out := struct {
		Results    []compactSearchHit `json:"results"`
		Total      int                `json:"total"`
		Page       int                `json:"page"`
		TotalPages int                `json:"total_pages"`
		Limit      int                `json:"limit"`
		SearchID   string             `json:"search_id,omitempty"`
		Counts     *search.TypeCounts `json:"counts,omitempty"`
	}{

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.