diff --git a/docs/ai-trace-observability-design.md b/docs/ai-trace-observability-design.md
new file mode 100644
index 000000000..24a96e4e9
--- /dev/null
+++ b/docs/ai-trace-observability-design.md
@@ -0,0 +1,685 @@
+# AI 助手 Trace 观测加强设计
+
+> **实施状态**:Phase 1 已在本分支实现完成(工作区未提交)。
+>
+> | 交付物 | 状态 |
+> | -- | -- |
+> | `pkg/mcp/tools/trace/search.go` + `Provider.SearchTraces` + Jaeger v1 查询 API 实现 | 已实现 |
+> | `pkg/mcp/tools/trace/analyzer.go`(关键路径 / self time / 服务维度 / 错误归因 / 依赖边) | 已实现 |
+> | `pkg/mcp/tools/trace/genai.go`(`gen_ai.*` 识别与归类) | 已实现 |
+> | `pkg/mcp/register.go` 注册 `search_traces`、`analyze_trace` | 已实现 |
+> | `ai/component/hooks/tracing.go` 对齐语义约定 | 已实现(`gen_ai.agent.name`、span 改名、`error.type` 兜底、`gen_ai.client.operation.exception` 事件) |
+| `gen_ai.provider.name` 取值映射(T2) | 已实现(`gemini` → `gcp.gemini`,其余按合法自定义值) |
+| `capture_reasoning` 独立策略(T3) | 已实现(与 `capture_content` 正交,默认 `none`) |
+> | `server.address` / `server.port`、`gen_ai.response.model` | 未做,见 6.4 与 14 节待确认事项 |
+> | Phase 2(metric / trace / log 联动)、Phase 3(跨 trace 聚合、多 provider) | 未开始 |
+>
+> 下文 2.1 与 6.2、6.4 描述的是 **Phase 1 实施前** 的基线状态,用于说明改造动机与差距来源。
+
+## 1. 背景
+
+2026-10-24 周会([Discussion #1612](https://github.com/apache/dubbo-admin/discussions/1612))的 **AI assistant** 章节记录了如下待办:
+
+| task | owner | 等级 | 进度 | issue/discussion | pr |
+| -- | -- | -- | -- | -- | -- |
+| trace的观测加强 | luoyi | P0 | 待开始 | - | - |
+
+该条目属于 **AI assistant** 章节,而不是 Admin 控制台章节,因此交付物是 **Agent 可用的观测能力**,不是给人看的控制台页面。
+
+对照 2026 路线图 [Discussion #1386](https://github.com/apache/dubbo-admin/discussions/1386):
+
+- **AI / SRE Agent** 第 7 项「设计Hook系统,接入otel trace」——Agent 自身已通过 `ai/component/hooks/tracing.go` 接入 OTel,但缺少把 trace 数据**回喂给 Agent 自己**的能力。
+- **可观测能力** 第 3 项「基于Grafana,探索metric,trace,log的联动」——这是**给人**的 Grafana 联动,与本设计是互补关系。
+
+trace 之所以是 AI 助手当前最大的盲区,是因为 AI 侧的观测能力存在一个结构性缺口:
+
+| 数据类型 | Agent 侧已有工具 | 能否自主发现问题 |
+| -- | -- | -- |
+| 集群/实例/服务拓扑 | `get_cluster_info`、`search_services`、`get_service_instances` 等 | 能 |
+| 日志 | `search_logs`、`analyze_error_logs`、`get_log_capabilities` | 能 |
+| 指标 | `query_prometheus`、`query_prometheus_range` | 能 |
+| **链路** | **只有 `get_trace_by_id`** | **不能** |
+
+`get_trace_by_id`(`pkg/mcp/tools/trace/tools.go`)要求调用方提供 `traceId`,而 `traceId` 是人肉从日志、告警或 Grafana 里捞出来的。Agent 能自己查日志和指标,却无法自己「找出」一条失败的 trace。因此,只要用户的问题停留在「最近 shop-order 有没有失败的请求」「这个接口为什么变慢」这一层,Agent 就必须先把问题退回给用户索要 traceId,观测能力形同虚设。
+
+## 2. 现状
+
+### 2.1 现有 trace 能力
+
+`pkg/mcp/tools/trace/` 当前包含:
+
+| 文件 | 内容 |
+| -- | -- |
+| `provider.go` | `Provider` 接口,当前只有 `GetTraceByID(ctx, traceID) (*Trace, error)` 一个方法;`newProvider` 按 `TraceProviderConfig.Type` 分发,`jaeger` 是唯一支持的类型 |
+| `tools.go` | `GetTraceByID` handler;`traceIDPattern` 校验 16/32 位十六进制;`providerConfigFromContext` 解析 provider;`Properties()` 返回 MCP input schema |
+| `jaeger.go` | Jaeger v1 `/api/traces/{traceID}` 查询;响应归一化与有界裁剪 |
+| `model.go` | 后端无关的 `Trace` / `Span` / `Event` 类型(以及本次新增的搜索/分析契约类型) |
+| `jaeger_test.go` | 4 个单测:查询与归一化、TraceID 校验、not found、span 裁剪 |
+
+`pkg/mcp/register.go` 只注册了一个 trace 工具 `get_trace_by_id`。`pkg/config/observability/tracing.go` 定义了 `TracingConfig` / `TraceProviderConfig`,`TraceProviderConfig` 带有 `BearerToken`、`Tenant` 两个认证字段,`TracingConfig.Sanitize()` 会把 `BearerToken` 替换为 `config.SanitizedValue`。
+
+### 2.2 邻近工作流及其边界
+
+以下工作流**已经存在或正在被别人推进**,本设计不重复实现,只做衔接:
+
+| 编号 | 标题 | 层次 | 与本设计的关系 |
+| -- | -- | -- | -- |
+| [#1606](https://github.com/apache/dubbo-admin/issues/1606) / PR [#1607](https://github.com/apache/dubbo-admin/pull/1607)(开放中) | Correlate Jaeger traces with Dubbo RPC metrics | OpenTelemetry Collector OTTL transform + Grafana `tracesToMetrics` | **互补**。它让**人**在 Grafana 里能由 trace 跳到 metric(`dubbo.application` → `application_name`、`dubbo.metric.interface` → `interface`、`rpc.method` → `method`),但它明确不新增 Admin API、Vue 页面和 spanmetrics pipeline,**Agent 依然无法自己发现或分析 trace** |
+| [#1537](https://github.com/apache/dubbo-admin/issues/1537) / [#1524](https://github.com/apache/dubbo-admin/issues/1524) | 日志观测与 metric-trace-log 关联 | Grafana 面板 + 关联 | 归属他人。本设计只复用其**数据维度**,不新增 Grafana 面板 |
+| PR [#1499](https://github.com/apache/dubbo-admin/pull/1499) | PromQL 与 trace 诊断工具 | MCP 工具 | 已合并,提供 `query_prometheus`、`query_prometheus_range`、`get_trace_by_id`。本设计是它的延伸,**不替换** `get_trace_by_id` |
+| PR [#1536](https://github.com/apache/dubbo-admin/pull/1536) | agent hooks 与 OpenTelemetry tracing | Agent 侧 | 已合并,`ai/component/hooks/tracing.go` 已能产出 `gen_ai.*` 属性。本设计负责**对齐语义约定**并让 Agent 能读回自己的 trace |
+
+### 2.3 明确不属于本设计的部分
+
+以下内容属于上述其他 issue 的范围,本设计**不做**:
+
+- 不新增 Admin HTTP API(`pkg/console/` 下无任何新增路由)。
+- 不新增或修改 Vue 页面(`ui-vue3/`)。
+- 不新增或修改 OpenTelemetry Collector pipeline(OTTL transform、spanmetrics connector 等)。
+- 不修改 Grafana datasource / dashboard 配置。
+- 不替换或下线已有的 `get_trace_by_id`。
+
+本设计的唯一产出面是:**`pkg/mcp/tools/trace/` 下的新 MCP 工具** + **`ai/component/hooks/tracing.go` 的属性对齐**。
+
+## 3. 目标与非目标
+
+### 3.1 目标
+
+- 让 Agent 能**自主发现** trace:从「服务 + 时间窗 + 条件」出发搜出候选 trace,而不需要人工提供 traceId。
+- 让 Agent 能**结构化诊断**一条 trace:关键路径、self time、服务耗时占比、错误归因、依赖边。
+- 识别 trace 内部的 **GenAI span**(模型调用、工具调用、Agent span),让 Agent 能回答「这次 LLM 调用花了多少 token、失败在哪一步」。
+- 全流程**有界**:沿用 `jaeger.go` 已有的裁剪语义,响应体积、span 数量、属性数量都有上限,超限时显式标记 `truncated`。
+- 让 `search_traces` → `analyze_trace` → `search_logs` / `query_prometheus` 形成一条 Agent 可自主执行的调查链路。
+
+### 3.2 非目标
+
+- 不做实时流式 trace 订阅(tail / watch)。
+- 不做跨 trace 的聚合统计(错误率趋势、慢调用分布),这属于 Phase 3。
+- 不做 trace 写入 / 采样策略配置。
+- 不在 MCP 工具输出中回显 prompt / completion 正文、tool 参数体。
+- 不做跨 Provider 的统一搜索语义(当前只有 Jaeger)。
+
+## 4. 方案总览
+
+```mermaid
+flowchart TB
+ subgraph AI["Dubbo Admin AI Agent - ai/"]
+ REACT["ReActAgent
决策与工具编排"]
+ HOOK["hooks/tracing.go
GenAI 语义约定对齐"]
+ OTLP["OTLP Exporter
gRPC / HTTP"]
+ end
+
+ subgraph MCP["MCP Server (pkg/mcp)"]
+ ST["search_traces
发现"]
+ AT["analyze_trace
诊断"]
+ GT["get_trace_by_id
原始明细"]
+ SL["search_logs
Loki"]
+ QP["query_prometheus
Prometheus"]
+ end
+
+ subgraph PROV["Provider 抽象 pkg/mcp/tools/trace/provider.go"]
+ JA["jaegerProvider
Jaeger v1 Query API"]
+ end
+
+ subgraph BACK["后端"]
+ JG["Jaeger / Trace Backend"]
+ LK["Loki"]
+ PM["Prometheus"]
+ end
+
+ REACT --> ST
+ REACT --> AT
+ REACT --> GT
+ REACT --> SL
+ REACT --> QP
+ ST --> PROV
+ AT --> PROV
+ GT --> PROV
+ PROV --> JA
+ JA --> JG
+ SL --> LK
+ QP --> PM
+ HOOK --> OTLP
+ OTLP --> JG
+```
+
+三条并行的观测面:
+
+1. **发现面**:`search_traces` 在 Provider 侧检索,把「一段时间内某服务的失败 trace」变成 Agent 可直接消费的结构化列表。
+2. **诊断面**:`analyze_trace` 拉取单条 trace 的 span 明细,在本地做关键路径与错误归因计算,返回结论而不是原始 span 列表。
+3. **旁路面**:`search_logs`(可按 `traceId` 过滤)与 `query_prometheus` 用于把诊断结论落到具体日志行和指标上,这条链路已由 PR #1499 与 #1537 铺好。
+
+Agent 自身的 `ai/component/hooks/tracing.go` 通过 OTLP 导出到同一个 Trace Backend,因此 Admin AI 自己的交互也会出现在 `search_traces` 的结果里(`traceKind == "genai"`),这使得「AI 刚才这次对话慢在哪」成为可回答的问题。
+
+## 5. 关键设计
+
+### 5.1 先发现,后分析
+
+这是整个设计的核心判断:**只提供 `analyze_trace` 而不提供 `search_traces` 是无效的**。
+
+一个只持有 traceId 的 Agent 无法开始任何调查——它无法回答「有哪些失败的 trace」「这类失败有多频繁」「是不是新出现的」。真实的问题形态是「最近 shop-order 是不是有问题」,Agent 必须先从「服务 + 时间窗」收敛到一批候选 trace,再逐条深入。
+
+因此两个工具的职责严格分离:
+
+| 工具 | 输入锚点 | 输出体量 | 用途 |
+| -- | -- | -- | -- |
+| `search_traces` | service(必需)+ 时间窗 + 过滤条件 | 每条 trace 一行摘要,列表有上限 | 缩小范围、发现异常、拿到 traceId |
+| `analyze_trace` | traceId(必需) | 单条 trace 的结构化诊断 | 定位原因、量化耗时、归因错误 |
+
+典型 Agent 调用序列:
+
+```mermaid
+sequenceDiagram
+ participant U as 用户
+ participant A as AI Agent
+ participant ST as search_traces
+ participant AT as analyze_trace
+ participant SL as search_logs
+ participant QP as query_prometheus
+
+ U->>A: 最近一小时 shop-order 有失败的请求吗
+ A->>ST: service=shop-order, startTime=..., endTime=..., limit=20
+ ST-->>A: 12 条候选,其中 3 条 hasError=true
+ A->>AT: traceId=<失败 trace>
+ AT-->>A: criticalPath + errorSpans(rootCause) + serviceBreakdown
+ A->>SL: traceId=<失败 trace>
+ SL-->>A: 异常日志行
+ A->>QP: query=histogram_quantile(... dubbo_application=...)
+ QP-->>A: p99 序列
+ A-->>U: 结论:3 条失败均由 payment-service 超时引起
+```
+
+### 5.2 有界响应与数据最小化
+
+`jaeger.go` 已经确立了一整套裁剪常量,这些语义在 `search` 与 `analysis` 中必须被继承,而不是另起一套:
+
+| 常量 | 值 | 作用 |
+| -- | -- | -- |
+| `traceQueryTimeout` | 15s | 单次 Provider 请求超时 |
+| `maxTraceResponseBytes` | 8 MiB | Provider 响应体上限 |
+| `maxTraceErrorBodyBytes` | 4096 | 上游错误体回显上限 |
+| `maxTraceSpans` | 500 | 单条 trace 参与分析的 span 上限 |
+| `maxTraceRoots` | 50 | 根 span 记录上限 |
+| `maxTraceServices` | 100 | 服务名列表上限 |
+| `maxSpanAttributes` | 32 | 单 span 属性数上限 |
+| `maxSpanEvents` | 10 | 单 span 事件数上限 |
+| `maxEventAttributes` | 16 | 单事件属性数上限 |
+| `maxAttributeValueBytes` | 512 | 单属性值字节上限(UTF-8 安全截断) |
+
+在此基础上新增两组上限:
+
+- **搜索侧**:`maxSearchLimit`(默认 20,上限 100)限制 `TraceSearchResult.Traces` 的长度,`maxSearchTraceSummaries`(100)限制本地保留的摘要数。Provider 侧按调用方条数再多取一条用于探测是否还有更多匹配,命中该上限即置 `Truncated = true`;否则一个刚好装满的结果会被误读为"已看全"。
+- **分析侧**:`maxCriticalPath`(默认 32)、`maxServiceBreakdown`(默认 50)、`maxSlowSpans`(默认 10)、`maxErrorSpans`(默认 20)、`maxDependencyEdges`(默认 50)、`maxGenAICalls`(默认 20)。超出部分丢弃,**只有在真正丢失分析数据时**才在 `TraceAnalysis.Truncated` 上置位:`maxCriticalPath` 截断的是仍在延续的链路,属于丢数据;`maxSlowSpans` 截断的是排名结果,不丢分析数据,因此不置位。
+
+数据最小化上有一条硬规则:**分析器只读取元数据属性,不把 `gen_ai.input.messages`、`gen_ai.output.messages`、`gen_ai.tool.call.arguments`、`gen_ai.tool.call.result` 复制进工具输出**。这些属性在 semconv 中是 `Opt-In` 级别,本身就可能包含 PII;分析器只消费 `gen_ai.request.model`、`gen_ai.usage.*` 这类聚合量。
+
+### 5.3 关键路径与 self time
+
+#### 5.3.1 span 树的构建
+
+输入是 `Trace.Spans`(已由 `normalizeJaegerTrace` 归一化)。构建过程:
+
+1. 建立 `map[spanID]*Span`,遍历 `Trace.Spans`。
+2. 父指针取 `Span.ParentSpanID`。`jaeger.go` 的 `parentSpanID(references)` 优先取 `refType == "CHILD_OF"` 的引用,没有则退回第一个引用;两者都为空即为根 span。因此**根 span = `ParentSpanID` 为空的 span**,与 `Trace.RootSpanIDs` 一致。
+3. `ParentSpanID` 指向的 span 不在集合内(被 `maxTraceSpans` 裁掉,或采样缺失)时,该 span 退化为**孤儿根**。孤儿根不参与父链计算,但其自身耗时仍计入 `ServiceBreakdown` 与 `SlowSpans`。
+4. 解析 `Span.StartTime`(RFC3339Nano)为 `time.Time`。解析失败的 span 退化为叶子节点并跳过 self time 计算,同时把 `TraceAnalysis.Truncated` 置位——时间基准不可信时继续计算会产出误导性的百分比。
+
+#### 5.3.2 self time 的计算
+
+定义:对 span `S`,令 `S` 的子集合为 `C(S)`,`S` 的时间区间为 `[S.start, S.end]`。
+
+```
+covered(S) = | ⋃_{c ∈ C(S)} [ max(c.start, S.start), min(c.end, S.end] ] |
+self(S) = S.duration - covered(S)
+wait(S) = covered(S)
+```
+
+`covered` 是**子 span 时间区间的并集长度**,而不是子 span duration 之和。这一点是必须的:并发调用(例如同时发起 3 个下游 RPC)的子 span 在时间上重叠,简单求和会让 `self` 变成负数、并让 `ServiceBreakdown.SharePercent` 严重失真。实现上采用区间合并:按 `start` 升序排序子区间,线性扫描合并重叠段,再累加合并后的总长度,复杂度 `O(n log n)`。
+
+边界处理:
+
+- 子 span 完全在父 span 之外(时钟漂移或采样错乱)→ 裁剪到父区间后长度归零,不计入 `covered`。
+- 子 span 区间为空或 duration 为 0 → 不计入。
+- `self` 计算结果小于 0(父区间被裁剪得更短)→ 置 0,并在该 span 上标记异常。
+
+#### 5.3.3 关键路径的选择
+
+关键路径定义为:**决定端到端时延的那条根→叶链路**。
+
+算法:
+
+1. 在所有根 span 中取 `end` 最大者作为关键路径起点(`end` 相同则取 duration 最大者)。若存在多个根,其余根的子树仍参与 `ServiceBreakdown`、`SlowSpans`、`ErrorSpans`、`Dependencies` 计算,但不进入 `criticalPath`。
+2. 从当前节点出发,在其子节点中选取 `end` 最大者(`end` 相同则取 duration 最大者)作为下一跳。理由:`end` 最大的子 span 是父 span 内**最后完成**的活动,它决定了父 span 的完成时刻。
+3. 重复直到到达叶子(无子节点)。整条链路即为 `TraceAnalysis.CriticalPath`。
+4. 每一跳记录 `CriticalPathEntry`:`spanId`、`serviceName`、`operationName`、`startTime`、`durationMicros`、`selfMicros`、`waitMicros`。
+
+这样得到的链路满足:链路上每一跳的 `end` 都不晚于其父节点的 `end`。在 span 列表未被裁剪时,链尾的 `end` 等于 trace 的墙钟终点;列表被 `maxTraceSpans` 裁剪时,关键路径只在保留下来的 span 上计算,链尾可能早于 provider 给出的墙钟终点(此时 `durationMicros` 取 provider 的完整包络,见 §5.2)。
+
+链路上的 `selfMicros` 之和是**这条关键路径上**的不可压缩耗时,不是整条 trace 的不可压缩耗时:不在关键路径上的并行分支同样有 self time,不计入这个和。它是 trace 实际工作量的下界,不是上界。
+
+`SlowSpan` 则按 **self time 降序**取前 N(默认 10),并用 `onCriticalPath` 标记该 span 是否落在关键路径上。self time 排名与 critical path 互补:前者回答「谁自己最慢」,后者回答「请求是怎么被拖长的」。
+
+> 注:`ServiceLatency.SharePercent` 的语义是「该服务的 span duration 之和占 trace 墙钟时长的比例」,分母是墙钟、分子是各 span 时长之和,因此在并发场景下可能超过 100%。这是刻意的「工作量占比」语义而非「时间占比」,调用方需要按此理解,不能把它当成互斥的时间切片。
+
+### 5.4 错误归因与根因检测
+
+**错误 span 判定**:`Span.Error == true` 或 `Span.Status == "ERROR"`。该状态来自 `jaeger.go` 的 `spanStatus(tags)`,它识别三类 tag:`error=true|1`、`otel.status_code=error`、`status.code=error`;`status.code=ok` 映射为 `Status == "OK"`。
+
+**error.type 分类**(按优先级取第一个可用的):
+
+| 优先级 | 来源 | 说明 |
+| -- | -- | -- |
+| 1 | span 属性 `error.type` | OTel 稳定属性,semconv 中为 `Conditionally Required`,值域如 `timeout`、`_OTHER` |
+| 2 | 事件 `gen_ai.client.operation.exception` 的属性 `exception.type` | semconv 的 GenAI 专用异常事件。本项目服务于 **Agent 循环的 GenAI 执行**,故排在传统异常事件之前 |
+| 3 | 事件 `exception` 的属性 `exception.type` | 标准 OTel 异常事件,面向传统 RPC 服务的异常面 |
+| 4 | span 属性 `exception.type` | 采集器把异常直接落在 span 属性/标签上时的兜底,事件仍是主来源 |
+| 5 | span 属性 `error.message` | **不是 OTel 标准属性**(error registry 只定义 `error.type`),仅在没有任何类型化来源时降级使用 |
+| 6 | 常量 `unknown` | 全部缺失 |
+
+`ErrorSpan.ErrorMessage` 按 `gen_ai.client.operation.exception` 事件 → 标准 `exception` 事件 → span 属性 `exception.message` → span 属性 `error.message` 的顺序取值,并受长度上限约束(错误类型 96 字符、错误描述 256 字符)。`error.message` 因为不是标准属性,排在最后。
+
+> 这里刻意区分了两类错误面:**GenAI 执行异常**(模型调用、工具调用失败,走 `gen_ai.client.operation.exception`)与**传统 API 服务异常**(Dubbo RPC 侧的普通异常,走标准 `exception` 事件)。本设计以前者为重点,但分析 Dubbo 链路时后者同样必须能读,故两者都保留、按 GenAI 优先排序。
+
+**token 口径**:`gen_ai.usage.total_tokens` **不是**规范属性,规范只定义了 `gen_ai.usage.input_tokens` 与 `gen_ai.usage.output_tokens`,「总量」是派生概念。因此 `GenAISummary.TotalTokens` **恒等于** `InputTokens + OutputTokens`;若某个非规范 provider 上报了 `gen_ai.usage.total_tokens`,它只作为单次调用明细 `GenAIModelCall.TotalTokens` 的参考值出现,**不参与汇总**(此时明细之和与汇总之和可能不等,属预期)。
+
+**根因检测**:在错误 span 构成的子树中,**没有任何后代 span 也失败**的 span 被标记 `RootCause = true`。反之,一个错误 span 如果其子树中还有失败 span,说明它只是**错误传播**的中间环节,不是根因。`ErrorSpans` 按「最深失败优先」排序,让 Agent 先看到最内层的失败。
+
+这套规则能覆盖 Dubbo 场景中最常见的两种形态:下游超时导致上游 RPC 报错(根因 = 下游 span),以及序列化失败(根因 = 唯一失败的 span)。
+
+### 5.5 服务依赖边
+
+依赖边直接由父子关系导出:对每个 span `c`(其父 `p` 存在于同一 trace 内),产生一条 `p.ServiceName → c.ServiceName` 的边。
+
+聚合规则:
+
+- 按 `(callerService, calleeService)` 去重,`spanCount` 为该对出现的次数。
+- `totalMicros` 为该对下所有子 span 的 duration 之和。
+- `errorCount` 为该对下 `Error == true` 的子 span 数量。
+- 按 `totalMicros` 降序排列。
+
+父子关系天然刻画了**同一条 trace 内的**调用拓扑,是 Dubbo 场景下最可靠的服务依赖信号。跨 trace 的全局拓扑不在本设计范围(属于 roadmap 的「服务拓扑/应用拓扑」条目)。
+
+## 6. GenAI 语义约定对齐
+
+### 6.1 规范来源与状态
+
+权威来源是 OpenTelemetry **semantic conventions for generative AI systems**,本设计对齐的是该仓库的只读 checkout(commit `6fd0d76`),文档位于 `docs/gen-ai/`:
+
+| 规范文件 | 内容 |
+| -- | -- |
+| `docs/gen-ai/README.md` | 索引 |
+| `docs/gen-ai/gen-ai-spans.md` | 模型类 span(chat / embeddings / retrieval / fetch_response / memory / execute_tool) |
+| `docs/gen-ai/gen-ai-agent-spans.md` | Agent 类 span(create_agent / invoke_agent / invoke_workflow / plan) |
+| `docs/gen-ai/client-inference.md` | 推理(chat)span 与事件 |
+| `docs/gen-ai/gen-ai-events.md` | GenAI 事件(`gen_ai.evaluation.result`) |
+| `docs/gen-ai/gen-ai-exceptions.md` | GenAI 异常事件 |
+| `docs/gen-ai/gen-ai-token-metrics.md` | 推理 token 指标 |
+
+**文档状态**:上述所有文件的标题下方均为 `**Status**: [Development][DocumentStatus]`,即**开发中、尚未稳定**。所有 `gen_ai.*` 属性在 registry 中的 Stability 均为 `Development`;例外是通用属性 `error.type`、`exception.message`、`exception.type`、`exception.stacktrace`、`server.address`、`server.port` 为 `Stable`。
+
+这是一份快速演进中的文档。本设计必须**钉住规范版本**(记录 commit `6fd0d76`),并在每次升级规范时重新核对属性名与枚举值,而不是把属性名写死为「永久正确」。
+
+### 6.2 属性对照表
+
+下表列出 `analyze_trace` 侧识别的属性,以及 Agent 侧 `ai/component/hooks/tracing.go` 需要发出的属性。
+
+| 属性 | 分析器用途 | 规范文件 | Agent 侧现状 | Agent 侧动作 |
+| -- | -- | -- | -- | -- |
+| `gen_ai.operation.name` | 判定 span 类别(模型调用 / 工具调用 / Agent) | `gen-ai-spans.md`、`gen-ai-agent-spans.md` | 已发:`invoke_agent`、`chat`、`execute_tool` | 保持 |
+| `gen_ai.provider.name` | 归类 provider(`openai`、`anthropic`、`aws.bedrock`、`azure.ai.openai`、`gcp.vertex_ai` …) | `gen-ai-spans.md` | 已发,但由 model 字符串按 `/` 切分推导 | **改为白名单映射**,见 6.4 |
+| `gen_ai.request.model` | 模型名(模型调用摘要) | `client-inference.md` | 已发(取 model 字符串 `/` 之后部分) | 保持 |
+| `gen_ai.response.model` | 实际响应模型 | `gen-ai-spans.md` | 未发 | 可选补充 |
+| `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` | token 聚合 | `gen-ai-spans.md`(span 属性,Recommended) | 已发(`EventModelCallEnd`) | 保持 |
+| `gen_ai.conversation.id` | 会话归组 | `gen-ai-agent-spans.md` | 已发(取 `SessionID`) | 保持 |
+| `gen_ai.agent.name` | Agent 名称 | `gen-ai-agent-spans.md`(invoke_agent 上 Conditionally Required) | **未发** | **Phase 1 必须补** |
+| `gen_ai.agent.id` | Agent 稳定标识 | `gen-ai-agent-spans.md` | 未发 | 有稳定标识时补 |
+| `gen_ai.tool.name` | 工具名 | `gen-ai-spans.md`(execute_tool 上 **Required**) | 已发 | 保持 |
+| `gen_ai.tool.call.id` | 工具调用标识 | `gen-ai-spans.md`(Recommended) | 已发 | 保持 |
+| `gen_ai.tool.call.arguments` / `gen_ai.tool.call.result` | 工具入参/结果 | `gen-ai-spans.md`(**Opt-In**) | 已发,受 capture policy 控制 | 保持,分析器**不读取** |
+| `gen_ai.input.messages` / `gen_ai.output.messages` | 提示词/回复 | `gen-ai-spans.md`(**Opt-In**) | 已发,受 capture policy 控制 | 保持,分析器**不读取** |
+| `error.type` | 错误分类 | `gen-ai-spans.md`(Conditionally Required) | 已发(`state.ErrorType` 非空时) | 保持 |
+| `exception.type` / `exception.message` | 异常类型与消息 | `gen-ai-exceptions.md` | 由 `span.RecordError` 产生的标准 `exception` 事件 | 见 6.4 |
+
+### 6.3 genai trace kind 的判定
+
+`TraceSummary.TraceKind` 的语义(见 `model.go`):**当 trace 中任一 span 携带 `gen_ai.*` 属性时,值为 `"genai"`,否则为空字符串**。
+
+实现上,`pkg/mcp/tools/trace/genai.go` 提供统一识别函数:
+
+- `hasGenAIAttributes(span)`:span 的 `Attributes` 中存在任意以 `gen_ai.` 为前缀的键。
+- 分类映射:
+ - `gen_ai.operation.name ∈ {chat, generate_content, text_completion, embeddings}` → **模型调用**(`GenAIModelCall`)。
+ - `gen_ai.operation.name == execute_tool` → **工具调用**(`GenAIToolCall`)。
+ - `gen_ai.operation.name ∈ {invoke_agent, create_agent, invoke_workflow, plan}` → **Agent span**(`GenAIAgent`)。
+
+之所以必须识别 LLM span,是因为在 Agent 场景里,**「失败」经常发生在模型调用或工具调用上,而不是 Dubbo RPC 上**。一条 `traceKind == "genai"` 的 trace 里,最有价值的诊断信息是:调了哪个模型、花了多少 token、哪个工具返回了错误、异常事件是什么。这些信息在通用的 RPC 视角下完全不可见。
+
+### 6.4 与当前 Agent hook 的不一致点
+
+`ai/component/hooks/tracing.go` 已由 PR #1536 合并,其属性与当前规范存在以下差距,本设计需要在 Phase 1 对齐:
+
+1. **`gen_ai.provider.name` 的取值不规范**。当前 `providerName(model)` 直接按 `/` 切分 model 字符串。规范枚举值是 `openai`、`anthropic`、`aws.bedrock`、`azure.ai.inference`、`azure.ai.openai`、`gcp.gemini`、`gcp.gen_ai`、`gcp.vertex_ai`、`deepseek`、`groq`、`mistral_ai`、`moonshot_ai`、`perplexity`、`x_ai`、`cohere`、`ibm.watsonx.ai` 等。切分得到的可能是 `azure` 而不是 `azure.ai.openai`。需要一张显式映射表,无法识别时按规范允许「自定义值」,但不应产生 `azure` 这种半截值。
+2. **缺少 `gen_ai.agent.name`**。规范在 `invoke_agent` span 上将其列为 Conditionally Required。缺了它,Admin 自己的 Agent span 在 trace 里就没有身份,分析器只能识别出「有一个 Agent span」,无法命名。当前 span 名 `invoke_agent` 正是规范中 `gen_ai.agent.name` 不可用时的合法回退形式,所以补上 `gen_ai.agent.name` 后 span 名应升级为 `invoke_agent {gen_ai.agent.name}`。
+3. **异常事件不是 GenAI 专用事件**。当前用 `span.RecordError(errors.New(state.Error))` 产生标准 `exception` 事件,其 `exception.type` 会是 `*errors.errorString` 这类包装类型,而 `gen-ai-exceptions.md` 明确指出「如果记录的异常类型是无意义的包装类型,应使用内层异常的类型」,并定义了 `gen_ai.client.operation.exception` 事件(severity SHOULD 为 WARN)。对齐后,模型调用与工具调用的失败应同时产生 `gen_ai.client.operation.exception` 事件并带上真实的 `exception.type`。
+4. **`agent.*` 系列属性是自定义扩展**。`agent.interaction.id`、`agent.iteration`、`agent.stage`、`agent.degraded`、`agent.fallback.used`、`agent.fallback.reason` **不在 semconv registry 中**,是 Dubbo Admin 私有的命名空间。它们本身合理,但必须在设计里明确标注为自定义属性,避免被误认为符合规范。
+ > **已确认(2026-10-25)**:原 `agent.usage.total_tokens` **已移除**。规范只定义 `gen_ai.usage.input_tokens` 与 `gen_ai.usage.output_tokens`,总量为两者之和;成本核算由企业侧 API 网关承担,Admin 只需给出「模型名 + token 数」。token 口径完全对齐 GenAI 规范,Dubbo Admin 不再维护第二套。
+5. **缺少 `server.address` / `server.port`**。规范列为 Recommended / Conditionally Required。补齐后,分析器可以指出模型调用打到了哪个 provider 端点,对「换了网关导致模型调用变慢」这类问题有价值。
+6. **token 指标口径未对齐**。规范在 span 属性层面用 `gen_ai.usage.input_tokens`(当前 hook 已符合),但在**指标**层面用的是 `gen_ai.client.inference.usage.input_tokens` 等 counter,并额外要求 `gen_ai.token.modality`(`text`/`image`/`audio`,无法判定时为 `unknown`)。当前 hook 不产出 modality,因此如果将来要按 semconv 聚合 token 指标,需要补 `gen_ai.token.modality=unknown`。
+7. **`gen_ai.system` 的历史重命名**。`gen_ai.system` 在早期规范版本中已被 `gen_ai.provider.name` 取代;在当前 checkout 的 `docs/registry/attributes/gen-ai.md` 中**已完全不存在 `gen_ai.system`**,仅保留 `gen_ai.provider.name`。本设计的 Agent hook 已经使用新名,无需兼容旧名;但阅读历史资料时需要留意。
+8. **缺少 `gen_ai.main_agent.*`**。当前规范在 registry 中新增了 `gen_ai.main_agent.id` / `gen_ai.main_agent.name` / `gen_ai.main_agent.description`,用于标识进程内的顶层 Agent 实体。Admin AI 自身就是顶层 Agent,但 hook 未发出这些属性,规范同时要求「不得由进程内临时标识生成 `gen_ai.main_agent.id`」,因此只有在具备稳定标识时才应补齐。这属于可选增强,不阻塞 Phase 1。
+
+## 7. 数据契约
+
+### 7.1 Go 类型
+
+以下类型定义在 `pkg/mcp/tools/trace/model.go`,JSON tag 即 MCP 工具返回给 Agent 的字段名。
+
+| 类型 | 字段(JSON) | 说明 |
+| -- | -- | -- |
+| `TraceSearchQuery` | — | Provider 侧检索入参(进程内结构,不直接序列化) |
+| | `Service` | 必填。根服务名,Provider 无法脱离服务锚点检索 |
+| | `Operation` | 可选,限定根操作名 |
+| | `TraceID` | 可选,锚定单条 trace |
+| | `Tags` | 可选,`map[string]string`,如 `rpc.system=apache_dubbo` |
+| | `StartTime` / `EndTime` | `time.Time`,零值表示不限 |
+| | `Limit` | 限制返回条数,非正值表示 Provider 默认 |
+| | `MinDuration` | `time.Duration`,过滤短于阈值的 trace |
+| `TraceSearchResult` | `traces` / `total` / `truncated` / `sourceEngine` | 搜索结果聚合 |
+| `TraceSummary` | `traceId` | trace 标识 |
+| | `rootService` / `rootOperation` | 根 span 的服务名与操作名 |
+| | `startTime` / `durationMicros` | trace 墙钟包络 |
+| | `spanCount` / `errorSpanCount` / `serviceCount` | 规模指标 |
+| | `hasError` | 是否存在错误 span |
+| | `traceKind` | `"genai"` 或空串 |
+| `TraceAnalysis` | `traceId` / `summary` | 标识与一句话结论 |
+| | `startTime` / `durationMicros` / `spanCount` / `serviceCount` / `errorSpanCount` | 规模指标 |
+| | `criticalPath` | `[]CriticalPathEntry` |
+| | `serviceBreakdown` | `[]ServiceLatency` |
+| | `slowSpans` | `[]SlowSpan` |
+| | `errorSpans` | `[]ErrorSpan` |
+| | `dependencies` | `[]DependencyEdge` |
+| | `genai` | `*GenAISummary`,无 GenAI 属性时为 `nil` |
+| | `truncated` | 分析是否基于裁剪后的 span 计算 |
+| `CriticalPathEntry` | `spanId` / `serviceName` / `operationName` / `startTime` / `durationMicros` | 关键路径一跳 |
+| | `selfMicros` / `waitMicros` | self time 与等待子 span 的时间 |
+| `ServiceLatency` | `serviceName` / `spanCount` / `errorSpanCount` | 服务的 span 规模 |
+| | `totalMicros` / `maxMicros` / `avgMicros` | 服务的耗时统计 |
+| | `sharePercent` | 占 trace 墙钟时长的比例(可为 > 100,见 5.3.3) |
+| `SlowSpan` | `spanId` / `serviceName` / `operationName` / `durationMicros` | 慢 span 标识 |
+| | `sharePercent` / `onCriticalPath` | 占比与是否在关键路径上 |
+| `ErrorSpan` | `spanId` / `serviceName` / `operationName` | 错误 span 标识 |
+| | `errorType` / `errorMessage` | 错误分类与描述 |
+| | `status` | 归一化 span 状态,如 `ERROR` |
+| | `rootCause` | 是否为根因(无失败后代) |
+| `DependencyEdge` | `callerService` / `calleeService` | 调用方与被调方 |
+| | `spanCount` / `totalMicros` / `errorCount` | 边的聚合统计 |
+| `GenAISummary` | `present` | 是否观察到 GenAI 属性 |
+| | `conversationIds` / `providers` / `models` | 去重后的标识集合 |
+| | `inputTokens` / `outputTokens` / `totalTokens` | token 聚合 |
+| | `modelCallCount` / `toolCallCount` / `agentCount` | 调用构成 |
+| | `modelCalls` / `toolCalls` / `agents` | 逐调用明细 |
+| `GenAIModelCall` | `spanId` / `serviceName` / `operationName` / `operation` | 模型调用标识与 `gen_ai.operation.name` |
+| | `providerName` / `requestModel` / `responseModel` | provider 与模型 |
+| | `inputTokens` / `outputTokens` / `totalTokens` / `durationMicros` / `error` | 用量、耗时与失败标记 |
+| `GenAIToolCall` | `spanId` / `serviceName` / `toolName` / `toolCallId` / `durationMicros` / `error` | 工具调用明细 |
+| `GenAIAgent` | `spanId` / `serviceName` / `agentName` / `agentId` / `durationMicros` / `error` | Agent span 明细 |
+
+### 7.2 MCP 工具签名
+
+| 工具 | 参数 | 类型 | 必填 | 说明 |
+| -- | -- | -- | -- | -- |
+| `search_traces` | `service` | string | **是** | 检索的服务锚点 |
+| | `operation` | string | 否 | 根操作名 |
+| | `traceId` | string | 否 | 锚定单条 trace(16/32 位十六进制) |
+| | `tags` | string | 否 | `k=v,k2=v2` 形式的标签过滤 |
+| | `startTime` | string | 否 | RFC3339 起始时间 |
+| | `endTime` | string | 否 | RFC3339 结束时间 |
+| | `minDurationMs` | integer | 否 | 最小耗时(毫秒) |
+| | `limit` | integer | 否 | 返回条数上限,默认 20 |
+| | `provider` | string | 否 | Provider 名,默认 `defaultProvider` |
+| `analyze_trace` | `traceId` | string | **是** | 16/32 位十六进制 trace 标识 |
+| | `provider` | string | 否 | Provider 名,默认 `defaultProvider` |
+| `get_trace_by_id` | `traceId` | string | **是** | 同上(已存在,本设计不改) |
+| | `provider` | string | 否 | 同上(已存在) |
+
+## 8. MCP 工具清单
+
+| 工具 | 用途 | 必需参数 | 典型 Agent 流程 |
+| -- | -- | -- | -- |
+| `search_traces` | 按服务与时间窗发现候选 trace,返回逐条摘要 | `service` | 「最近一小时 X 服务有没有失败的请求」→ 先搜出候选,筛 `hasError=true` |
+| `analyze_trace` | 对单条 trace 做结构化诊断:关键路径、耗时分解、错误归因、依赖边、GenAI 摘要 | `traceId` | 「这条 trace 为什么慢 / 为什么失败」→ 定位到具体 span 与根因 |
+| `get_trace_by_id` | 返回原始 trace 明细(服务、span 关系、耗时、错误状态) | `traceId` | 需要逐 span 细节时兜底,已有工具,不改动 |
+
+`search_traces` 与 `analyze_trace` 在 `pkg/mcp/register.go` 中注册,Description 与 InputSchema 沿用现有风格(`get_trace_by_id` 的 Description 为英文,工具名与参数名保持英文)。
+
+## 9. 错误与边界
+
+所有错误都以 `common.ErrorResult(err)` 返回,即 `ToolResult{IsError: true, Content: [{Type: "text", Text: err.Error()}]}`,**handler 永远返回 `nil` 作为 Go error,不 panic**。这与 `tools.go` 现有的 `GetTraceByID` 行为一致。
+
+| 场景 | 判定 | 对 Agent 的呈现 |
+| -- | -- | -- |
+| Provider 未配置 | `ctx.Config().Observability.Tracing == nil` | `isError=true`,「trace provider is not configured」 |
+| 默认 Provider 未配置 | `tracing.Default()` 不存在 | `isError=true`,「default trace provider is not configured」 |
+| 指定 Provider 不存在 | `tracing.Get(name)` 不存在 | `isError=true`,「trace provider %q is not configured」 |
+| Provider 类型不支持 | `newProvider` 命中 default 分支 | `isError=true`,「unsupported trace provider type: %s」 |
+| Trace 不存在 | Jaeger 返回 404 或 `data` 为空 | `isError=true`,「trace %s was not found」 |
+| TraceID 非法 | 不匹配 `^(?:[0-9a-fA-F]{16}\|[0-9a-fA-F]{32})$` | `isError=true`,「traceId must be a 16 or 32 character hexadecimal value」 |
+| 上游报错 | 非 2xx,或 `errors[]` 非空 | `isError=true`,包含状态码与截断后的错误体(≤ `maxTraceErrorBodyBytes`) |
+| 上游响应体过大 | 超过 `maxTraceResponseBytes` | `isError=true`,「jaeger response exceeds %d bytes」 |
+| 响应不一致 | `data` 长度 ≠ 1 或 traceId 不匹配 | `isError=true`,「jaeger returned an inconsistent response for trace %s」 |
+| 搜索结果为空 | 正常返回 | **`isError=false`**,`traces: []`、`total: 0`;这是合法业务结果而非错误 |
+| 搜索被裁剪 | Provider 命中条数上限(按调用方条数再多取一条探测) | `isError=false`,`truncated: true` |
+| 分析基于裁剪数据 | 底层 `Trace.Truncated` 为真 | `isError=false`,`TraceAnalysis.Truncated = true`,Agent 需在结论中注明数据不完整 |
+| 参数非法 | `startTime`/`endTime` 非 RFC3339、`tags` 格式错误、`minDurationMs` 为负 | `isError=true`,指明具体参数 |
+
+所有上游失败都会经 `jaegerProvider.logFailure` 记录结构化日志,字段为 `provider`、`engine`、`trace_id`(掩码)、`elapsed_ms`、`error_type`,可选 `upstream_status`。
+
+## 10. 安全与隐私
+
+| 方面 | 措施 |
+| -- | -- |
+| 凭据不外泄 | `TraceProviderConfig.BearerToken` 只用于出站请求的 `Authorization: Bearer ` 头,不进入任何工具输出;`TracingConfig.Sanitize()` 在配置展示时把 `BearerToken` 替换为 `config.SanitizedValue`;`get_observability_capabilities` 只返回 provider 的 `name` 与 `type` |
+| 租户隔离 | `TraceProviderConfig.Tenant` 通过 `X-Scope-OrgID` 头(常量 `tenantHeader`)传给后端,不出现在响应中 |
+| TraceID 掩码 | 日志中的 trace_id 经 `maskedTraceID()` 处理,仅保留末 8 位并加 `...` 前缀 |
+| 传输错误净化 | Go 的 `http.Client` 返回 `*url.Error`,其 message 内嵌完整请求 URL(含 query string)。**失败日志**与**工具返回**中的传输错误都经 `sanitizeTransportError()` 解包为内层错误,只保留 `dial tcp ...` 这类诊断信息,避免检索条件、`tags` 值与被配置为 endpoint 的凭据进入日志或模型上下文。为杜绝凭据经 URL 泄漏,`newJaegerProvider` 拒绝带 query 或 fragment 的 endpoint,凭据只允许走 `bearerToken`。工具结果中**上游响应体**的业务敏感数据目前仍**没有过滤机制**,见 14.2 |
+| 服务名 | `service` 既是检索锚点也是链路内的服务标识。它是基础设施元数据(与 OTel 的 resource 属性 `service.name` 同类),不是用户数据也不是凭据;设计文档 §9 明确要求失败日志记录服务名。真正需要防范的是 `tags` 过滤值、span 属性里的业务字段以及内容类属性 |
+| 错误体回显 | 上游错误体读取上限 `maxTraceErrorBodyBytes = 4096`,避免把超长 HTML 错误页灌进模型上下文 |
+| 属性值截断 | 单属性值上限 512 字节,UTF-8 安全截断(`truncateUTF8`) |
+| 内容不外泄 | 分析器不读取也不输出 `gen_ai.input.messages`、`gen_ai.output.messages`、`gen_ai.tool.call.arguments`、`gen_ai.tool.call.result` |
+| Agent 侧内容捕获 | `ai/component/hooks/config.go` 提供 `CaptureNone` / `CaptureTruncated` / `CaptureFull` 三档(`ai/component/hooks/component.go`)。`none` 完全不发出内容属性;`truncated` 在 `truncatedContentLimit = 4096` 字节内截断,单个 JSON 字符串再截到 512 字符;`full` 原样输出。默认配置为 `CaptureNone` |
+| 提示注入 | trace 属性、错误消息、span 名都是**不可信数据**,只能作为当前轮观察数据传给模型,不得作为指令 |
+
+其中内容捕获策略与分析器口径是配合的:`CaptureNone` 下 Agent 自己的 trace 里根本不含提示词正文,分析器拿到的只有模型名与 token 数——这正是我们想要的最小数据集。
+
+## 11. 测试策略
+
+### 11.1 单元测试矩阵
+
+| 文件 | 覆盖点 |
+| -- | -- |
+| `pkg/mcp/tools/trace/search_test.go` | `tags` 字符串解析(`k=v,k2=v2`、空串、缺 `=`、重复键、含空格);`startTime`/`endTime` 的 RFC3339 解析与非法值;`minDurationMs` → `MinDuration` 换算;`limit` 默认值与上限裁剪;`traceId` 格式校验;provider 未配置 / 指定 provider 不存在;空结果返回 `total=0` 且 `isError=false`;`traceKind` 判定(含含 `gen_ai.*` 与不含两种用例);`truncated` 标记 |
+| `pkg/mcp/tools/trace/analyzer_test.go` | span 树构建(正常父子、孤儿 span、`ParentSpanID` 悬空);**self time**:单子 span、两个**重叠**并发子 span(验证并集而非求和)、三层嵌套、子 span 越界、负值归零;**critical path**:多分支取 `end` 最大者、多根取 `end` 最大根、叶子终止;`serviceBreakdown` 的 `totalMicros`/`maxMicros`/`avgMicros`/`sharePercent`;`slowSpans` 按 self time 排序与 `onCriticalPath` 标记;**错误归因**:`error.type` 优先、`exception.type` 次之、`error.message` 兜底、全缺失时为 `unknown`;**根因检测**:最深层失败 `rootCause=true`、传播型错误 `rootCause=false`;**依赖边**:去重、`spanCount`/`totalMicros`/`errorCount` 聚合、按 `totalMicros` 排序;`truncated` 传播;时间解析失败时降级不 panic |
+| `pkg/mcp/tools/trace/genai_test.go` | `gen_ai.operation.name` 到模型调用/工具调用/Agent 的分类;`gen_ai.provider.name`/`gen_ai.request.model`/`gen_ai.response.model` 提取;`gen_ai.usage.input_tokens`/`output_tokens` 解析与求和;`gen_ai.conversation.id` 去重;`gen_ai.agent.name`/`gen_ai.agent.id` 提取;无 `gen_ai.*` 属性时 `GenAI` 为 `nil`;`error.type` 分类 |
+| `pkg/mcp/tools/trace/jaeger_test.go` | 保持现有 4 个用例不变(回归保护) |
+
+### 11.2 Agent 侧 hook 测试
+
+`ai/component/hooks/tracing_test.go` 补充属性断言:
+
+- span 名:`chat {model}`、`execute_tool {tool}`、`invoke_agent {agent.name}`(补齐 `gen_ai.agent.name` 后)。
+- span kind:模型调用为 `CLIENT`,Agent / 工具调用为 `INTERNAL`。
+- 属性存在性与取值:`gen_ai.operation.name`、`gen_ai.provider.name`(映射表命中与未命中两种)、`gen_ai.request.model`、`gen_ai.conversation.id`、`gen_ai.usage.input_tokens`、`gen_ai.usage.output_tokens`、`gen_ai.tool.name`、`gen_ai.tool.call.id`、`error.type`。
+- 异常路径:产生 `gen_ai.client.operation.exception` 事件且 `exception.type` 为内层真实类型。
+- 内容捕获策略三档:`none` 不产生内容属性;`truncated` 长度受限;`full` 原样。
+
+### 11.3 集成测试
+
+- 用 `httptest` 起一个假 Jaeger:校验 `search_traces` 实际发出的查询参数(`service`/`operation`/`tags`/`start`/`end`/`limit`),以及 `analyze_trace` 对完整 trace 的端到端诊断输出。
+- MCP Server 侧:`tools/list` 能看到 `search_traces` 与 `analyze_trace`;调用参数缺 `service` 时返回 `isError=true`。
+
+## 12. 分阶段实施
+
+### Phase 1:P0 核心(本次 P0 任务的主体)
+
+交付物:
+
+- `pkg/mcp/tools/trace/search.go`:Provider 侧检索能力(`Provider` 接口新增 `SearchTraces` 方法,Jaeger 实现走 v1 查询 API),MCP 工具 `search_traces`。
+- `pkg/mcp/tools/trace/analyzer.go`:本地诊断计算与 MCP 工具 `analyze_trace`。
+- `pkg/mcp/tools/trace/genai.go`:GenAI 属性识别与归类。
+- `pkg/mcp/register.go`:注册两个新工具。
+- `ai/component/hooks/tracing.go`:按 6.4 对齐语义约定(provider 映射表、`gen_ai.agent.name`、GenAI 异常事件、`server.address`/`server.port`)。
+- 上述单元测试与集成测试。
+
+退出标准:
+
+- `go test ./pkg/mcp/tools/trace/...` 全绿;`ai` 模块 `go test ./...` 全绿。
+- MCP `tools/list` 出现 `search_traces` 与 `analyze_trace`。
+- 在一个接入真实 Jaeger 的环境中,Agent 能在**零人工 traceId** 的前提下回答「最近一小时 X 服务的失败 trace 有哪些」。
+- 现有 `get_trace_by_id` 行为无回归。
+
+### Phase 2:metric / trace / log 联动
+
+交付物:
+
+- 在 `analyze_trace` 的输出中补齐可直接用于 PromQL 的维度(服务名、接口名),与 PR #1607 引入的 `dubbo.application` / `dubbo.metric.interface` 标签对齐。
+- 在 Agent 的工作流层串联 `search_logs`(按 `traceId` 过滤)与 `query_prometheus`,形成「trace → 日志行 → 指标序列」的完整调查闭环。
+- 与 #1537 的 metric-trace-log 关联方案对齐标签口径,避免两套命名。
+
+退出标准:
+
+- Agent 能对单条失败 trace 给出「错误 span + 对应日志行 + 相关指标序列」的联合结论。
+- PromQL 中使用的标签名与 #1607 / #1537 的口径一致。
+
+### Phase 3:纵深能力
+
+交付物:
+
+- Collector 侧 spanmetrics / exemplars 打通(与 #1607 互补,不重复造)。
+- 多 Provider 支持(如 Tempo),把 `Provider` 接口的检索能力抽象为可插拔。
+- 跨 trace 聚合:按服务统计错误率与慢调用分布,支撑「最近一小时错误率是上升还是下降」这类问题。
+
+退出标准:
+
+- Provider 接口的实现数量 ≥ 2,且新增 Provider 不需要改动 MCP 工具层。
+- Agent 能给出跨 trace 的时间趋势结论。
+
+## 13. 度量与验收
+
+### 13.1 核心验收场景
+
+| 场景 | 问题 | 期望 |
+| -- | -- | -- |
+| 发现 | 「最近一小时 shop-order 有失败的请求吗」 | Agent 自主调用 `search_traces`,**零人工 traceId**,返回失败 trace 列表 |
+| 定位慢 | 「这条 trace 为什么慢」 | 返回 `criticalPath` 与 top `slowSpans`,指出瓶颈服务与占比 |
+| 归因失败 | 「这条 trace 为什么失败」 | 返回 `errorSpans`,其中 `rootCause=true` 的那条带 `errorType` |
+| 拓扑 | 「这条 trace 经过了哪些服务」 | 返回 `dependencies` 调用边 |
+| LLM 可观测 | 「这次 AI 对话花了多少 token、哪一步失败」 | 返回 `genai` 摘要,含 `models`、`totalTokens`、`modelCalls`、`toolCalls` |
+
+### 13.2 可实际执行的检查
+
+```bash
+# 单元测试
+go test ./pkg/mcp/tools/trace/...
+(cd ai && go test ./component/hooks/...)
+
+# 工具已注册(MCP Server 运行后调用 tools/list)
+# 期望能看到 "search_traces" 与 "analyze_trace"
+
+# 空结果必须是正常结果而不是错误
+# 调用 search_traces {service: "<不存在的服务>", startTime: ""}
+# 期望 isError=false 且 traces=[] 、total=0
+```
+
+### 13.3 效果指标
+
+- 观测类问题中,**不需要用户提供 traceId** 的比例(目标:trace 相关问题 100% 由 Agent 自主发现)。
+- `search_traces` → `analyze_trace` 的平均调用轮数(目标:定位单条 trace ≤ 2 轮工具调用)。
+- 工具返回中被裁剪(`truncated=true`)的比例,用于反向调整 `maxTraceSpans` 等上限。
+
+## 14. 风险与待确认事项
+
+### 14.1 结论状态(2026-10-25 已确认)
+
+| # | 事项 | 状态 |
+| -- | -- | -- |
+| 1 | Jaeger v1 查询 API 表达能力有限 | 保留为已知限制;如需更强检索后端(Jaeger gRPC query / Tempo search)在 Phase 3 一并考虑 |
+| 2 | `service` 参数语义需在部署环境验证 | 保留为部署验证项,需真实 Jaeger 环境确认 |
+| 3 | 是否暴露 Console API | **本期不做**,记为 To Do T1(控制台已有 Grafana trace 面板,再做一套会造成页面与日志冗余) |
+| 4 | 采样缺口 | **结论**:采样是客户端行为,缺的是「数据不完整」这一信息;`search_traces` 返回需携带采样信息,记为 To Do T5 |
+| 5 | token 指标是否对齐 GenAI token metric | **结论**:span 属性层已对齐;指标层(`gen_ai.client.inference.usage.*` + `gen_ai.token.modality`)本期不做——Admin 不产出 token 指标,成本由企业网关核算 |
+| 6 | 规范版本钉死 | **结论**:以 `6fd0d76` 为锁定基准 commit,升级规范时必须重跑对齐检查 |
+| 7 | `gen_ai.provider.name` 映射表 | **结论并已实现**:不做大而全映射表。按实际配置落地别名表(`gemini` → `gcp.gemini`),16 个 well-known 值落成参考集合但**从不过滤**枚举外的值——规范允许自定义值。模型名整串透传 |
+| 8 | 多租户下的 trace 可见性 | 保留为部署确认项 |
+| 9 | 多 provider 检索优先级 | **结论**:做成配置参数 `tracing.searchProviders`,留空时行为不变。记为 To Do T4(Phase 3) |
+| 10 | `capture_content` 与大模型思考过程 | **结论并已实现**:拆成两个独立参数,新增 `capture_reasoning`(默认 `none`);思考过程按规范以 `reasoning` part 承载在 `gen_ai.output.messages` 中,已补上 `convertGenAIMessages` 的转换与 JSON 层策略过滤 |
+| 11 | GenAI 异常事件的 severity | **结论**:Go SDK 的 span 事件无 severity 字段且 `EventOption` 不可在包外实现,先以 span status = Error 兜底;如后端需要按 severity 过滤,再以事件属性承载 severity number。记为 To Do T6 |
+| 12 | 统一收集平台选型 | **结论**:Admin 只保证出口符合 OTLP 标准协议,不指定 URL、后端类型或厂商 SDK;企业可用 Collector 或代理统一接入 |
+| 13 | 存储与脱敏 | **结论**:开发阶段全量提供,企业侧过滤逻辑自行处理;Admin 只需保证「出口标准、属性可过滤、不绑后端、敏感面可控」四项能力 |
+
+### 14.2 To Do 清单(本期不做,已记录)
+
+- [ ] **T1** trace 检索暴露为 Console API —— 控制台已有 Grafana trace 面板,再做一套会造成页面与日志冗余;等 Agent 侧能力稳定后评估。
+- [x] **T2** `gen_ai.provider.name` 取值映射 —— **已完成**。按 `ai/component/models/models.yaml` 的实际配置落地别名表:`gemini` → `gcp.gemini`(规范依据:base_url 为 `generativelanguage.googleapis.com`,规范对 `gcp.gemini` 的描述即「Also known as the AI Studio API」)。`dashscope`、`siliconflow` 无规范枚举项,按规范「otherwise, a custom value MAY be used」作为合法自定义值原样上报。模型名(第一个 `/` 之后的全部内容,含斜杠)整串透传。
+- [x] **T3** `capture_reasoning` 独立配置参数 + `gen_ai.output.messages` 的 reasoning part 转换 —— **已完成**。新增 `capture_reasoning`(`none`/`truncated`/`full`,默认 `none`),与 `capture_content` 正交;`convertGenAIMessages` 补上 `type: "reasoning"` 的 part 转换;策略在 JSON 层生效,先于整体截断执行。
+- [ ] **T4** 多 provider 检索合并的配置参数 `tracing.searchProviders`(留空时行为不变),Phase 3。
+- [ ] **T5** 检索结果携带采样信息,让 Agent 在返回 0 条时提示「当前采样率 X,结论仅供参考」,Phase 2。
+- [ ] **T6** GenAI 异常事件的 severity 表达(以事件属性承载 severity number),按需启用。
+- [ ] **T7** **工具返回的敏感内容过滤机制尚未实现**。传输错误已经净化:请求 URL(endpoint 路径与全部检索条件)不会进入工具返回,只有 `dial tcp ...` 这类诊断信息与模型自选的 traceId / service 锚点保留。**仍未净化**的是上游响应体与 span 内容——工具入参/返回值可能包含用户标识与业务数据。生产环境需要按部署场景决定过滤/脱敏口径(例如按工具白名单、按属性名过滤、或在 Collector 侧统一处理)。机制未实现,代码中已留 TODO 标记。
+
+### 14.3 风险
+
+| 风险 | 影响 | 缓解 |
+| -- | -- | -- |
+| `maxTraceSpans = 500` 在超大 trace 上裁剪子树 | 关键路径断裂、self time 失真 | 明确标记 `truncated`;延迟分母改用 provider 的完整包络,裁剪不再让占比虚高,但关键路径本身仍只在保留的 span 上计算;Phase 3 引入采样式采样或按服务子树裁剪 |
+| 错误根因误判(中间层也是 ERROR) | 结论指向错误的服务 | 根因规则基于「无失败后代」;同时保留全部 `errorSpans` 供 Agent 复核 |
+| 规范持续演进导致属性漂移 | 分析器识别不到 GenAI span | 钉住 commit;识别逻辑对未知 `gen_ai.operation.name` 做宽松归类(未知值仍标记为 GenAI span) |
+| 与 #1607 的标签口径不一致 | Phase 2 联动失败 | Phase 1 结束前与 #1607 的 `dubbo.application` / `dubbo.metric.interface` 对齐命名 |
+| 工具输出膨胀 | 挤占模型上下文 | 全部列表有上限,超限置 `truncated`;分析器不输出原始 span 列表 |
+
+## 15. 关联链接
+
+### 15.1 讨论与 issue
+
+- [2026-10-24 Weekly(Discussion #1612)](https://github.com/apache/dubbo-admin/discussions/1612) —— 本 P0 任务的来源
+- [2026 路线图(Discussion #1386)](https://github.com/apache/dubbo-admin/discussions/1386) —— AI/SRE-Agent 第 7 项「设计Hook系统,接入otel trace」;可观测能力第 3 项「基于Grafana,探索metric,trace,log的联动」
+- [Issue #1606:Correlate Jaeger traces with Dubbo RPC metrics](https://github.com/apache/dubbo-admin/issues/1606)
+- [PR #1607:feat(observability): correlate Jaeger traces with Dubbo RPC metrics](https://github.com/apache/dubbo-admin/pull/1607)
+- [Issue #1537:[Proposal] Add log observability and metric-trace-log correlation](https://github.com/apache/dubbo-admin/issues/1537)
+- [Issue #1524:[OSPP] Observability optimization in Admin](https://github.com/apache/dubbo-admin/issues/1524)
+- [PR #1499:feat: add PromQL and trace diagnosis tools](https://github.com/apache/dubbo-admin/pull/1499)
+- [PR #1536:feat(ai): add agent hooks and OpenTelemetry tracing](https://github.com/apache/dubbo-admin/pull/1536)
+
+### 15.2 规范
+
+- OpenTelemetry semantic conventions for generative AI systems(只读 checkout,commit `6fd0d76`)
+ - `docs/gen-ai/README.md`
+ - `docs/gen-ai/gen-ai-spans.md`
+ - `docs/gen-ai/gen-ai-agent-spans.md`
+ - `docs/gen-ai/client-inference.md`
+ - `docs/gen-ai/gen-ai-events.md`
+ - `docs/gen-ai/gen-ai-exceptions.md`
+ - `docs/gen-ai/gen-ai-token-metrics.md`
+ - `docs/registry/attributes/gen-ai.md`
+- [Document Status 定义](https://opentelemetry.io/docs/specs/otel/document-status)
+
+### 15.3 仓库内相关文件
+
+| 路径 | 作用 |
+| -- | -- |
+| `pkg/mcp/tools/trace/provider.go` | `Provider` 接口与 Provider 工厂 |
+| `pkg/mcp/tools/trace/tools.go` | `get_trace_by_id` handler、provider 解析、input schema |
+| `pkg/mcp/tools/trace/jaeger.go` | Jaeger v1 Provider、归一化与有界裁剪 |
+| `pkg/mcp/tools/trace/model.go` | `Trace`/`Span`/`Event` 与搜索、分析契约类型 |
+| `pkg/mcp/tools/trace/search.go` | 新增:`search_traces` |
+| `pkg/mcp/tools/trace/analyzer.go` | 新增:`analyze_trace` |
+| `pkg/mcp/tools/trace/genai.go` | 新增:GenAI 属性识别 |
+| `pkg/mcp/register.go` | MCP 工具注册 |
+| `pkg/mcp/common/types.go`、`pkg/mcp/common/utils.go` | `ToolDef`、`PropertyDef`、`ToolResult`、`ArgsHelper`、`JsonResult`、`ErrorResult` |
+| `pkg/config/observability/tracing.go` | `TracingConfig` / `TraceProviderConfig` |
+| `pkg/mcp/tools/log/tools.go` | `search_logs`(支持 `traceId` 过滤) |
+| `pkg/mcp/tools/metrics/prometheus.go` | `query_prometheus` / `query_prometheus_range` 与查询上限 |
+| `pkg/mcp/tools/observability.go` | `get_observability_capabilities` |
+| `ai/component/hooks/tracing.go` | Agent 侧 OTel hook |
+| `ai/component/hooks/component.go` | OTLP exporter 与内容捕获策略常量 |
+| `ai/component/hooks/config.go` | hook 配置与默认 `CaptureNone` |
+| `docs/ai-context-management-design.md` | 同系列设计文档(上下文管理) |
diff --git a/pkg/mcp/DEVELOPMENT.md b/pkg/mcp/DEVELOPMENT.md
index 79f4121de..8db393f00 100644
--- a/pkg/mcp/DEVELOPMENT.md
+++ b/pkg/mcp/DEVELOPMENT.md
@@ -417,6 +417,8 @@ Dubbo Admin MCP 提供以下只读诊断工具:
| `query_prometheus` | Prometheus | 执行即时 PromQL 查询 |
| `query_prometheus_range` | Prometheus | 执行最长 24 小时的区间 PromQL 查询 |
| `get_trace_by_id` | Jaeger | 按 TraceID 查询规范化调用链 |
+| `search_traces` | Jaeger | 按服务、时间窗与过滤条件发现候选链路,返回逐条摘要 |
+| `analyze_trace` | Jaeger | 对单条链路做结构化诊断:关键路径、耗时分解、错误归因、依赖边、GenAI 用量 |
| `get_observability_capabilities` | 本地配置 | 查询数据源和查询限制 |
Prometheus 工具复用 `observability.prometheus`。区间查询的 `step` 不得低于 15 秒,返回结果最多包含 100 个时间序列和 5000 个采样点。结果被裁剪时,响应中的 `truncated` 为 `true`。
@@ -436,3 +438,5 @@ observability:
```
诊断工具不会记录完整 PromQL、Bearer Token 或完整 trace attributes。工具只读,不提供流量、配置或部署变更能力。
+
+`search_traces` 必须提供 `service` 锚点,默认请求 20 条、最多 100 条,本地最多保留 100 条摘要;后端命中条数上限时 `truncated` 为 `true`。`analyze_trace` 最多输出 32 跳关键路径、10 个慢 span、20 个错误 span、20 个 GenAI 明细,并给出一句话结论。分析器只读取 `gen_ai.*` 元数据属性,不回显 prompt、completion 与工具入参等 Opt-In 内容。
diff --git a/pkg/mcp/register.go b/pkg/mcp/register.go
index 2d8856a58..8733ecbd6 100644
--- a/pkg/mcp/register.go
+++ b/pkg/mcp/register.go
@@ -406,6 +406,28 @@ func RegisterTools(server *Server) {
Handler: tracetools.GetTraceByID,
})
+ server.RegisterTool(&common.ToolDef{
+ Name: "search_traces",
+ Description: "Search distributed traces by service, time window, and optional filters, returning bounded per-trace summaries to discover candidate traces",
+ InputSchema: common.InputSchema{
+ Type: "object",
+ Properties: tracetools.SearchProperties(),
+ Required: []string{"service"},
+ },
+ Handler: tracetools.SearchTraces,
+ })
+
+ server.RegisterTool(&common.ToolDef{
+ Name: "analyze_trace",
+ Description: "Diagnose a distributed trace by TraceID and return the critical path, service latency breakdown, slow spans, error root causes, dependency edges, and GenAI usage",
+ InputSchema: common.InputSchema{
+ Type: "object",
+ Properties: tracetools.AnalyzeProperties(),
+ Required: []string{"traceId"},
+ },
+ Handler: tracetools.AnalyzeTrace,
+ })
+
server.RegisterTool(&common.ToolDef{
Name: "get_observability_capabilities",
Description: "Report Prometheus and trace providers, capabilities, and enforced query limits",
diff --git a/pkg/mcp/register_test.go b/pkg/mcp/register_test.go
index 5d423b657..af644b77a 100644
--- a/pkg/mcp/register_test.go
+++ b/pkg/mcp/register_test.go
@@ -67,6 +67,8 @@ func TestRegisterObservabilityTools(t *testing.T) {
{name: "query_prometheus", required: []string{"query"}},
{name: "query_prometheus_range", required: []string{"query", "startTime", "endTime", "step"}},
{name: "get_trace_by_id", required: []string{"traceId"}},
+ {name: "search_traces", required: []string{"service"}},
+ {name: "analyze_trace", required: []string{"traceId"}},
{name: "get_observability_capabilities"},
}
for _, test := range tests {
diff --git a/pkg/mcp/tools/trace/analyzer.go b/pkg/mcp/tools/trace/analyzer.go
new file mode 100644
index 000000000..0a097eab0
--- /dev/null
+++ b/pkg/mcp/tools/trace/analyzer.go
@@ -0,0 +1,831 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package trace
+
+import (
+ "context"
+ "fmt"
+ "math"
+ "sort"
+ "strconv"
+ "strings"
+ "time"
+ "unicode/utf8"
+
+ consolectx "github.com/apache/dubbo-admin/pkg/console/context"
+ "github.com/apache/dubbo-admin/pkg/mcp/common"
+)
+
+const (
+ // Bounded output limits. They keep the tool payload small enough to stay inside
+ // the model context window while still describing the most relevant spans.
+ maxCriticalPath = 32
+ maxServiceBreakdown = 50
+ maxSlowSpans = 10
+ maxErrorSpans = 20
+ maxDependencyEdges = 50
+ maxGenAICalls = 20
+
+ maxAnalysisSummaryChars = 300
+ maxErrorTypeChars = 96
+ maxErrorMessageChars = 256
+)
+
+const (
+ errorTypeAttribute = "error.type"
+ errorMessageAttribute = "error.message"
+ exceptionTypeAttribute = "exception.type"
+ exceptionMessageAttribute = "exception.message"
+ spanEventNameAttribute = "event"
+ standardExceptionEvent = "exception"
+ genAIExceptionEvent = "gen_ai.client.operation.exception"
+)
+
+// analysisSpan is the analyzer's internal view of a span with a resolved timeline.
+type analysisSpan struct {
+ span *Span
+ position int
+ startMicros int64
+ endMicros int64
+ timeValid bool
+ selfMicros int64
+ waitMicros int64
+ parent *analysisSpan
+ children []*analysisSpan
+}
+
+// timelineInterval is a half-open interval on the absolute microsecond timeline.
+type timelineInterval struct {
+ start int64
+ end int64
+}
+
+// AnalyzeTrace diagnoses a single distributed trace and returns a structured report.
+func AnalyzeTrace(ctx consolectx.Context, args map[string]any) (*common.ToolResult, error) {
+ helper := common.NewArgsHelper(args)
+ traceID := helper.GetString("traceId", "")
+ if !traceIDPattern.MatchString(traceID) {
+ return common.ErrorResult(fmt.Errorf("traceId must be a 16 or 32 character hexadecimal value")), nil
+ }
+ providerConfig, err := providerConfigFromContext(ctx, helper.GetString("provider", ""))
+ if err != nil {
+ return common.ErrorResult(err), nil
+ }
+ provider, err := newProvider(providerConfig)
+ if err != nil {
+ return common.ErrorResult(err), nil
+ }
+ trace, err := provider.GetTraceByID(traceRequestContext(ctx), traceID)
+ if err != nil {
+ return common.ErrorResult(err), nil
+ }
+ return common.JsonResult(BuildTraceAnalysis(trace))
+}
+
+// AnalyzeProperties returns the MCP input schema properties for trace diagnosis.
+func AnalyzeProperties() map[string]common.PropertyDef {
+ return map[string]common.PropertyDef{
+ "traceId": {Type: "string", Description: "Jaeger 64-bit or 128-bit hexadecimal TraceID"},
+ "provider": {Type: "string", Description: "Optional trace provider name; defaults to defaultProvider"},
+ }
+}
+
+// BuildTraceAnalysis turns a normalized trace into a structured diagnosis.
+// It is a pure function so the diagnosis rules can be verified without a backend.
+func BuildTraceAnalysis(trace *Trace) *TraceAnalysis {
+ analysis := &TraceAnalysis{}
+ if trace == nil {
+ analysis.Summary = "no trace data available"
+ return analysis
+ }
+ analysis.TraceID = trace.TraceID
+ analysis.Truncated = trace.Truncated
+
+ nodes := buildAnalysisTree(trace.Spans)
+ computeSelfTimes(nodes, analysis)
+ criticalPath, criticalPathSet := buildCriticalPath(nodes, analysis)
+ analysis.CriticalPath = criticalPath
+
+ wallClock, traceStart := analysisEnvelope(nodes, trace)
+ analysis.DurationMicros = wallClock
+ analysis.StartTime = traceStart
+ analysis.SpanCount = maxInt(trace.SpanCount, len(trace.Spans))
+
+ breakdown := buildServiceBreakdown(nodes, wallClock, analysis)
+ analysis.ServiceBreakdown = breakdown
+ analysis.ServiceCount = maxInt(trace.ServiceCount, len(breakdown))
+
+ slowSpans := buildSlowSpans(nodes, wallClock, criticalPathSet)
+ analysis.SlowSpans = make([]SlowSpan, 0, len(slowSpans))
+ for _, entry := range slowSpans {
+ analysis.SlowSpans = append(analysis.SlowSpans, entry.slow)
+ }
+
+ errorSpans, errorCount := buildErrorSpans(nodes, analysis)
+ analysis.ErrorSpans = errorSpans
+ // The provider already counted errors over the complete trace; that count must not
+ // be hidden by the bounded error span list.
+ analysis.ErrorSpanCount = maxInt(trace.ErrorSpanCount, errorCount)
+
+ analysis.Dependencies = buildDependencyEdges(nodes, analysis)
+
+ if genai := BuildGenAISummary(trace.Spans); genai.Present {
+ analysis.GenAI = genai
+ }
+
+ analysis.Summary = buildAnalysisSummary(analysis, slowSpans)
+ return analysis
+}
+
+// buildAnalysisTree links the spans into a parent/child forest. A span whose parent
+// is missing from the trace degrades into an orphan root: it still contributes to the
+// aggregate statistics but it cannot extend a parent chain.
+func buildAnalysisTree(spans []Span) []*analysisSpan {
+ nodes := make([]*analysisSpan, 0, len(spans))
+ index := make(map[string]*analysisSpan, len(spans))
+ for i := range spans {
+ span := &spans[i]
+ node := &analysisSpan{span: span, position: i}
+ if parsed, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(span.StartTime)); err == nil {
+ node.timeValid = true
+ node.startMicros = parsed.UnixMicro()
+ node.endMicros = node.startMicros + span.DurationMicros
+ } else {
+ node.endMicros = span.DurationMicros
+ }
+ nodes = append(nodes, node)
+ if span.SpanID != "" {
+ index[span.SpanID] = node
+ }
+ }
+ for _, node := range nodes {
+ parentID := node.span.ParentSpanID
+ if parentID == "" {
+ continue
+ }
+ parent, ok := index[parentID]
+ if !ok || parent == node {
+ continue
+ }
+ node.parent = parent
+ parent.children = append(parent.children, node)
+ }
+ return nodes
+}
+
+// computeSelfTimes derives the exclusive time of every span. The covered time is the
+// union of the clipped child intervals, so concurrent children are not counted twice.
+func computeSelfTimes(nodes []*analysisSpan, analysis *TraceAnalysis) {
+ for _, node := range nodes {
+ if !node.timeValid {
+ // Without a trustworthy start time the span degrades to a leaf. It
+ // cannot be placed on the timeline, so it must not claim any time
+ // of its own: counting its full duration here would let the parent
+ // account for the same time a second time.
+ node.selfMicros = 0
+ node.waitMicros = 0
+ analysis.Truncated = true
+ continue
+ }
+ intervals := make([]timelineInterval, 0, len(node.children))
+ for _, child := range node.children {
+ if !child.timeValid {
+ analysis.Truncated = true
+ continue
+ }
+ start := maxInt64(child.startMicros, node.startMicros)
+ end := minInt64(child.endMicros, node.endMicros)
+ if end <= start {
+ continue
+ }
+ intervals = append(intervals, timelineInterval{start: start, end: end})
+ }
+ covered := unionLength(intervals)
+ node.waitMicros = covered
+ node.selfMicros = node.span.DurationMicros - covered
+ if node.selfMicros < 0 {
+ node.selfMicros = 0
+ }
+ }
+}
+
+// unionLength returns the length of the union of the given intervals.
+func unionLength(intervals []timelineInterval) int64 {
+ if len(intervals) == 0 {
+ return 0
+ }
+ sort.Slice(intervals, func(i, j int) bool {
+ if intervals[i].start != intervals[j].start {
+ return intervals[i].start < intervals[j].start
+ }
+ return intervals[i].end < intervals[j].end
+ })
+ total := int64(0)
+ current := intervals[0]
+ for _, next := range intervals[1:] {
+ if next.start > current.end {
+ total += current.end - current.start
+ current = next
+ continue
+ }
+ if next.end > current.end {
+ current.end = next.end
+ }
+ }
+ return total + current.end - current.start
+}
+
+// buildCriticalPath walks the latency determining root-to-leaf chain.
+// The second return value holds the analyzer internal positions of the spans on the
+// path. The position is used instead of the business span id because a span id can
+// be empty or duplicated, which would make two different spans overwrite each other.
+func buildCriticalPath(nodes []*analysisSpan, analysis *TraceAnalysis) ([]CriticalPathEntry, map[int]struct{}) {
+ roots := make([]*analysisSpan, 0, len(nodes))
+ for _, node := range nodes {
+ if node.parent == nil {
+ roots = append(roots, node)
+ }
+ }
+ entries := make([]CriticalPathEntry, 0, maxCriticalPath)
+ onPath := map[int]struct{}{}
+ visited := map[int]struct{}{}
+ current := selectCriticalRoot(roots)
+ for current != nil {
+ if len(entries) >= maxCriticalPath {
+ // The budget is exhausted while the chain still continues, so the
+ // remaining hops are dropped and the analysis is incomplete.
+ analysis.Truncated = true
+ break
+ }
+ if _, seen := visited[current.position]; seen {
+ break
+ }
+ visited[current.position] = struct{}{}
+ onPath[current.position] = struct{}{}
+ entries = append(entries, CriticalPathEntry{
+ SpanID: current.span.SpanID,
+ ServiceName: current.span.ServiceName,
+ OperationName: current.span.OperationName,
+ StartTime: current.span.StartTime,
+ DurationMicros: current.span.DurationMicros,
+ SelfMicros: current.selfMicros,
+ WaitMicros: current.waitMicros,
+ })
+ current = selectCriticalChild(current)
+ }
+ return entries, onPath
+}
+
+// selectCriticalRoot picks the root that finishes last, because that root determines
+// the end of the request.
+func selectCriticalRoot(roots []*analysisSpan) *analysisSpan {
+ var best *analysisSpan
+ for _, root := range roots {
+ if !root.timeValid {
+ continue
+ }
+ if best == nil || betterCriticalRoot(root, best) {
+ best = root
+ }
+ }
+ if best != nil {
+ return best
+ }
+ for _, root := range roots {
+ if best == nil || betterCriticalRoot(root, best) {
+ best = root
+ }
+ }
+ return best
+}
+
+func betterCriticalRoot(candidate, current *analysisSpan) bool {
+ if candidate.endMicros != current.endMicros {
+ return candidate.endMicros > current.endMicros
+ }
+ if candidate.span.DurationMicros != current.span.DurationMicros {
+ return candidate.span.DurationMicros > current.span.DurationMicros
+ }
+ // The start time is derived from the end time and the duration, so it is
+ // already equal here and cannot be used as a tie breaker.
+ return candidate.span.SpanID < current.span.SpanID
+}
+
+// selectCriticalChild picks the child that finishes last, because it is the activity
+// that determined when the parent completed.
+func selectCriticalChild(node *analysisSpan) *analysisSpan {
+ var best *analysisSpan
+ for _, child := range node.children {
+ if !child.timeValid {
+ // A span without a trustworthy timestamp degrades to a leaf and is not
+ // allowed to extend the chain.
+ continue
+ }
+ if best == nil || betterCriticalChild(child, best) {
+ best = child
+ }
+ }
+ return best
+}
+
+func betterCriticalChild(candidate, current *analysisSpan) bool {
+ if candidate.endMicros != current.endMicros {
+ return candidate.endMicros > current.endMicros
+ }
+ if candidate.span.DurationMicros != current.span.DurationMicros {
+ return candidate.span.DurationMicros > current.span.DurationMicros
+ }
+ return candidate.span.SpanID < current.span.SpanID
+}
+
+// analysisEnvelope returns the wall clock duration and the start timestamp of the trace.
+func analysisEnvelope(nodes []*analysisSpan, trace *Trace) (int64, string) {
+ var minStart, maxEnd int64
+ hasStart := false
+ startText := trace.StartTime
+ for _, node := range nodes {
+ if !node.timeValid {
+ continue
+ }
+ if !hasStart || node.startMicros < minStart {
+ minStart = node.startMicros
+ startText = node.span.StartTime
+ hasStart = true
+ }
+ if node.endMicros > maxEnd {
+ maxEnd = node.endMicros
+ }
+ }
+ if !hasStart {
+ return trace.DurationMicros, trace.StartTime
+ }
+ // The provider envelope covers every raw span, the node envelope only the
+ // retained ones. A clipped span list would make the node envelope shorter
+ // than the trace and inflate every latency percentage that uses it as the
+ // denominator, so the provider value is preferred whenever it is available.
+ if trace.DurationMicros > 0 {
+ return trace.DurationMicros, startText
+ }
+ return maxEnd - minStart, startText
+}
+
+// buildServiceBreakdown aggregates the spans per service.
+func buildServiceBreakdown(nodes []*analysisSpan, wallClock int64, analysis *TraceAnalysis) []ServiceLatency {
+ aggregates := map[string]*ServiceLatency{}
+ for _, node := range nodes {
+ span := node.span
+ entry, ok := aggregates[span.ServiceName]
+ if !ok {
+ entry = &ServiceLatency{ServiceName: span.ServiceName}
+ aggregates[span.ServiceName] = entry
+ }
+ entry.SpanCount++
+ if spanFailed(span) {
+ entry.ErrorSpanCount++
+ }
+ entry.TotalMicros += span.DurationMicros
+ if span.DurationMicros > entry.MaxMicros {
+ entry.MaxMicros = span.DurationMicros
+ }
+ }
+ result := make([]ServiceLatency, 0, len(aggregates))
+ for _, entry := range aggregates {
+ if entry.SpanCount > 0 {
+ entry.AvgMicros = entry.TotalMicros / int64(entry.SpanCount)
+ }
+ // The share is the service workload relative to the wall clock of the trace.
+ // Concurrent spans can push it above 100 percent on purpose.
+ entry.SharePercent = percentOf(entry.TotalMicros, wallClock)
+ result = append(result, *entry)
+ }
+ sort.SliceStable(result, func(i, j int) bool {
+ if result[i].TotalMicros != result[j].TotalMicros {
+ return result[i].TotalMicros > result[j].TotalMicros
+ }
+ return result[i].ServiceName < result[j].ServiceName
+ })
+ if len(result) > maxServiceBreakdown {
+ result = result[:maxServiceBreakdown]
+ analysis.Truncated = true
+ }
+ return result
+}
+
+// slowSpanEntry pairs a slow span with the analyzer internal identity of the span
+// it was derived from. The position is used instead of the business span id so
+// that spans without an id, or with duplicated ids, cannot overwrite each other.
+type slowSpanEntry struct {
+ slow SlowSpan
+ position int
+ selfMicros int64
+}
+
+// buildSlowSpans ranks the spans by the time they consumed on their own.
+// The exclusive time is returned alongside the report so the summary can quote it
+// even for spans that are not part of the critical path.
+func buildSlowSpans(nodes []*analysisSpan, wallClock int64, criticalPath map[int]struct{}) []slowSpanEntry {
+ candidates := make([]slowSpanEntry, 0, len(nodes))
+ for _, node := range nodes {
+ span := node.span
+ _, onPath := criticalPath[node.position]
+ candidates = append(candidates, slowSpanEntry{
+ slow: SlowSpan{
+ SpanID: span.SpanID,
+ ServiceName: span.ServiceName,
+ OperationName: span.OperationName,
+ DurationMicros: span.DurationMicros,
+ SharePercent: percentOf(node.selfMicros, wallClock),
+ OnCriticalPath: onPath,
+ },
+ position: node.position,
+ selfMicros: node.selfMicros,
+ })
+ }
+ sort.SliceStable(candidates, func(i, j int) bool {
+ if candidates[i].selfMicros != candidates[j].selfMicros {
+ return candidates[i].selfMicros > candidates[j].selfMicros
+ }
+ return candidates[i].position < candidates[j].position
+ })
+ // The list is a ranking, so cutting it to the top entries does not lose any
+ // analysis data. The truncated flag stays untouched: it would tell the agent
+ // that the diagnosis is based on incomplete data, and a ranking is complete
+ // even when only its best entries are reported.
+ if len(candidates) > maxSlowSpans {
+ candidates = candidates[:maxSlowSpans]
+ }
+ return candidates
+}
+
+// errorSpanEntry pairs an error span with the analyzer node it was derived from.
+// The node is kept so the ordering can use the start time of the span without
+// indexing the table by the business span id.
+type errorSpanEntry struct {
+ span ErrorSpan
+ node *analysisSpan
+}
+
+// buildErrorSpans collects the failed spans and marks the root causes.
+func buildErrorSpans(nodes []*analysisSpan, analysis *TraceAnalysis) ([]ErrorSpan, int) {
+ failed := make([]*analysisSpan, 0, len(nodes))
+ for _, node := range nodes {
+ if spanFailed(node.span) {
+ failed = append(failed, node)
+ }
+ }
+ entries := make([]errorSpanEntry, 0, len(failed))
+ for _, node := range failed {
+ span := node.span
+ errorType, errorMessage := errorDetails(span)
+ entries = append(entries, errorSpanEntry{
+ span: ErrorSpan{
+ SpanID: span.SpanID,
+ ServiceName: span.ServiceName,
+ OperationName: span.OperationName,
+ ErrorType: errorType,
+ ErrorMessage: errorMessage,
+ Status: errorStatus(span),
+ RootCause: !hasFailedDescendant(node),
+ },
+ node: node,
+ })
+ }
+ // Root causes come first so the agent sees the deepest failure before the
+ // ancestors that only propagate it.
+ sort.SliceStable(entries, func(i, j int) bool {
+ left, right := entries[i], entries[j]
+ if left.span.RootCause != right.span.RootCause {
+ return left.span.RootCause
+ }
+ if left.node.startMicros != right.node.startMicros {
+ return left.node.startMicros < right.node.startMicros
+ }
+ return left.node.position < right.node.position
+ })
+ count := len(entries)
+ if len(entries) > maxErrorSpans {
+ entries = entries[:maxErrorSpans]
+ analysis.Truncated = true
+ }
+ result := make([]ErrorSpan, 0, len(entries))
+ for _, entry := range entries {
+ result = append(result, entry.span)
+ }
+ return result, count
+}
+
+// errorDetails resolves the error classification and the human readable description.
+//
+// The sources are ordered "generative AI first, standard second, non standard
+// last":
+// - The span attribute error.type stays the leading source because the OpenTelemetry
+// error registry defines it as the stable error type of a span.
+// - The generative AI exception event comes before the standard exception event
+// because this project serves the generative AI execution of the agent loop, so
+// the typed error of the model call is the more relevant classification.
+// - error.message is only the last resort because it is not an OpenTelemetry
+// standard attribute: the registry only defines error.type, and the stable
+// attribute for the exception text is exception.message. error.message is a
+// habit of the Java ecosystem and of some collectors, so it is used only when
+// no typed error is available, and it then degrades into the error type.
+func errorDetails(span *Span) (string, string) {
+ errorType := ""
+ if value, ok := lookupAttributeOK(span.Attributes, errorTypeAttribute); ok {
+ errorType = strings.TrimSpace(value)
+ }
+ exceptionType, exceptionMessage := exceptionDetails(span)
+ if errorType == "" {
+ errorType = exceptionType
+ }
+ errorMessage := exceptionMessage
+ if errorMessage == "" {
+ if value, ok := lookupAttributeOK(span.Attributes, errorMessageAttribute); ok {
+ errorMessage = strings.TrimSpace(value)
+ }
+ }
+ if errorType == "" && errorMessage != "" {
+ // Degraded classification: without a typed error the message is the only
+ // signal left, so it degrades into the error type.
+ errorType = errorMessage
+ }
+ if errorType == "" {
+ errorType = "unknown"
+ }
+ return truncateRunes(errorType, maxErrorTypeChars), truncateRunes(errorMessage, maxErrorMessageChars)
+}
+
+// exceptionDetails reads the exception type and message from the span events.
+// The generative AI exception event wins over the standard exception event,
+// because this project serves the generative AI execution of the agent loop and
+// the generative AI event carries the typed error of the model call. Collectors
+// also report the exception as a plain span attribute, so the attributes are read
+// as a fallback once the events have been exhausted.
+func exceptionDetails(span *Span) (string, string) {
+ exceptionType := ""
+ exceptionMessage := ""
+ collect := func(attributes map[string]string) {
+ if exceptionType == "" {
+ if value, ok := lookupAttributeOK(attributes, exceptionTypeAttribute); ok {
+ exceptionType = strings.TrimSpace(value)
+ }
+ }
+ if exceptionMessage == "" {
+ if value, ok := lookupAttributeOK(attributes, exceptionMessageAttribute); ok {
+ exceptionMessage = strings.TrimSpace(value)
+ }
+ }
+ }
+ for _, event := range span.Events {
+ if strings.EqualFold(eventName(event), genAIExceptionEvent) {
+ collect(event.Attributes)
+ }
+ }
+ for _, event := range span.Events {
+ if strings.EqualFold(eventName(event), standardExceptionEvent) {
+ collect(event.Attributes)
+ }
+ }
+ // The span attributes are the last resort: the two documented exception
+ // events stay the primary source, and no other event is consulted.
+ collect(span.Attributes)
+ return exceptionType, exceptionMessage
+}
+
+func eventName(event Event) string {
+ if value, ok := lookupAttributeOK(event.Attributes, spanEventNameAttribute); ok {
+ return value
+ }
+ return ""
+}
+
+// errorStatus returns the normalized status of a failed span.
+func errorStatus(span *Span) string {
+ if spanFailed(span) {
+ return "ERROR"
+ }
+ if span.Status == "" {
+ return "UNSET"
+ }
+ return span.Status
+}
+
+// hasFailedDescendant reports whether a failed span only propagates an error that
+// happened deeper in the trace.
+func hasFailedDescendant(node *analysisSpan) bool {
+ visited := map[int]struct{}{}
+ stack := make([]*analysisSpan, 0, len(node.children))
+ stack = append(stack, node.children...)
+ for len(stack) > 0 {
+ current := stack[len(stack)-1]
+ stack = stack[:len(stack)-1]
+ if _, seen := visited[current.position]; seen {
+ continue
+ }
+ visited[current.position] = struct{}{}
+ if spanFailed(current.span) {
+ return true
+ }
+ stack = append(stack, current.children...)
+ }
+ return false
+}
+
+// buildDependencyEdges derives the cross service call graph of the trace.
+func buildDependencyEdges(nodes []*analysisSpan, analysis *TraceAnalysis) []DependencyEdge {
+ aggregates := map[string]*DependencyEdge{}
+ for _, node := range nodes {
+ if node.parent == nil {
+ continue
+ }
+ caller := node.parent.span.ServiceName
+ callee := node.span.ServiceName
+ if caller == callee {
+ continue
+ }
+ key := caller + "\x00" + callee
+ edge, ok := aggregates[key]
+ if !ok {
+ edge = &DependencyEdge{CallerService: caller, CalleeService: callee}
+ aggregates[key] = edge
+ }
+ edge.SpanCount++
+ edge.TotalMicros += node.span.DurationMicros
+ if spanFailed(node.span) {
+ edge.ErrorCount++
+ }
+ }
+ result := make([]DependencyEdge, 0, len(aggregates))
+ for _, edge := range aggregates {
+ result = append(result, *edge)
+ }
+ sort.SliceStable(result, func(i, j int) bool {
+ if result[i].TotalMicros != result[j].TotalMicros {
+ return result[i].TotalMicros > result[j].TotalMicros
+ }
+ if result[i].CallerService != result[j].CallerService {
+ return result[i].CallerService < result[j].CallerService
+ }
+ return result[i].CalleeService < result[j].CalleeService
+ })
+ if len(result) > maxDependencyEdges {
+ result = result[:maxDependencyEdges]
+ analysis.Truncated = true
+ }
+ return result
+}
+
+// buildAnalysisSummary renders the one line conclusion of the diagnosis.
+func buildAnalysisSummary(analysis *TraceAnalysis, slowSpans []slowSpanEntry) string {
+ base := fmt.Sprintf("trace %d %s, %d %s, %s, %d %s",
+ analysis.ServiceCount, pluralize(analysis.ServiceCount, "service", "services"),
+ analysis.SpanCount, pluralize(analysis.SpanCount, "span", "spans"),
+ formatMicros(analysis.DurationMicros),
+ analysis.ErrorSpanCount, pluralize(analysis.ErrorSpanCount, "error", "errors"))
+
+ var slowPart string
+ if len(slowSpans) > 0 {
+ slowest := slowSpans[0]
+ slowPart = fmt.Sprintf("; slowest: %s/%s self %s of %s (%.0f%% of trace)",
+ sanitizeSummaryText(slowest.slow.ServiceName), sanitizeSummaryText(slowest.slow.OperationName),
+ formatMicros(slowest.selfMicros), formatMicros(slowest.slow.DurationMicros), slowest.slow.SharePercent)
+ }
+ var rootPart string
+ for _, span := range analysis.ErrorSpans {
+ if !span.RootCause {
+ continue
+ }
+ rootPart = fmt.Sprintf("; root cause: %s/%s %s %s",
+ sanitizeSummaryText(span.ServiceName), sanitizeSummaryText(span.OperationName),
+ span.Status, sanitizeSummaryText(span.ErrorType))
+ break
+ }
+ for _, candidate := range []string{base + slowPart + rootPart, base + rootPart, base} {
+ // The budget counts runes, not bytes: a Chinese service name uses three
+ // bytes per character and would otherwise push the diagnosis out of the
+ // summary even though it still fits.
+ if utf8.RuneCountInString(candidate) <= maxAnalysisSummaryChars {
+ return candidate
+ }
+ }
+ return truncateRunes(base+slowPart+rootPart, maxAnalysisSummaryChars)
+}
+
+// pluralize returns the singular or the plural form depending on the count.
+func pluralize(count int, singular, plural string) string {
+ if count == 1 {
+ return singular
+ }
+ return plural
+}
+
+// lookupAttributeOK resolves an attribute value, tolerating casing and padding
+// differences introduced by different collectors.
+func lookupAttributeOK(attributes map[string]string, name string) (string, bool) {
+ if value, ok := attributes[name]; ok {
+ return value, true
+ }
+ bestKey := ""
+ bestValue := ""
+ found := false
+ for key, value := range attributes {
+ if !strings.EqualFold(strings.TrimSpace(key), name) {
+ continue
+ }
+ if !found || key < bestKey {
+ bestKey, bestValue, found = key, value, true
+ }
+ }
+ return bestValue, found
+}
+
+// percentOf computes a share with two decimals. A zero denominator yields zero
+// instead of a division by zero.
+func percentOf(value, total int64) float64 {
+ if total <= 0 {
+ return 0
+ }
+ return math.Round(float64(value)/float64(total)*100*100) / 100
+}
+
+// formatMicros renders a microsecond duration in a compact human readable form.
+func formatMicros(micros int64) string {
+ switch {
+ case micros < 1000:
+ return strconv.FormatInt(micros, 10) + "us"
+ case micros < 1000*1000:
+ return trimDecimal(float64(micros)/1000, 1) + "ms"
+ default:
+ return trimDecimal(float64(micros)/(1000*1000), 2) + "s"
+ }
+}
+
+func trimDecimal(value float64, decimals int) string {
+ formatted := strconv.FormatFloat(value, 'f', decimals, 64)
+ if strings.Contains(formatted, ".") {
+ formatted = strings.TrimRight(formatted, "0")
+ formatted = strings.TrimSuffix(formatted, ".")
+ }
+ return formatted
+}
+
+// sanitizeSummaryText keeps the one line summary free of line breaks.
+func sanitizeSummaryText(value string) string {
+ replaced := strings.NewReplacer("\r", " ", "\n", " ", "\t", " ").Replace(value)
+ return strings.TrimSpace(replaced)
+}
+
+// truncateRunes limits a string by rune count so multi byte characters stay intact.
+func truncateRunes(value string, limit int) string {
+ if limit <= 0 {
+ return ""
+ }
+ runes := []rune(value)
+ if len(runes) <= limit {
+ return value
+ }
+ return string(runes[:limit])
+}
+
+func maxInt(a, b int) int {
+ if a > b {
+ return a
+ }
+ return b
+}
+
+func maxInt64(a, b int64) int64 {
+ if a > b {
+ return a
+ }
+ return b
+}
+
+func minInt64(a, b int64) int64 {
+ if a < b {
+ return a
+ }
+ return b
+}
+
+// traceRequestContext derives the outbound request context from the console context.
+func traceRequestContext(ctx consolectx.Context) context.Context {
+ if ctx == nil || ctx.AppContext() == nil {
+ return context.Background()
+ }
+ return ctx.AppContext()
+}
diff --git a/pkg/mcp/tools/trace/analyzer_test.go b/pkg/mcp/tools/trace/analyzer_test.go
new file mode 100644
index 000000000..0a872141f
--- /dev/null
+++ b/pkg/mcp/tools/trace/analyzer_test.go
@@ -0,0 +1,841 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package trace
+
+import (
+ "fmt"
+ "strings"
+ "testing"
+ "time"
+ "unicode/utf8"
+)
+
+// analysisBase is the reference point of the test timeline.
+const analysisBase = "2026-10-24T10:00:00.000000000Z"
+
+func spanAt(offsetMicros int64, spanID, parentID, service, operation string, durationMicros int64) Span {
+ return Span{
+ SpanID: spanID,
+ ParentSpanID: parentID,
+ ServiceName: service,
+ OperationName: operation,
+ StartTime: shiftTime(analysisBase, offsetMicros),
+ DurationMicros: durationMicros,
+ }
+}
+
+func shiftTime(base string, offsetMicros int64) string {
+ parsed, err := time.Parse(time.RFC3339Nano, base)
+ if err != nil {
+ panic(err)
+ }
+ return parsed.Add(time.Duration(offsetMicros) * time.Microsecond).UTC().Format(time.RFC3339Nano)
+}
+
+func TestBuildTraceAnalysisLinearChain(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "GET /checkout", 100000),
+ spanAt(10000, "02", "01", "order-service", "checkout", 80000),
+ spanAt(20000, "03", "02", "payment-service", "charge", 50000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.TraceID != testTraceID {
+ t.Fatalf("unexpected trace id: %s", analysis.TraceID)
+ }
+ if analysis.SpanCount != 3 || analysis.ServiceCount != 3 {
+ t.Fatalf("unexpected counters: %#v", analysis)
+ }
+ if analysis.DurationMicros != 100000 {
+ t.Fatalf("unexpected duration: %d", analysis.DurationMicros)
+ }
+ if len(analysis.CriticalPath) != 3 {
+ t.Fatalf("unexpected critical path: %#v", analysis.CriticalPath)
+ }
+ if analysis.CriticalPath[0].SpanID != "01" || analysis.CriticalPath[2].SpanID != "03" {
+ t.Fatalf("unexpected critical path order: %#v", analysis.CriticalPath)
+ }
+ // The root waits for the child for 80ms, the child waits for its own child for 50ms.
+ if analysis.CriticalPath[0].SelfMicros != 20000 || analysis.CriticalPath[0].WaitMicros != 80000 {
+ t.Fatalf("unexpected root self time: %#v", analysis.CriticalPath[0])
+ }
+ if analysis.CriticalPath[1].SelfMicros != 30000 || analysis.CriticalPath[1].WaitMicros != 50000 {
+ t.Fatalf("unexpected child self time: %#v", analysis.CriticalPath[1])
+ }
+ if analysis.CriticalPath[2].SelfMicros != 50000 || analysis.CriticalPath[2].WaitMicros != 0 {
+ t.Fatalf("unexpected leaf self time: %#v", analysis.CriticalPath[2])
+ }
+}
+
+func TestBuildTraceAnalysisCriticalPathFollowsSlowBranch(t *testing.T) {
+ // The root fans out into a fast branch and a slow branch. The critical path has
+ // to follow the branch that finishes last.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "GET /checkout", 200000),
+ spanAt(1000, "02", "01", "order-service", "fast", 10000),
+ spanAt(2000, "03", "01", "payment-service", "slow", 150000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if len(analysis.CriticalPath) != 2 {
+ t.Fatalf("unexpected critical path: %#v", analysis.CriticalPath)
+ }
+ if analysis.CriticalPath[1].SpanID != "03" || analysis.CriticalPath[1].ServiceName != "payment-service" {
+ t.Fatalf("expected the critical path to follow the slow branch: %#v", analysis.CriticalPath)
+ }
+}
+
+func TestBuildTraceAnalysisCriticalPathPrefersLatestRoot(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "early", 10000),
+ spanAt(5000, "02", "", "frontend", "late", 60000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if len(analysis.CriticalPath) != 1 || analysis.CriticalPath[0].SpanID != "02" {
+ t.Fatalf("expected the latest root to start the critical path: %#v", analysis.CriticalPath)
+ }
+}
+
+func TestBuildTraceAnalysisSelfTimeUsesUnionOfChildren(t *testing.T) {
+ // Two concurrent children overlap, so the covered time is the union and not the
+ // sum of both durations.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ spanAt(10000, "02", "01", "order-service", "a", 50000),
+ spanAt(20000, "03", "01", "order-service", "b", 50000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.CriticalPath[0].WaitMicros != 60000 {
+ t.Fatalf("expected the union of the concurrent children, got %d", analysis.CriticalPath[0].WaitMicros)
+ }
+ if analysis.CriticalPath[0].SelfMicros != 40000 {
+ t.Fatalf("unexpected self time: %d", analysis.CriticalPath[0].SelfMicros)
+ }
+}
+
+func TestBuildTraceAnalysisSelfTimeClipsChildrenToParent(t *testing.T) {
+ // The child reaches beyond the parent, so only the overlap counts.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ spanAt(50000, "02", "01", "order-service", "child", 900000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.CriticalPath[0].WaitMicros != 50000 {
+ t.Fatalf("expected the child to be clipped, got %d", analysis.CriticalPath[0].WaitMicros)
+ }
+ if analysis.CriticalPath[0].SelfMicros != 50000 {
+ t.Fatalf("unexpected self time: %d", analysis.CriticalPath[0].SelfMicros)
+ }
+}
+
+func TestBuildTraceAnalysisServiceBreakdown(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ spanAt(10000, "02", "01", "order-service", "a", 60000),
+ spanAt(20000, "03", "01", "order-service", "b", 30000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if len(analysis.ServiceBreakdown) != 2 {
+ t.Fatalf("unexpected breakdown: %#v", analysis.ServiceBreakdown)
+ }
+ order := analysis.ServiceBreakdown[0]
+ if order.ServiceName != "frontend" || order.TotalMicros != 100000 || order.SpanCount != 1 {
+ t.Fatalf("unexpected breakdown entry: %#v", order)
+ }
+ if order.MaxMicros != 100000 || order.AvgMicros != 100000 {
+ t.Fatalf("unexpected breakdown statistics: %#v", order)
+ }
+ if order.SharePercent != 100 {
+ t.Fatalf("unexpected share: %v", order.SharePercent)
+ }
+ second := analysis.ServiceBreakdown[1]
+ if second.ServiceName != "order-service" || second.TotalMicros != 90000 || second.SpanCount != 2 {
+ t.Fatalf("unexpected breakdown entry: %#v", second)
+ }
+ if second.MaxMicros != 60000 || second.AvgMicros != 45000 {
+ t.Fatalf("unexpected breakdown statistics: %#v", second)
+ }
+ // The service workload is 90ms against a 100ms wall clock, concurrent spans may
+ // push the share above 100 percent but never below the real ratio.
+ if second.SharePercent != 90 {
+ t.Fatalf("unexpected share: %v", second.SharePercent)
+ }
+}
+
+func TestBuildTraceAnalysisServiceBreakdownSortsByTotalTime(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ spanAt(10000, "02", "01", "alpha-service", "a", 50000),
+ spanAt(20000, "03", "01", "beta-service", "b", 50000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.ServiceBreakdown[1].ServiceName != "alpha-service" {
+ t.Fatalf("expected equal totals to be sorted by service name: %#v", analysis.ServiceBreakdown)
+ }
+}
+
+func TestBuildTraceAnalysisSlowSpansUseSelfTime(t *testing.T) {
+ // The early branch consumes the most time on its own, but it is not the branch
+ // that finishes last, so it is not part of the critical path.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 500000),
+ spanAt(0, "02", "01", "order-service", "early-branch", 400000),
+ spanAt(100000, "03", "01", "payment-service", "late-branch", 380000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if len(analysis.SlowSpans) != 3 {
+ t.Fatalf("unexpected slow spans: %#v", analysis.SlowSpans)
+ }
+ if analysis.SlowSpans[0].SpanID != "02" {
+ t.Fatalf("expected the branch with the largest self time to rank first: %#v", analysis.SlowSpans)
+ }
+ if analysis.SlowSpans[0].DurationMicros != 400000 {
+ t.Fatalf("expected the total duration in the slow span: %#v", analysis.SlowSpans[0])
+ }
+ if analysis.SlowSpans[0].SharePercent != 80 {
+ t.Fatalf("unexpected share: %v", analysis.SlowSpans[0].SharePercent)
+ }
+ if analysis.SlowSpans[0].OnCriticalPath {
+ t.Fatal("did not expect the early branch to be part of the critical path")
+ }
+ if !analysis.SlowSpans[1].OnCriticalPath {
+ t.Fatal("expected the late branch to be part of the critical path")
+ }
+ if len(analysis.CriticalPath) != 2 || analysis.CriticalPath[1].SpanID != "03" {
+ t.Fatalf("unexpected critical path: %#v", analysis.CriticalPath)
+ }
+}
+
+func TestBuildTraceAnalysisDependencyEdges(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ spanAt(10000, "02", "01", "order-service", "a", 40000),
+ spanAt(20000, "03", "01", "order-service", "b", 30000),
+ spanAt(30000, "04", "02", "payment-service", "charge", 20000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if len(analysis.Dependencies) != 2 {
+ t.Fatalf("unexpected dependencies: %#v", analysis.Dependencies)
+ }
+ edges := map[string]DependencyEdge{}
+ for _, edge := range analysis.Dependencies {
+ edges[edge.CallerService+"->"+edge.CalleeService] = edge
+ }
+ frontend := edges["frontend->order-service"]
+ if frontend.SpanCount != 2 || frontend.TotalMicros != 70000 || frontend.ErrorCount != 0 {
+ t.Fatalf("unexpected edge: %#v", frontend)
+ }
+ payment := edges["order-service->payment-service"]
+ if payment.SpanCount != 1 || payment.TotalMicros != 20000 {
+ t.Fatalf("unexpected edge: %#v", payment)
+ }
+ // The edge with the largest total time comes first.
+ if analysis.Dependencies[0].CallerService != "frontend" {
+ t.Fatalf("unexpected dependency order: %#v", analysis.Dependencies)
+ }
+}
+
+func TestBuildTraceAnalysisErrorAttributionAndRootCause(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ spanAt(10000, "02", "01", "order-service", "checkout", 80000),
+ spanAt(20000, "03", "02", "payment-service", "charge", 50000),
+ },
+ }
+ trace.Spans[0].Error = true
+ trace.Spans[0].Status = "ERROR"
+ trace.Spans[0].Attributes = map[string]string{"error.type": "_OTHER"}
+ trace.Spans[1].Error = true
+ trace.Spans[1].Status = "ERROR"
+ trace.Spans[1].Attributes = map[string]string{"error.type": "timeout", "error.message": "payment gateway timeout"}
+ trace.Spans[2].Error = true
+ trace.Spans[2].Status = "ERROR"
+
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.ErrorSpanCount != 3 {
+ t.Fatalf("unexpected error count: %d", analysis.ErrorSpanCount)
+ }
+ if len(analysis.ErrorSpans) != 3 {
+ t.Fatalf("unexpected error spans: %#v", analysis.ErrorSpans)
+ }
+ // The deepest failure is the root cause, the ancestors only propagate it.
+ if !analysis.ErrorSpans[0].RootCause || analysis.ErrorSpans[0].ServiceName != "payment-service" {
+ t.Fatalf("expected the deepest failure to be the root cause: %#v", analysis.ErrorSpans)
+ }
+ for _, span := range analysis.ErrorSpans[1:] {
+ if span.RootCause {
+ t.Fatalf("expected the propagated failures to not be root causes: %#v", analysis.ErrorSpans)
+ }
+ }
+ byService := map[string]ErrorSpan{}
+ for _, span := range analysis.ErrorSpans {
+ byService[span.ServiceName] = span
+ }
+ order := byService["order-service"]
+ if order.ErrorType != "timeout" {
+ t.Fatalf("unexpected error type: %#v", order)
+ }
+ if order.ErrorMessage != "payment gateway timeout" {
+ t.Fatalf("unexpected error message: %#v", order)
+ }
+ if byService["frontend"].ErrorType != "_OTHER" {
+ t.Fatalf("unexpected error type: %#v", byService["frontend"])
+ }
+ if byService["payment-service"].ErrorType != "unknown" {
+ t.Fatalf("expected a fallback error type: %#v", byService["payment-service"])
+ }
+}
+
+func TestBuildTraceAnalysisReadsExceptionEvent(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "payment-service", "charge", 50000),
+ },
+ }
+ trace.Spans[0].Error = true
+ trace.Spans[0].Status = "ERROR"
+ trace.Spans[0].Events = []Event{{
+ Timestamp: analysisBase,
+ Attributes: map[string]string{
+ "event": "exception",
+ "exception.type": "java.net.SocketTimeoutException",
+ "exception.message": "Read timed out",
+ },
+ }}
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.ErrorSpans[0].ErrorType != "java.net.SocketTimeoutException" {
+ t.Fatalf("unexpected error type: %#v", analysis.ErrorSpans[0])
+ }
+ if analysis.ErrorSpans[0].ErrorMessage != "Read timed out" {
+ t.Fatalf("unexpected error message: %#v", analysis.ErrorSpans[0])
+ }
+}
+
+func TestBuildTraceAnalysisPrefersGenAIExceptionEventOverErrorMessage(t *testing.T) {
+ // error.message is not an OpenTelemetry standard attribute, so the typed
+ // exception of the generative AI event has to win over it.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{spanAt(0, "01", "", "dubbo-admin-sre-agent", "chat gpt-4o", 50000)},
+ }
+ trace.Spans[0].Error = true
+ trace.Spans[0].Status = "ERROR"
+ trace.Spans[0].Attributes = map[string]string{"error.message": "upstream timeout"}
+ trace.Spans[0].Events = []Event{{
+ Timestamp: analysisBase,
+ Attributes: map[string]string{
+ "event": "gen_ai.client.operation.exception",
+ "exception.type": "openai.APIConnectionError",
+ "exception.message": "connection reset by peer",
+ },
+ }}
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.ErrorSpans[0].ErrorMessage != "connection reset by peer" {
+ t.Fatalf("expected the generative AI exception message to win: %#v", analysis.ErrorSpans[0])
+ }
+ if analysis.ErrorSpans[0].ErrorType != "openai.APIConnectionError" {
+ t.Fatalf("unexpected error type: %#v", analysis.ErrorSpans[0])
+ }
+}
+
+func TestBuildTraceAnalysisPrefersGenAIExceptionEventOverStandardExceptionEvent(t *testing.T) {
+ // Both events describe the same failure. The generative AI one wins because
+ // this project serves the generative AI execution of the agent loop.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{spanAt(0, "01", "", "dubbo-admin-sre-agent", "chat gpt-4o", 50000)},
+ }
+ trace.Spans[0].Error = true
+ trace.Spans[0].Status = "ERROR"
+ trace.Spans[0].Events = []Event{
+ {
+ Timestamp: analysisBase,
+ Attributes: map[string]string{
+ "event": "exception",
+ "exception.type": "java.lang.IllegalStateException",
+ "exception.message": "state mismatch",
+ },
+ },
+ {
+ Timestamp: analysisBase,
+ Attributes: map[string]string{
+ "event": "gen_ai.client.operation.exception",
+ "exception.type": "openai.RateLimitError",
+ "exception.message": "rate limit exceeded",
+ },
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.ErrorSpans[0].ErrorType != "openai.RateLimitError" {
+ t.Fatalf("expected the generative AI exception type to win: %#v", analysis.ErrorSpans[0])
+ }
+ if analysis.ErrorSpans[0].ErrorMessage != "rate limit exceeded" {
+ t.Fatalf("expected the generative AI exception message to win: %#v", analysis.ErrorSpans[0])
+ }
+}
+
+func TestBuildTraceAnalysisFallsBackToErrorMessageAttribute(t *testing.T) {
+ // error.message is the only signal left, so it provides the message and
+ // degrades into the error type as well.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{spanAt(0, "01", "", "payment-service", "charge", 50000)},
+ }
+ trace.Spans[0].Error = true
+ trace.Spans[0].Status = "ERROR"
+ trace.Spans[0].Attributes = map[string]string{"error.message": "payment gateway timeout"}
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.ErrorSpans[0].ErrorMessage != "payment gateway timeout" {
+ t.Fatalf("unexpected error message: %#v", analysis.ErrorSpans[0])
+ }
+ if analysis.ErrorSpans[0].ErrorType != "payment gateway timeout" {
+ t.Fatalf("expected the message to degrade into the error type: %#v", analysis.ErrorSpans[0])
+ }
+}
+
+func TestBuildTraceAnalysisCountsErrorsBeyondListLimit(t *testing.T) {
+ spans := make([]Span, 0, maxErrorSpans+10)
+ for i := 0; i < maxErrorSpans+10; i++ {
+ span := spanAt(int64(i)*1000, fmt.Sprintf("%02d", i), "", "payment-service", "charge", 1000)
+ span.Error = true
+ span.Status = "ERROR"
+ spans = append(spans, span)
+ }
+ analysis := BuildTraceAnalysis(&Trace{TraceID: testTraceID, Spans: spans})
+ if len(analysis.ErrorSpans) != maxErrorSpans {
+ t.Fatalf("expected the list to be capped, got %d", len(analysis.ErrorSpans))
+ }
+ if analysis.ErrorSpanCount != maxErrorSpans+10 {
+ t.Fatalf("expected the real error count, got %d", analysis.ErrorSpanCount)
+ }
+ if !analysis.Truncated {
+ t.Fatal("expected the analysis to be marked as truncated")
+ }
+}
+
+func TestBuildTraceAnalysisHandlesOrphanSpans(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ spanAt(10000, "02", "missing-parent", "order-service", "orphan", 40000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.SpanCount != 2 || analysis.ServiceCount != 2 {
+ t.Fatalf("unexpected counters: %#v", analysis)
+ }
+ // The orphan still contributes to the service breakdown.
+ found := false
+ for _, entry := range analysis.ServiceBreakdown {
+ if entry.ServiceName == "order-service" && entry.SpanCount == 1 {
+ found = true
+ }
+ }
+ if !found {
+ t.Fatalf("expected the orphan in the breakdown: %#v", analysis.ServiceBreakdown)
+ }
+}
+
+func TestBuildTraceAnalysisSurvivesCyclicReferences(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "02", "frontend", "a", 100000),
+ spanAt(1000, "02", "01", "order-service", "b", 50000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if len(analysis.CriticalPath) > 2 {
+ t.Fatalf("expected the cycle to be broken, got %#v", analysis.CriticalPath)
+ }
+ if analysis.SpanCount != 2 {
+ t.Fatalf("unexpected span count: %d", analysis.SpanCount)
+ }
+}
+
+func TestBuildTraceAnalysisSurvivesMalformedStartTime(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ {SpanID: "02", ParentSpanID: "01", ServiceName: "order-service", OperationName: "broken", StartTime: "not-a-timestamp", DurationMicros: 5000},
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.SpanCount != 2 {
+ t.Fatalf("unexpected span count: %d", analysis.SpanCount)
+ }
+ if !analysis.Truncated {
+ t.Fatal("expected the analysis to be marked as truncated")
+ }
+}
+
+func TestBuildTraceAnalysisSurvivesSelfReference(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "01", "frontend", "self", 100000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.SpanCount != 1 {
+ t.Fatalf("unexpected span count: %d", analysis.SpanCount)
+ }
+}
+
+func TestBuildTraceAnalysisSummary(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "order-service", "checkout", 250000),
+ spanAt(10000, "02", "01", "payment-service", "charge", 200000),
+ },
+ }
+ trace.Spans[1].Error = true
+ trace.Spans[1].Status = "ERROR"
+ trace.Spans[1].Attributes = map[string]string{"error.type": "timeout"}
+ analysis := BuildTraceAnalysis(trace)
+ if !strings.HasPrefix(analysis.Summary, "trace 2 services, 2 spans, 250ms, 1 error") {
+ t.Fatalf("unexpected summary: %q", analysis.Summary)
+ }
+ // The charge span consumes 200ms on its own, the checkout span only 50ms.
+ if !strings.Contains(analysis.Summary, "slowest: payment-service/charge self 200ms of 200ms (80% of trace)") {
+ t.Fatalf("unexpected summary: %q", analysis.Summary)
+ }
+ if !strings.Contains(analysis.Summary, "root cause: payment-service/charge ERROR timeout") {
+ t.Fatalf("unexpected summary: %q", analysis.Summary)
+ }
+ if utf8.RuneCountInString(analysis.Summary) > maxAnalysisSummaryChars {
+ t.Fatalf("expected a bounded summary, got %d characters", utf8.RuneCountInString(analysis.Summary))
+ }
+ if strings.ContainsAny(analysis.Summary, "\r\n") {
+ t.Fatalf("expected a single line summary, got %q", analysis.Summary)
+ }
+}
+
+func TestBuildTraceAnalysisSummaryStaysBounded(t *testing.T) {
+ spans := []Span{
+ spanAt(0, "01", "", strings.Repeat("very-long-service-name-", 8), strings.Repeat("very-long-operation-name-", 8), 1000),
+ }
+ spans[0].Error = true
+ spans[0].Status = "ERROR"
+ analysis := BuildTraceAnalysis(&Trace{TraceID: testTraceID, Spans: spans})
+ if utf8.RuneCountInString(analysis.Summary) > maxAnalysisSummaryChars {
+ t.Fatalf("expected a bounded summary, got %d characters", utf8.RuneCountInString(analysis.Summary))
+ }
+}
+
+func TestBuildTraceAnalysisHandlesEmptyTrace(t *testing.T) {
+ analysis := BuildTraceAnalysis(&Trace{TraceID: testTraceID})
+ if analysis.SpanCount != 0 || analysis.ServiceCount != 0 {
+ t.Fatalf("unexpected counters: %#v", analysis)
+ }
+ if analysis.Summary == "" {
+ t.Fatal("expected a summary even for an empty trace")
+ }
+}
+
+func TestAnalyzePropertiesExposeRequiredParameters(t *testing.T) {
+ properties := AnalyzeProperties()
+ for _, name := range []string{"traceId", "provider"} {
+ if _, ok := properties[name]; !ok {
+ t.Fatalf("analyze properties are missing %q", name)
+ }
+ }
+}
+
+// criticalPathChain builds a linear chain of spans for the critical path budget.
+func criticalPathChain(hops int) []*analysisSpan {
+ spans := make([]Span, 0, hops)
+ for i := 0; i < hops; i++ {
+ parentID := ""
+ if i > 0 {
+ parentID = fmt.Sprintf("%02d", i-1)
+ }
+ spans = append(spans, spanAt(int64(i)*1000, fmt.Sprintf("%02d", i), parentID, "order-service", "hop", 10000))
+ }
+ return buildAnalysisTree(spans)
+}
+
+func TestBuildCriticalPathStopsAtBudgetWithoutTruncation(t *testing.T) {
+ // A chain that ends exactly at the budget does not lose any hop, so the
+ // analysis must not be reported as truncated.
+ analysis := &TraceAnalysis{}
+ entries, _ := buildCriticalPath(criticalPathChain(maxCriticalPath), analysis)
+ if len(entries) != maxCriticalPath {
+ t.Fatalf("unexpected critical path length: %d", len(entries))
+ }
+ if analysis.Truncated {
+ t.Fatal("did not expect the analysis to be truncated when the chain ends at the budget")
+ }
+
+ // One hop more than the budget does lose a hop.
+ analysis = &TraceAnalysis{}
+ entries, _ = buildCriticalPath(criticalPathChain(maxCriticalPath+1), analysis)
+ if len(entries) != maxCriticalPath {
+ t.Fatalf("unexpected critical path length: %d", len(entries))
+ }
+ if !analysis.Truncated {
+ t.Fatal("expected the analysis to be truncated when the chain exceeds the budget")
+ }
+}
+
+func TestBuildTraceAnalysisUsesProviderEnvelopeForClippedSpans(t *testing.T) {
+ // The provider computes the duration over every raw span and only then clips
+ // the span list. The analysis has to use the provider envelope as the
+ // latency denominator, otherwise the reported duration is shorter than the
+ // one the search reported and every latency percentage is inflated.
+ trace := &Trace{
+ TraceID: testTraceID,
+ DurationMicros: 500000,
+ Truncated: true,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "GET /checkout", 100000),
+ spanAt(10000, "02", "01", "order-service", "checkout", 80000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.DurationMicros != 500000 {
+ t.Fatalf("duration = %d, want %d", analysis.DurationMicros, 500000)
+ }
+ // The service breakdown divides by the envelope, so the frontend service
+ // covers a fifth of the whole trace and not of the retained spans only.
+ var frontend *ServiceLatency
+ for _, entry := range analysis.ServiceBreakdown {
+ if entry.ServiceName == "frontend" {
+ frontend = &entry
+ break
+ }
+ }
+ if frontend == nil {
+ t.Fatalf("expected a frontend entry in the breakdown: %#v", analysis.ServiceBreakdown)
+ }
+ if frontend.SharePercent != 20 {
+ t.Fatalf("frontend share = %v, want 20", frontend.SharePercent)
+ }
+}
+
+func TestBuildTraceAnalysisDoesNotReportTruncationForTheSlowSpanRanking(t *testing.T) {
+ // The slow span list is a ranking: cutting it to the best entries does not
+ // lose any analysis data. The truncated flag would tell the agent that the
+ // diagnosis is based on incomplete data, so it must stay unset.
+ spans := make([]Span, 0, maxSlowSpans+5)
+ for i := 0; i < maxSlowSpans+5; i++ {
+ spans = append(spans, spanAt(int64(i)*1000, fmt.Sprintf("%02d", i), "", "frontend", "GET /checkout", 10000))
+ }
+ trace := &Trace{TraceID: testTraceID, Spans: spans}
+ analysis := BuildTraceAnalysis(trace)
+ if len(analysis.SlowSpans) != maxSlowSpans {
+ t.Fatalf("slow spans = %d, want %d", len(analysis.SlowSpans), maxSlowSpans)
+ }
+ if analysis.Truncated {
+ t.Fatal("did not expect the analysis to be truncated by the slow span ranking")
+ }
+}
+
+func TestBuildTraceAnalysisHandlesSpansWithoutSpanID(t *testing.T) {
+ // Both spans share an empty span id. An index keyed by the business id would
+ // let one span overwrite the other and mislabel the critical path.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ {SpanID: "", ParentSpanID: "", ServiceName: "frontend", OperationName: "root", StartTime: analysisBase, DurationMicros: 100000},
+ {SpanID: "", ParentSpanID: "", ServiceName: "order-service", OperationName: "orphan", StartTime: shiftTime(analysisBase, 10000), DurationMicros: 50000},
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ // The critical path starts at the root that finishes last.
+ if len(analysis.CriticalPath) != 1 || analysis.CriticalPath[0].ServiceName != "frontend" {
+ t.Fatalf("unexpected critical path: %#v", analysis.CriticalPath)
+ }
+ // Only the span on the critical path may be marked as such.
+ onPath := 0
+ for _, slow := range analysis.SlowSpans {
+ if !slow.OnCriticalPath {
+ continue
+ }
+ onPath++
+ if slow.ServiceName != "frontend" {
+ t.Fatalf("expected only the root to be on the critical path: %#v", analysis.SlowSpans)
+ }
+ }
+ if onPath != 1 {
+ t.Fatalf("expected exactly one span on the critical path: %#v", analysis.SlowSpans)
+ }
+ // The summary must quote the self time of the slowest span and not the one of
+ // another span that happens to share the same identifier.
+ if !strings.Contains(analysis.Summary, "slowest: frontend/root self 100ms of 100ms") {
+ t.Fatalf("unexpected summary: %q", analysis.Summary)
+ }
+}
+
+func TestBuildTraceAnalysisSkipsSelfTimeForUnparsableStartTime(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ {SpanID: "02", ParentSpanID: "01", ServiceName: "order-service", OperationName: "broken", StartTime: "not-a-timestamp", DurationMicros: 50000},
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if !analysis.Truncated {
+ t.Fatal("expected the analysis to be marked as truncated")
+ }
+ // The span without a timestamp must not claim a time of its own, otherwise
+ // the parent would account for the same 50ms a second time.
+ for _, slow := range analysis.SlowSpans {
+ if slow.SpanID == "02" && slow.SharePercent != 0 {
+ t.Fatalf("expected no self time for the span without a timestamp: %#v", slow)
+ }
+ }
+ if analysis.CriticalPath[0].SelfMicros != 100000 {
+ t.Fatalf("unexpected self time of the root: %#v", analysis.CriticalPath[0])
+ }
+}
+
+func TestBuildTraceAnalysisReadsExceptionSpanAttributes(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{spanAt(0, "01", "", "payment-service", "charge", 50000)},
+ }
+ trace.Spans[0].Error = true
+ trace.Spans[0].Status = "ERROR"
+ trace.Spans[0].Attributes = map[string]string{
+ "exception.type": "java.lang.IllegalStateException",
+ "exception.message": "state mismatch",
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.ErrorSpans[0].ErrorType != "java.lang.IllegalStateException" {
+ t.Fatalf("unexpected error type: %#v", analysis.ErrorSpans[0])
+ }
+ if analysis.ErrorSpans[0].ErrorMessage != "state mismatch" {
+ t.Fatalf("unexpected error message: %#v", analysis.ErrorSpans[0])
+ }
+}
+
+func TestBuildTraceAnalysisPrefersExceptionEventOverSpanAttributes(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{spanAt(0, "01", "", "payment-service", "charge", 50000)},
+ }
+ trace.Spans[0].Error = true
+ trace.Spans[0].Status = "ERROR"
+ trace.Spans[0].Attributes = map[string]string{
+ "exception.type": "java.lang.IllegalStateException",
+ "exception.message": "state mismatch",
+ }
+ trace.Spans[0].Events = []Event{{
+ Timestamp: analysisBase,
+ Attributes: map[string]string{
+ "event": "exception",
+ "exception.type": "java.net.SocketTimeoutException",
+ "exception.message": "Read timed out",
+ },
+ }}
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.ErrorSpans[0].ErrorType != "java.net.SocketTimeoutException" {
+ t.Fatalf("expected the event to win over the span attribute: %#v", analysis.ErrorSpans[0])
+ }
+ if analysis.ErrorSpans[0].ErrorMessage != "Read timed out" {
+ t.Fatalf("expected the event to win over the span attribute: %#v", analysis.ErrorSpans[0])
+ }
+}
+
+func TestBuildTraceAnalysisSummaryCountsRunes(t *testing.T) {
+ // A Chinese name uses three bytes per character, so a byte budget would drop
+ // the diagnosis even though it still fits into the character budget.
+ longName := strings.Repeat("服", 60)
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", longName, longName, 100000),
+ spanAt(10000, "02", "01", "order-service", "charge", 50000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if !strings.Contains(analysis.Summary, "slowest:") {
+ t.Fatalf("expected the slowest span in the summary: %q", analysis.Summary)
+ }
+ if utf8.RuneCountInString(analysis.Summary) > maxAnalysisSummaryChars {
+ t.Fatalf("expected a bounded summary, got %d characters", utf8.RuneCountInString(analysis.Summary))
+ }
+}
+
+func TestBuildTraceAnalysisSummaryUsesSingularForms(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{spanAt(0, "01", "", "frontend", "root", 100000)},
+ }
+ analysis := BuildTraceAnalysis(trace)
+ if !strings.HasPrefix(analysis.Summary, "trace 1 service, 1 span, 100ms, 0 errors") {
+ t.Fatalf("unexpected summary: %q", analysis.Summary)
+ }
+}
+
+func TestBuildTraceAnalysisServiceShareCanExceedHundred(t *testing.T) {
+ // Two concurrent children each cover the whole window of the parent, so the
+ // service workload is twice the wall clock of the trace. The share is a
+ // workload ratio and not a time slice, so it may exceed 100 percent.
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "frontend", "root", 100000),
+ spanAt(0, "02", "01", "order-service", "a", 100000),
+ spanAt(0, "03", "01", "order-service", "b", 100000),
+ },
+ }
+ analysis := BuildTraceAnalysis(trace)
+ share := 0.0
+ for _, entry := range analysis.ServiceBreakdown {
+ if entry.ServiceName == "order-service" {
+ share = entry.SharePercent
+ }
+ }
+ if share <= 100 {
+ t.Fatalf("expected the concurrent workload to exceed 100 percent, got %v", share)
+ }
+}
diff --git a/pkg/mcp/tools/trace/genai.go b/pkg/mcp/tools/trace/genai.go
new file mode 100644
index 000000000..742593635
--- /dev/null
+++ b/pkg/mcp/tools/trace/genai.go
@@ -0,0 +1,423 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package trace
+
+import (
+ "math"
+ "sort"
+ "strconv"
+ "strings"
+ "time"
+)
+
+// The generative AI semantic conventions are still in development, so the analyzer pins
+// the recognized attribute set instead of trusting an open-ended namespace.
+// Reference: OpenTelemetry semantic conventions for generative AI systems, commit 6fd0d76.
+const genAIPrefix = "gen_ai."
+
+// Recognized generative AI attributes.
+const (
+ genAIOperationName = "gen_ai.operation.name"
+ genAIProviderName = "gen_ai.provider.name"
+ genAIRequestModel = "gen_ai.request.model"
+ genAIResponseModel = "gen_ai.response.model"
+ genAIConversationID = "gen_ai.conversation.id"
+ genAIAgentName = "gen_ai.agent.name"
+ genAIAgentID = "gen_ai.agent.id"
+ genAIAgentDescription = "gen_ai.agent.description"
+ genAIAgentVersion = "gen_ai.agent.version"
+ genAIToolName = "gen_ai.tool.name"
+ genAIToolCallID = "gen_ai.tool.call.id"
+ genAIToolCallArguments = "gen_ai.tool.call.arguments"
+ genAIToolCallResult = "gen_ai.tool.call.result"
+ genAIInputMessages = "gen_ai.input.messages"
+ genAIOutputMessages = "gen_ai.output.messages"
+ genAIUsageInputTokens = "gen_ai.usage.input_tokens"
+ genAIUsageOutputTokens = "gen_ai.usage.output_tokens"
+ genAIUsageReasoningTokens = "gen_ai.usage.reasoning.output_tokens"
+ genAIUsageCacheReadTokens = "gen_ai.usage.cache_read.input_tokens"
+ genAIUsageCacheWriteTokens = "gen_ai.usage.cache_write.input_tokens"
+ genAIResponseFinishReasons = "gen_ai.response.finish_reasons"
+ genAIResponseID = "gen_ai.response.id"
+ genAIDataSourceID = "gen_ai.data_source.id"
+ genAIOutputType = "gen_ai.output.type"
+)
+
+// genAIUsageTotalTokens is not part of the semantic conventions, the conventions
+// only define the input and the output token counters. It is honored when a
+// provider emits it anyway, because some vendors already report the aggregate.
+// The value stays a reference value in the detail of a single call, the aggregate
+// of the trace is always the sum of the input and the output tokens.
+const genAIUsageTotalTokens = "gen_ai.usage.total_tokens"
+
+// genAIContentAttributes hold prompts, completions and tool payloads. They are Opt-In
+// in the semantic conventions and may embed PII, so their values are never read.
+var genAIContentAttributes = map[string]struct{}{
+ genAIInputMessages: {},
+ genAIOutputMessages: {},
+ genAIToolCallArguments: {},
+ genAIToolCallResult: {},
+}
+
+// genAIRecognizedAttributes is the pinned set of attributes the analyzer is allowed to
+// read. An unknown gen_ai.* attribute still marks a span as generative AI, but its
+// value is never interpreted.
+var genAIRecognizedAttributes = map[string]struct{}{
+ genAIOperationName: {},
+ genAIProviderName: {},
+ genAIRequestModel: {},
+ genAIResponseModel: {},
+ genAIConversationID: {},
+ genAIAgentName: {},
+ genAIAgentID: {},
+ genAIAgentDescription: {},
+ genAIAgentVersion: {},
+ genAIToolName: {},
+ genAIToolCallID: {},
+ genAIToolCallArguments: {},
+ genAIToolCallResult: {},
+ genAIInputMessages: {},
+ genAIOutputMessages: {},
+ genAIUsageInputTokens: {},
+ genAIUsageOutputTokens: {},
+ genAIUsageReasoningTokens: {},
+ genAIUsageCacheReadTokens: {},
+ genAIUsageCacheWriteTokens: {},
+ genAIResponseFinishReasons: {},
+ genAIResponseID: {},
+ genAIDataSourceID: {},
+ genAIOutputType: {},
+ genAIUsageTotalTokens: {},
+}
+
+// genAITokenAttributes lists every token attribute the analyzer recognizes. The cache
+// and reasoning counters are breakdowns of the input/output counters, so they are never
+// added on top of them.
+var genAITokenAttributes = []string{
+ genAIUsageInputTokens,
+ genAIUsageOutputTokens,
+ genAIUsageReasoningTokens,
+ genAIUsageCacheReadTokens,
+ genAIUsageCacheWriteTokens,
+ genAIUsageTotalTokens,
+}
+
+// genAIModelSignalAttributes mark a span as a model invocation even when the operation
+// name is missing or carries a custom value.
+var genAIModelSignalAttributes = []string{
+ genAIRequestModel,
+ genAIResponseModel,
+ genAIResponseID,
+ genAIResponseFinishReasons,
+ genAIOutputType,
+ genAIDataSourceID,
+}
+
+var (
+ genAIModelOperations = map[string]struct{}{
+ "chat": {},
+ "generate_content": {},
+ "text_completion": {},
+ "embeddings": {},
+ "fetch_response": {},
+ }
+ genAIToolOperations = map[string]struct{}{
+ "execute_tool": {},
+ }
+ genAIAgentOperations = map[string]struct{}{
+ "invoke_agent": {},
+ "create_agent": {},
+ "invoke_workflow": {},
+ "plan": {},
+ }
+)
+
+// genAIClass is the mutually exclusive category a generative AI span falls into.
+type genAIClass int
+
+const (
+ genAIClassNone genAIClass = iota
+ genAIClassModel
+ genAIClassTool
+ genAIClassAgent
+)
+
+// genAIOrderedSpan pairs a span with its resolved start time so the per-category
+// detail lists can be emitted in chronological order.
+type genAIOrderedSpan struct {
+ span *Span
+ startMicros int64
+}
+
+// HasGenAISpans reports whether any span carries generative AI attributes.
+func HasGenAISpans(spans []Span) bool {
+ for i := range spans {
+ if HasGenAIAttributes(spans[i].Attributes) {
+ return true
+ }
+ }
+ return false
+}
+
+// HasGenAIAttributes reports whether an attribute map contains generative AI attributes.
+// The search path needs this check on raw provider tags, so it must not require a
+// normalized span.
+func HasGenAIAttributes(tags map[string]string) bool {
+ for key := range tags {
+ if isGenAIAttributeKey(key) {
+ return true
+ }
+ }
+ return false
+}
+
+// BuildGenAISummary aggregates the generative AI footprint of the given spans.
+// It never fails: spans without generative AI attributes yield Present=false.
+func BuildGenAISummary(spans []Span) *GenAISummary {
+ summary := &GenAISummary{Present: false}
+ ordered := make([]genAIOrderedSpan, 0, len(spans))
+ conversationIDs := map[string]struct{}{}
+ providers := map[string]struct{}{}
+ models := map[string]struct{}{}
+ for i := range spans {
+ span := &spans[i]
+ if !HasGenAIAttributes(span.Attributes) {
+ continue
+ }
+ summary.Present = true
+ ordered = append(ordered, genAIOrderedSpan{span: span, startMicros: spanStartMicros(span)})
+ addString(conversationIDs, genAIAttribute(span.Attributes, genAIConversationID))
+ addString(providers, genAIAttribute(span.Attributes, genAIProviderName))
+ addString(models, genAIAttribute(span.Attributes, genAIRequestModel))
+ addString(models, genAIAttribute(span.Attributes, genAIResponseModel))
+ summary.InputTokens += genAIIntAttribute(span.Attributes, genAIUsageInputTokens)
+ summary.OutputTokens += genAIIntAttribute(span.Attributes, genAIUsageOutputTokens)
+ }
+ // The semantic conventions define the input and the output token counters
+ // only, there is no standard total token attribute. The total is therefore a
+ // derived value with a single definition: it always equals input plus output.
+ // A provider that reports the non standard gen_ai.usage.total_tokens only
+ // shows up as a reference value in the detail of a single call, it never
+ // takes part in the aggregate, so a consumer never has to choose between
+ // two definitions of the total.
+ summary.TotalTokens = summary.InputTokens + summary.OutputTokens
+ if !summary.Present {
+ return summary
+ }
+ sort.SliceStable(ordered, func(i, j int) bool {
+ if ordered[i].startMicros != ordered[j].startMicros {
+ return ordered[i].startMicros < ordered[j].startMicros
+ }
+ return ordered[i].span.SpanID < ordered[j].span.SpanID
+ })
+ summary.ConversationIDs = sortedKeys(conversationIDs)
+ summary.Providers = sortedKeys(providers)
+ summary.Models = sortedKeys(models)
+
+ for _, item := range ordered {
+ span := item.span
+ switch classifyGenAISpan(span) {
+ case genAIClassModel:
+ inputTokens := genAIIntAttribute(span.Attributes, genAIUsageInputTokens)
+ outputTokens := genAIIntAttribute(span.Attributes, genAIUsageOutputTokens)
+ totalTokens := genAISpanTotalTokens(span.Attributes)
+ summary.ModelCallCount++
+ if len(summary.ModelCalls) < maxGenAICalls {
+ summary.ModelCalls = append(summary.ModelCalls, GenAIModelCall{
+ SpanID: span.SpanID,
+ ServiceName: span.ServiceName,
+ OperationName: span.OperationName,
+ Operation: genAIAttribute(span.Attributes, genAIOperationName),
+ ProviderName: genAIAttribute(span.Attributes, genAIProviderName),
+ RequestModel: genAIAttribute(span.Attributes, genAIRequestModel),
+ ResponseModel: genAIAttribute(span.Attributes, genAIResponseModel),
+ InputTokens: inputTokens,
+ OutputTokens: outputTokens,
+ TotalTokens: totalTokens,
+ DurationMicros: span.DurationMicros,
+ Error: spanFailed(span),
+ })
+ } else {
+ summary.Truncated = true
+ }
+ case genAIClassTool:
+ summary.ToolCallCount++
+ if len(summary.ToolCalls) < maxGenAICalls {
+ summary.ToolCalls = append(summary.ToolCalls, GenAIToolCall{
+ SpanID: span.SpanID,
+ ServiceName: span.ServiceName,
+ ToolName: genAIAttribute(span.Attributes, genAIToolName),
+ ToolCallID: genAIAttribute(span.Attributes, genAIToolCallID),
+ DurationMicros: span.DurationMicros,
+ Error: spanFailed(span),
+ })
+ } else {
+ summary.Truncated = true
+ }
+ case genAIClassAgent:
+ summary.AgentCount++
+ if len(summary.Agents) < maxGenAICalls {
+ summary.Agents = append(summary.Agents, GenAIAgent{
+ SpanID: span.SpanID,
+ ServiceName: span.ServiceName,
+ AgentName: genAIAttribute(span.Attributes, genAIAgentName),
+ AgentID: genAIAttribute(span.Attributes, genAIAgentID),
+ DurationMicros: span.DurationMicros,
+ Error: spanFailed(span),
+ })
+ } else {
+ summary.Truncated = true
+ }
+ }
+ }
+ return summary
+}
+
+// classifyGenAISpan assigns exactly one category to a generative AI span.
+// The precedence is model operation, then tool, then agent, and finally a model call
+// inferred from model or token attributes.
+func classifyGenAISpan(span *Span) genAIClass {
+ operation := strings.ToLower(strings.TrimSpace(genAIAttribute(span.Attributes, genAIOperationName)))
+ if _, ok := genAIModelOperations[operation]; ok {
+ return genAIClassModel
+ }
+ if _, ok := genAIToolOperations[operation]; ok {
+ return genAIClassTool
+ }
+ if _, ok := genAIAgentOperations[operation]; ok {
+ return genAIClassAgent
+ }
+ if _, ok := genAIAttributeOK(span.Attributes, genAIToolName); ok {
+ return genAIClassTool
+ }
+ if _, ok := genAIAttributeOK(span.Attributes, genAIAgentName); ok {
+ return genAIClassAgent
+ }
+ for _, name := range genAIModelSignalAttributes {
+ if _, ok := genAIAttributeOK(span.Attributes, name); ok {
+ return genAIClassModel
+ }
+ }
+ for _, name := range genAITokenAttributes {
+ if _, ok := genAIAttributeOK(span.Attributes, name); ok {
+ return genAIClassModel
+ }
+ }
+ return genAIClassNone
+}
+
+// genAISpanTotalTokens returns the token total of a single span. A provider that
+// reports the aggregate of the span wins, otherwise the total is derived from the
+// input and the output counters of that span. The value is only meaningful per
+// span: it is a reference value in the detail of a single call and never takes
+// part in the aggregate of the trace, whose total is always input plus output.
+func genAISpanTotalTokens(attributes map[string]string) int {
+ if override, ok := genAIAttributeOK(attributes, genAIUsageTotalTokens); ok {
+ if parsed, err := strconv.Atoi(strings.TrimSpace(override)); err == nil {
+ return parsed
+ }
+ }
+ return genAIIntAttribute(attributes, genAIUsageInputTokens) + genAIIntAttribute(attributes, genAIUsageOutputTokens)
+}
+
+func isGenAIAttributeKey(key string) bool {
+ return strings.HasPrefix(strings.ToLower(strings.TrimSpace(key)), genAIPrefix)
+}
+
+// genAIAttribute reads a recognized, non-content generative AI attribute.
+func genAIAttribute(attributes map[string]string, name string) string {
+ value, _ := genAIAttributeOK(attributes, name)
+ return value
+}
+
+// genAIAttributeOK reads a recognized, non-content generative AI attribute.
+func genAIAttributeOK(attributes map[string]string, name string) (string, bool) {
+ if _, content := genAIContentAttributes[name]; content {
+ return "", false
+ }
+ if _, recognized := genAIRecognizedAttributes[name]; !recognized {
+ return "", false
+ }
+ return lookupAttributeOK(attributes, name)
+}
+
+func genAIIntAttribute(attributes map[string]string, name string) int {
+ raw, ok := genAIAttributeOK(attributes, name)
+ if !ok {
+ return 0
+ }
+ return parseIntegralAttribute(strings.TrimSpace(raw))
+}
+
+// parseIntegralAttribute reads an integer attribute value. The attribute map is
+// filled from the provider payload, where a number arrives as a JSON number.
+// The decoder turns it into a float64 and the formatter renders it with the
+// 'g' format, so a token count of 1000000 becomes "1e+06". A plain integer
+// parse drops such a value silently, so the integral scientific notation is
+// accepted as well.
+func parseIntegralAttribute(raw string) int {
+ if raw == "" {
+ return 0
+ }
+ if parsed, err := strconv.Atoi(raw); err == nil {
+ return parsed
+ }
+ value, err := strconv.ParseFloat(raw, 64)
+ if err != nil {
+ return 0
+ }
+ if value != math.Trunc(value) || value > math.MaxInt64 || value < math.MinInt64 {
+ return 0
+ }
+ return int(value)
+}
+
+func addString(target map[string]struct{}, value string) {
+ trimmed := strings.TrimSpace(value)
+ if trimmed == "" {
+ return
+ }
+ target[trimmed] = struct{}{}
+}
+
+func sortedKeys(values map[string]struct{}) []string {
+ if len(values) == 0 {
+ return nil
+ }
+ result := make([]string, 0, len(values))
+ for key := range values {
+ result = append(result, key)
+ }
+ sort.Strings(result)
+ return result
+}
+
+// spanStartMicros converts a span start timestamp into unix microseconds.
+// Unparsable timestamps degrade to zero so the analyzer never panics.
+func spanStartMicros(span *Span) int64 {
+ parsed, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(span.StartTime))
+ if err != nil {
+ return 0
+ }
+ return parsed.UnixMicro()
+}
+
+// spanFailed reports whether a span is in an error state.
+func spanFailed(span *Span) bool {
+ return span.Error || strings.EqualFold(span.Status, "ERROR")
+}
diff --git a/pkg/mcp/tools/trace/genai_test.go b/pkg/mcp/tools/trace/genai_test.go
new file mode 100644
index 000000000..fbea7e3c2
--- /dev/null
+++ b/pkg/mcp/tools/trace/genai_test.go
@@ -0,0 +1,334 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package trace
+
+import (
+ "encoding/json"
+ "strings"
+ "testing"
+)
+
+func genAISpan(spanID string, attributes map[string]string) Span {
+ return Span{
+ SpanID: spanID,
+ ServiceName: "dubbo-admin-sre-agent",
+ OperationName: spanID,
+ StartTime: analysisBase,
+ Attributes: attributes,
+ }
+}
+
+func TestHasGenAIAttributes(t *testing.T) {
+ if HasGenAIAttributes(map[string]string{"rpc.system": "apache_dubbo"}) {
+ t.Fatal("did not expect a non generative AI attribute to match")
+ }
+ if !HasGenAIAttributes(map[string]string{"gen_ai.operation.name": "chat"}) {
+ t.Fatal("expected a generative AI attribute to match")
+ }
+ if !HasGenAIAttributes(map[string]string{" GEN_AI.Operation.Name ": "chat"}) {
+ t.Fatal("expected the detection to tolerate casing and padding")
+ }
+}
+
+func TestHasGenAISpans(t *testing.T) {
+ if HasGenAISpans([]Span{{SpanID: "01"}}) {
+ t.Fatal("did not expect a plain span to match")
+ }
+ if !HasGenAISpans([]Span{{SpanID: "01"}, {SpanID: "02", Attributes: map[string]string{"gen_ai.agent.name": "sre"}}}) {
+ t.Fatal("expected a generative AI span to match")
+ }
+}
+
+func TestBuildGenAISummaryClassifiesModelCall(t *testing.T) {
+ summary := BuildGenAISummary([]Span{
+ genAISpan("01", map[string]string{
+ "gen_ai.operation.name": "chat",
+ "gen_ai.provider.name": "openai",
+ "gen_ai.request.model": "gpt-4o",
+ "gen_ai.response.model": "gpt-4o-2024-08-06",
+ "gen_ai.conversation.id": "conv-1",
+ "gen_ai.usage.input_tokens": "120",
+ "gen_ai.usage.output_tokens": "30",
+ }),
+ })
+ if !summary.Present {
+ t.Fatal("expected the summary to be present")
+ }
+ if summary.ModelCallCount != 1 || summary.ToolCallCount != 0 || summary.AgentCount != 0 {
+ t.Fatalf("unexpected call counts: %#v", summary)
+ }
+ if len(summary.ModelCalls) != 1 {
+ t.Fatalf("unexpected model calls: %#v", summary.ModelCalls)
+ }
+ call := summary.ModelCalls[0]
+ if call.Operation != "chat" || call.ProviderName != "openai" || call.RequestModel != "gpt-4o" || call.ResponseModel != "gpt-4o-2024-08-06" {
+ t.Fatalf("unexpected model call: %#v", call)
+ }
+ if call.InputTokens != 120 || call.OutputTokens != 30 || call.TotalTokens != 150 {
+ t.Fatalf("unexpected token usage: %#v", call)
+ }
+ if summary.InputTokens != 120 || summary.OutputTokens != 30 || summary.TotalTokens != 150 {
+ t.Fatalf("unexpected token totals: %#v", summary)
+ }
+ if len(summary.Providers) != 1 || summary.Providers[0] != "openai" {
+ t.Fatalf("unexpected providers: %#v", summary.Providers)
+ }
+ if len(summary.Models) != 2 || summary.Models[0] != "gpt-4o" {
+ t.Fatalf("unexpected models: %#v", summary.Models)
+ }
+ if len(summary.ConversationIDs) != 1 || summary.ConversationIDs[0] != "conv-1" {
+ t.Fatalf("unexpected conversations: %#v", summary.ConversationIDs)
+ }
+}
+
+func TestBuildGenAISummaryClassifiesToolAndAgent(t *testing.T) {
+ summary := BuildGenAISummary([]Span{
+ genAISpan("01", map[string]string{
+ "gen_ai.operation.name": "execute_tool",
+ "gen_ai.tool.name": "search_logs",
+ "gen_ai.tool.call.id": "call-1",
+ }),
+ genAISpan("02", map[string]string{
+ "gen_ai.operation.name": "invoke_agent",
+ "gen_ai.agent.name": "dubbo-admin-sre-agent",
+ "gen_ai.agent.id": "agent-42",
+ }),
+ })
+ if summary.ToolCallCount != 1 || summary.AgentCount != 1 || summary.ModelCallCount != 0 {
+ t.Fatalf("unexpected call counts: %#v", summary)
+ }
+ if len(summary.ToolCalls) != 1 || summary.ToolCalls[0].ToolName != "search_logs" || summary.ToolCalls[0].ToolCallID != "call-1" {
+ t.Fatalf("unexpected tool calls: %#v", summary.ToolCalls)
+ }
+ if len(summary.Agents) != 1 || summary.Agents[0].AgentName != "dubbo-admin-sre-agent" || summary.Agents[0].AgentID != "agent-42" {
+ t.Fatalf("unexpected agents: %#v", summary.Agents)
+ }
+}
+
+func TestBuildGenAISummaryClassifiesByAttributesWithoutOperationName(t *testing.T) {
+ summary := BuildGenAISummary([]Span{
+ genAISpan("01", map[string]string{"gen_ai.tool.name": "search_logs"}),
+ genAISpan("02", map[string]string{"gen_ai.agent.name": "sre"}),
+ genAISpan("03", map[string]string{"gen_ai.request.model": "gpt-4o"}),
+ genAISpan("04", map[string]string{"gen_ai.usage.input_tokens": "10"}),
+ })
+ if summary.ToolCallCount != 1 || summary.AgentCount != 1 || summary.ModelCallCount != 2 {
+ t.Fatalf("unexpected call counts: %#v", summary)
+ }
+}
+
+func TestBuildGenAISummaryKeepsProviderTokenTotalAsCallDetailOnly(t *testing.T) {
+ // The semantic conventions define the input and the output token counters
+ // only. The total of the summary is therefore always their sum, the value
+ // reported by the provider stays a reference in the detail of the call.
+ summary := BuildGenAISummary([]Span{
+ genAISpan("01", map[string]string{
+ "gen_ai.operation.name": "chat",
+ "gen_ai.usage.input_tokens": "10",
+ "gen_ai.usage.output_tokens": "5",
+ "gen_ai.usage.total_tokens": "99",
+ }),
+ })
+ if summary.ModelCalls[0].TotalTokens != 99 {
+ t.Fatalf("expected the provider total to be kept in the call detail: %#v", summary.ModelCalls[0])
+ }
+ if summary.TotalTokens != summary.InputTokens+summary.OutputTokens {
+ t.Fatalf("expected the total to equal the input plus the output tokens: %#v", summary)
+ }
+ if summary.TotalTokens != 15 {
+ t.Fatalf("expected the total to be derived from the input and the output tokens: %#v", summary)
+ }
+}
+
+func TestBuildGenAISummaryWithoutGenAIAttributes(t *testing.T) {
+ summary := BuildGenAISummary([]Span{{SpanID: "01", ServiceName: "order-service"}})
+ if summary.Present {
+ t.Fatal("did not expect the summary to be present")
+ }
+ if summary.TotalTokens != 0 || summary.ModelCallCount != 0 {
+ t.Fatalf("unexpected summary: %#v", summary)
+ }
+}
+
+func TestBuildGenAISummaryDerivesAggregateTotalFromInputAndOutput(t *testing.T) {
+ // Only one span reports its own total. The aggregate must not be influenced
+ // by it, the total is always the sum of the input and the output tokens.
+ summary := BuildGenAISummary([]Span{
+ genAISpan("01", map[string]string{
+ "gen_ai.operation.name": "chat",
+ "gen_ai.usage.input_tokens": "100",
+ "gen_ai.usage.output_tokens": "50",
+ }),
+ genAISpan("02", map[string]string{
+ "gen_ai.operation.name": "chat",
+ "gen_ai.usage.input_tokens": "10",
+ "gen_ai.usage.output_tokens": "5",
+ "gen_ai.usage.total_tokens": "999",
+ }),
+ })
+ if summary.InputTokens != 110 || summary.OutputTokens != 55 {
+ t.Fatalf("unexpected token counters: %#v", summary)
+ }
+ // The reported total of the second span does not take part in the aggregate.
+ if summary.TotalTokens != 110+55 {
+ t.Fatalf("expected the total to equal the input plus the output tokens, got %d", summary.TotalTokens)
+ }
+ if len(summary.ModelCalls) != 2 {
+ t.Fatalf("unexpected model calls: %#v", summary.ModelCalls)
+ }
+ totals := map[string]int{}
+ for _, call := range summary.ModelCalls {
+ totals[call.SpanID] = call.TotalTokens
+ }
+ // The detail of a single call keeps the value reported by the provider.
+ if totals["01"] != 150 || totals["02"] != 999 {
+ t.Fatalf("unexpected per call totals: %#v", totals)
+ }
+}
+
+func TestBuildGenAISummaryDoesNotReadContentAttributes(t *testing.T) {
+ summary := BuildGenAISummary([]Span{
+ genAISpan("01", map[string]string{
+ "gen_ai.operation.name": "chat",
+ "gen_ai.request.model": "gpt-4o",
+ "gen_ai.input.messages": `{"messages":[{"role":"user","content":"secret prompt"}]}`,
+ "gen_ai.output.messages": `{"messages":[{"role":"assistant","content":"secret answer"}]}`,
+ "gen_ai.tool.call.arguments": `{"query":"secret argument"}`,
+ "gen_ai.tool.call.result": `{"rows":["secret result"]}`,
+ }),
+ })
+ if summary.ModelCallCount != 1 {
+ t.Fatalf("unexpected call counts: %#v", summary)
+ }
+ for _, call := range summary.ModelCalls {
+ if call.RequestModel != "gpt-4o" {
+ t.Fatalf("unexpected model call: %#v", call)
+ }
+ }
+ // The content attributes must not reach the tool output at all.
+ encoded, err := json.Marshal(summary)
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, secret := range []string{"secret prompt", "secret answer", "secret argument", "secret result"} {
+ if strings.Contains(string(encoded), secret) {
+ t.Fatalf("expected the content attribute to be omitted, got %s", encoded)
+ }
+ }
+}
+
+func TestBuildGenAISummaryTruncatesDetailLists(t *testing.T) {
+ spans := make([]Span, 0, maxGenAICalls+5)
+ for i := 0; i < maxGenAICalls+5; i++ {
+ spans = append(spans, genAISpan(string(rune('a'+i%26))+string(rune('a'+i/26)), map[string]string{
+ "gen_ai.operation.name": "chat",
+ "gen_ai.request.model": "gpt-4o",
+ }))
+ }
+ summary := BuildGenAISummary(spans)
+ if summary.ModelCallCount != maxGenAICalls+5 {
+ t.Fatalf("expected the real call count, got %d", summary.ModelCallCount)
+ }
+ if len(summary.ModelCalls) != maxGenAICalls {
+ t.Fatalf("expected the detail list to be capped, got %d", len(summary.ModelCalls))
+ }
+ if !summary.Truncated {
+ t.Fatal("expected the summary to be marked as truncated")
+ }
+}
+
+func TestBuildGenAISummarySortsDetailsByStartTime(t *testing.T) {
+ late := genAISpan("02", map[string]string{"gen_ai.operation.name": "chat"})
+ late.StartTime = shiftTime(analysisBase, 5000)
+ early := genAISpan("01", map[string]string{"gen_ai.operation.name": "chat"})
+ early.StartTime = shiftTime(analysisBase, 1000)
+ summary := BuildGenAISummary([]Span{late, early})
+ if len(summary.ModelCalls) != 2 {
+ t.Fatalf("unexpected model calls: %#v", summary.ModelCalls)
+ }
+ if summary.ModelCalls[0].SpanID != "01" || summary.ModelCalls[1].SpanID != "02" {
+ t.Fatalf("unexpected model call order: %#v", summary.ModelCalls)
+ }
+}
+
+func TestGenAIOperationClassification(t *testing.T) {
+ tests := []struct {
+ operation string
+ expected genAIClass
+ }{
+ {operation: "chat", expected: genAIClassModel},
+ {operation: "generate_content", expected: genAIClassModel},
+ {operation: "text_completion", expected: genAIClassModel},
+ {operation: "embeddings", expected: genAIClassModel},
+ {operation: "fetch_response", expected: genAIClassModel},
+ {operation: "execute_tool", expected: genAIClassTool},
+ {operation: "invoke_agent", expected: genAIClassAgent},
+ {operation: "create_agent", expected: genAIClassAgent},
+ {operation: "invoke_workflow", expected: genAIClassAgent},
+ {operation: "plan", expected: genAIClassAgent},
+ }
+ for _, test := range tests {
+ t.Run(test.operation, func(t *testing.T) {
+ span := genAISpan("01", map[string]string{"gen_ai.operation.name": test.operation})
+ if actual := classifyGenAISpan(&span); actual != test.expected {
+ t.Fatalf("expected %v, got %v", test.expected, actual)
+ }
+ })
+ }
+}
+
+func TestGenAIAnalysisEmbedsSummaryInTraceAnalysis(t *testing.T) {
+ trace := &Trace{
+ TraceID: testTraceID,
+ Spans: []Span{
+ spanAt(0, "01", "", "dubbo-admin-sre-agent", "chat gpt-4o", 1000),
+ spanAt(100, "02", "01", "dubbo-admin-sre-agent", "execute_tool search_logs", 500),
+ },
+ }
+ trace.Spans[0].Attributes = map[string]string{"gen_ai.operation.name": "chat", "gen_ai.usage.input_tokens": "5"}
+ trace.Spans[1].Attributes = map[string]string{"gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": "search_logs"}
+ analysis := BuildTraceAnalysis(trace)
+ if analysis.GenAI == nil {
+ t.Fatal("expected a generative AI summary")
+ }
+ if !analysis.GenAI.Present || analysis.GenAI.ModelCallCount != 1 || analysis.GenAI.ToolCallCount != 1 {
+ t.Fatalf("unexpected generative AI summary: %#v", analysis.GenAI)
+ }
+}
+
+func TestGenAIIntAttributeAcceptsIntegralScientificNotation(t *testing.T) {
+ // The attribute map is filled from a JSON payload, so a number arrives as a
+ // float64 and the formatter renders it with the 'g' format: 1000000 becomes
+ // "1e+06". A plain integer parse drops such a value silently, which would
+ // report a million token model call as a call without any token.
+ attributes := map[string]string{
+ "gen_ai.usage.input_tokens": "1e+06",
+ "gen_ai.usage.output_tokens": "1500000",
+ "gen_ai.usage.total_tokens": "12.5",
+ }
+ if got := genAIIntAttribute(attributes, "gen_ai.usage.input_tokens"); got != 1000000 {
+ t.Fatalf("input tokens = %d, want %d", got, 1000000)
+ }
+ if got := genAIIntAttribute(attributes, "gen_ai.usage.output_tokens"); got != 1500000 {
+ t.Fatalf("output tokens = %d, want %d", got, 1500000)
+ }
+ // A fractional value is not a token count and must not be rounded.
+ if got := genAIIntAttribute(attributes, "gen_ai.usage.total_tokens"); got != 0 {
+ t.Fatalf("total tokens = %d, want %d", got, 0)
+ }
+}
diff --git a/pkg/mcp/tools/trace/jaeger.go b/pkg/mcp/tools/trace/jaeger.go
index 354c5ebf2..23898059e 100644
--- a/pkg/mcp/tools/trace/jaeger.go
+++ b/pkg/mcp/tools/trace/jaeger.go
@@ -20,6 +20,7 @@ package trace
import (
"context"
"encoding/json"
+ "errors"
"fmt"
"io"
"net/http"
@@ -116,6 +117,20 @@ func newJaegerProvider(cfg observabilitycfg.TraceProviderConfig) (*jaegerProvide
if err != nil {
return nil, fmt.Errorf("invalid jaeger endpoint: %w", err)
}
+ if baseURL.Scheme != "http" && baseURL.Scheme != "https" {
+ return nil, fmt.Errorf("invalid jaeger endpoint: scheme must be http or https")
+ }
+ if baseURL.Host == "" {
+ return nil, fmt.Errorf("invalid jaeger endpoint: host is missing")
+ }
+ // A query string on the endpoint would be carried into every request URL and
+ // into the transport error of a failed lookup, which reaches the tool result
+ // and the model context. Credentials belong in bearerToken, so the query is
+ // rejected instead of being forwarded. The error message does not repeat the
+ // endpoint, because the endpoint is exactly what may hold the secret.
+ if baseURL.RawQuery != "" || baseURL.Fragment != "" {
+ return nil, fmt.Errorf("invalid jaeger endpoint: query parameters are not supported, use bearerToken for credentials")
+ }
return &jaegerProvider{
config: cfg,
baseURL: baseURL,
@@ -145,7 +160,12 @@ func (p *jaegerProvider) GetTraceByID(ctx context.Context, traceID string) (*Tra
resp, err := p.client.Do(req)
if err != nil {
p.logFailure(traceID, startedAt, 0, "transport", err)
- return nil, fmt.Errorf("jaeger trace query failed: %w", err)
+ // The transport error embeds the request URL, which repeats the configured
+ // endpoint and carries the retrieval conditions. It is sanitized before it
+ // reaches the tool result and the model context. The trace identifier is
+ // kept, because the model chose it and has to be able to report which
+ // lookup failed.
+ return nil, fmt.Errorf("jaeger trace query failed for trace %s: %w", traceID, sanitizedTransportError(err))
}
defer resp.Body.Close()
if resp.StatusCode == http.StatusNotFound {
@@ -184,6 +204,237 @@ func (p *jaegerProvider) GetTraceByID(ctx context.Context, traceID string) (*Tra
return result, nil
}
+// SearchTraces discovers candidate traces through the Jaeger v1 query API.
+// The summaries are computed from the raw spans so that no information is lost to
+// the normalization applied by the single trace lookup.
+func (p *jaegerProvider) SearchTraces(ctx context.Context, query TraceSearchQuery) (*TraceSearchResult, error) {
+ if strings.TrimSpace(query.Service) == "" {
+ return nil, fmt.Errorf("service is required for trace search")
+ }
+ values := url.Values{}
+ values.Set("service", query.Service)
+ if query.Operation != "" {
+ values.Set("operation", query.Operation)
+ }
+ if query.TraceID != "" {
+ values.Set("traceID", query.TraceID)
+ }
+ if len(query.Tags) > 0 {
+ encoded, err := json.Marshal(query.Tags)
+ if err != nil {
+ return nil, fmt.Errorf("failed to encode trace search tags: %w", err)
+ }
+ values.Set("tags", string(encoded))
+ }
+ if !query.StartTime.IsZero() {
+ values.Set("start", strconv.FormatInt(query.StartTime.UnixMicro(), 10))
+ }
+ if !query.EndTime.IsZero() {
+ values.Set("end", strconv.FormatInt(query.EndTime.UnixMicro(), 10))
+ }
+ if query.MinDuration > 0 {
+ // Jaeger parses minDuration with time.ParseDuration, so the value has to
+ // carry a unit. A bare microsecond count is rejected with a 400
+ // ("missing unit in duration"), so the duration is sent in its string
+ // form ("250ms") instead.
+ values.Set("minDuration", query.MinDuration.String())
+ }
+ if query.Limit > 0 {
+ // One trace more than the caller asked for is requested. A full result
+ // then proves that more traces match than were returned, which is what
+ // the truncated flag has to tell the agent.
+ values.Set("limit", strconv.Itoa(query.Limit+1))
+ }
+
+ endpoint := *p.baseURL
+ endpoint.Path = strings.TrimRight(endpoint.Path, "/") + "/api/traces"
+ endpoint.RawQuery = values.Encode()
+ startedAt := time.Now()
+ req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint.String(), nil)
+ if err != nil {
+ return nil, fmt.Errorf("failed to create jaeger search request: %w", err)
+ }
+ req.Header.Set("Accept", "application/json")
+ if p.config.BearerToken != "" {
+ req.Header.Set("Authorization", "Bearer "+p.config.BearerToken)
+ }
+ if p.config.Tenant != "" {
+ req.Header.Set(tenantHeader, p.config.Tenant)
+ }
+
+ resp, err := p.client.Do(req)
+ if err != nil {
+ p.logSearchFailure(query, startedAt, 0, "transport", err)
+ // The transport error embeds the request URL, which repeats the configured
+ // endpoint and carries the retrieval conditions. It is sanitized before it
+ // reaches the tool result and the model context. The service anchor is
+ // kept, because the model chose it and has to be able to report which
+ // lookup failed.
+ return nil, fmt.Errorf("jaeger trace search failed for service %s: %w", query.Service, sanitizedTransportError(err))
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices {
+ body, _ := io.ReadAll(io.LimitReader(resp.Body, maxTraceErrorBodyBytes))
+ p.logSearchFailure(query, startedAt, resp.StatusCode, "upstream_status", nil)
+ return nil, fmt.Errorf("jaeger trace search failed with status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
+ }
+ body, err := readTraceBody(resp.Body)
+ if err != nil {
+ p.logSearchFailure(query, startedAt, resp.StatusCode, "response_limit", err)
+ return nil, err
+ }
+ var upstream jaegerResponse
+ if err := json.Unmarshal(body, &upstream); err != nil {
+ p.logSearchFailure(query, startedAt, resp.StatusCode, "decode", err)
+ return nil, fmt.Errorf("failed to decode jaeger search response: %w", err)
+ }
+ if len(upstream.Errors) > 0 {
+ p.logSearchFailure(query, startedAt, resp.StatusCode, "upstream_error", nil)
+ return nil, fmt.Errorf("jaeger trace search failed: %s", upstream.Errors[0].Message)
+ }
+ result := summarizeJaegerTraces(upstream.Data, query.Limit)
+ logger.Sugar().Infow("trace search completed", "provider", p.config.Name, "engine", p.config.Type, "service", query.Service, "elapsed_ms", time.Since(startedAt).Milliseconds(), "traces", len(result.Traces), "total", result.Total, "truncated", result.Truncated)
+ return result, nil
+}
+
+// jaegerTraceSummary keeps the numeric start time next to the summary so the result
+// can be ordered on the absolute timeline instead of on the formatted timestamp.
+type jaegerTraceSummary struct {
+ summary TraceSummary
+ startMicros int64
+}
+
+// summarizeJaegerTraces converts the raw traces into bounded per-trace summaries.
+func summarizeJaegerTraces(traces []jaegerTrace, limit int) *TraceSearchResult {
+ result := &TraceSearchResult{Traces: []TraceSummary{}, SourceEngine: "jaeger"}
+ summaries := make([]jaegerTraceSummary, 0, len(traces))
+ for _, trace := range traces {
+ summary, startMicros := summarizeJaegerTrace(trace)
+ summaries = append(summaries, jaegerTraceSummary{summary: summary, startMicros: startMicros})
+ }
+ sort.SliceStable(summaries, func(i, j int) bool {
+ if summaries[i].startMicros != summaries[j].startMicros {
+ return summaries[i].startMicros < summaries[j].startMicros
+ }
+ return summaries[i].summary.TraceID < summaries[j].summary.TraceID
+ })
+ // The provider was asked for one trace more than the caller wants, so a
+ // result that reached that limit proves that more traces match than were
+ // returned. Without the extra trace a full result would look complete.
+ total := len(traces)
+ if limit > 0 && len(summaries) > limit {
+ result.Truncated = true
+ summaries = summaries[:limit]
+ total = limit
+ }
+ if len(summaries) > maxSearchTraceSummaries {
+ summaries = summaries[:maxSearchTraceSummaries]
+ result.Truncated = true
+ }
+ for _, item := range summaries {
+ result.Traces = append(result.Traces, item.summary)
+ }
+ result.Total = total
+ return result
+}
+
+// summarizeJaegerTrace derives the summary of one trace from all of its raw spans.
+func summarizeJaegerTrace(input jaegerTrace) (TraceSummary, int64) {
+ summary := TraceSummary{TraceID: input.TraceID}
+ services := map[string]struct{}{}
+ var minStart, maxEnd int64
+ hasStart := false
+ var rootSpan jaegerSpan
+ hasRoot := false
+ genai := false
+ // The span ids of the trace decide which spans are roots: a span whose parent
+ // is not part of the trace is an orphan and roots the trace on its own.
+ spanIDs := make(map[string]struct{}, len(input.Spans))
+ for _, rawSpan := range input.Spans {
+ if rawSpan.SpanID != "" {
+ spanIDs[rawSpan.SpanID] = struct{}{}
+ }
+ }
+ for _, rawSpan := range input.Spans {
+ process := input.Processes[rawSpan.ProcessID]
+ if rawSpan.Process != nil {
+ process = *rawSpan.Process
+ }
+ if process.ServiceName != "" {
+ services[process.ServiceName] = struct{}{}
+ }
+ if errorSpan, _ := spanStatus(rawSpan.Tags); errorSpan {
+ summary.ErrorSpanCount++
+ }
+ if hasGenAIJaegerTags(rawSpan.Tags) {
+ genai = true
+ }
+ if !hasStart || rawSpan.StartTime < minStart {
+ minStart = rawSpan.StartTime
+ hasStart = true
+ }
+ if end := rawSpan.StartTime + rawSpan.Duration; end > maxEnd {
+ maxEnd = end
+ }
+ if isJaegerRootSpan(rawSpan, spanIDs) && (!hasRoot || betterJaegerRoot(rawSpan, rootSpan)) {
+ rootSpan = rawSpan
+ hasRoot = true
+ summary.RootService = process.ServiceName
+ summary.RootOperation = rawSpan.OperationName
+ }
+ }
+ summary.SpanCount = len(input.Spans)
+ summary.ServiceCount = len(services)
+ summary.HasError = summary.ErrorSpanCount > 0
+ if hasStart {
+ summary.StartTime = jaegerMicrosTime(minStart)
+ summary.DurationMicros = maxEnd - minStart
+ }
+ if genai {
+ summary.TraceKind = "genai"
+ }
+ return summary, minStart
+}
+
+// isJaegerRootSpan reports whether the span roots the trace. A span is a root when
+// it has no parent reference at all, or when the referenced parent is not part of
+// the trace. The latter happens for orphans whose parent was dropped by the
+// sampling or by the span limit. The analyzer uses the same rule, so the search
+// summary and the diagnosis agree on the root of a trace.
+func isJaegerRootSpan(rawSpan jaegerSpan, spanIDs map[string]struct{}) bool {
+ parentID := parentSpanID(rawSpan.References)
+ if parentID == "" {
+ return true
+ }
+ _, present := spanIDs[parentID]
+ return !present
+}
+
+// betterJaegerRoot prefers the root span that finishes last, because that span
+// determines the end of the request.
+func betterJaegerRoot(candidate, current jaegerSpan) bool {
+ candidateEnd := candidate.StartTime + candidate.Duration
+ currentEnd := current.StartTime + current.Duration
+ if candidateEnd != currentEnd {
+ return candidateEnd > currentEnd
+ }
+ if candidate.Duration != current.Duration {
+ return candidate.Duration > current.Duration
+ }
+ return candidate.SpanID < current.SpanID
+}
+
+// hasGenAIJaegerTags detects generative AI attributes on raw provider tags without
+// allocating a normalized attribute map.
+func hasGenAIJaegerTags(tags []jaegerKeyValue) bool {
+ for _, tag := range tags {
+ if isGenAIAttributeKey(tag.Key) {
+ return true
+ }
+ }
+ return false
+}
+
func normalizeJaegerTrace(input jaegerTrace) *Trace {
result := &Trace{TraceID: input.TraceID, SourceEngine: "jaeger", SpanCount: len(input.Spans), Spans: []Span{}}
serviceSet := map[string]struct{}{}
@@ -394,13 +645,74 @@ func truncateUTF8(value string, limit int) (string, bool) {
return value, true
}
+// logFailure records a failed trace query. The error is sanitized, because the
+// design document section 10 only allows the provider name, the service name, the
+// elapsed time, the error type and the upstream status code in a failure log: the
+// query payload must never be logged.
func (p *jaegerProvider) logFailure(traceID string, startedAt time.Time, status int, errorType string, err error) {
fields := []any{"provider", p.config.Name, "engine", p.config.Type, "trace_id", maskedTraceID(traceID), "elapsed_ms", time.Since(startedAt).Milliseconds(), "error_type", errorType}
if status != 0 {
fields = append(fields, "upstream_status", status)
}
- if err != nil {
- fields = append(fields, "error", err)
+ if sanitized := sanitizeTransportError(err); sanitized != nil {
+ fields = append(fields, "error", sanitized)
}
logger.Sugar().Warnw("trace query failed", fields...)
}
+
+// logSearchFailure records a failed search. The query payload and the credentials
+// are deliberately left out so the log never leaks the lookup filters. The service
+// name is kept because it is the only anchor of the failed lookup, but the error
+// is sanitized because it would otherwise repeat the whole request URL.
+func (p *jaegerProvider) logSearchFailure(query TraceSearchQuery, startedAt time.Time, status int, errorType string, err error) {
+ fields := []any{"provider", p.config.Name, "engine", p.config.Type, "service", query.Service, "elapsed_ms", time.Since(startedAt).Milliseconds(), "error_type", errorType}
+ if query.TraceID != "" {
+ fields = append(fields, "trace_id", maskedTraceID(query.TraceID))
+ }
+ if status != 0 {
+ fields = append(fields, "upstream_status", status)
+ }
+ if sanitized := sanitizeTransportError(err); sanitized != nil {
+ fields = append(fields, "error", sanitized)
+ }
+ logger.Sugar().Warnw("trace search failed", fields...)
+}
+
+// TODO: the tool result still returns the upstream error text of a failed
+// response unchanged, and that text can carry sensitive data of the returned
+// spans, such as user identifiers or business data. The request URL is no
+// longer part of it, because the transport error is sanitized. A production
+// deployment still has to decide per deployment scenario whether the span
+// content of the tool result is filtered or redacted. That mechanism is not
+// implemented yet, this round only records the decision.
+
+// sanitizeTransportError removes the request URL from a transport failure.
+// http.Client.Do reports transport problems as *url.Error, whose message embeds
+// the full request URL including the query string. The query carries the caller's
+// filters and may contain business identifiers or secrets, so only the inner
+// error is kept: it still explains the failure ("dial tcp ...: connection refused")
+// without repeating the payload. The function guards the failure logs and the
+// tool result.
+func sanitizeTransportError(err error) error {
+ if err == nil {
+ return nil
+ }
+ var urlErr *url.Error
+ if errors.As(err, &urlErr) {
+ if urlErr.Err == nil {
+ return nil
+ }
+ return urlErr.Err
+ }
+ return err
+}
+
+// sanitizedTransportError returns the sanitized form of a transport failure and
+// never returns nil. The tool result needs a message in every case, while the
+// failure logs only add the field when there is something to report.
+func sanitizedTransportError(err error) error {
+ if sanitized := sanitizeTransportError(err); sanitized != nil {
+ return sanitized
+ }
+ return errors.New("unknown transport error")
+}
diff --git a/pkg/mcp/tools/trace/jaeger_logging_test.go b/pkg/mcp/tools/trace/jaeger_logging_test.go
new file mode 100644
index 000000000..3eb670b24
--- /dev/null
+++ b/pkg/mcp/tools/trace/jaeger_logging_test.go
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package trace
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "net/url"
+ "os"
+ "strings"
+ "testing"
+ "time"
+)
+
+// The logger is a process wide singleton that writes to a file next to the
+// working directory. The tests therefore read the lines that their own call
+// appended instead of replacing the logger.
+
+// traceLogFile is the default output path of the shared logger.
+const traceLogFile = "logs/dubbo-admin.log"
+
+// traceLogOffset returns the current size of the log file so that a test can
+// read only the lines that its own call appends.
+func traceLogOffset(t *testing.T) int64 {
+ t.Helper()
+ info, err := os.Stat(traceLogFile)
+ if err != nil {
+ if os.IsNotExist(err) {
+ return 0
+ }
+ t.Fatal(err)
+ }
+ return info.Size()
+}
+
+// traceLogLines returns the log lines that were appended after the offset.
+func traceLogLines(t *testing.T, offset int64) []string {
+ t.Helper()
+ data, err := os.ReadFile(traceLogFile)
+ if err != nil {
+ if os.IsNotExist(err) {
+ return nil
+ }
+ t.Fatal(err)
+ }
+ if int64(len(data)) < offset {
+ offset = 0
+ }
+ var lines []string
+ for _, line := range strings.Split(string(data[offset:]), "\n") {
+ if strings.TrimSpace(line) != "" {
+ lines = append(lines, line)
+ }
+ }
+ return lines
+}
+
+// traceLogSince waits for the log lines that were appended after the offset.
+func traceLogSince(t *testing.T, offset int64) []string {
+ t.Helper()
+ deadline := time.Now().Add(2 * time.Second)
+ for {
+ lines := traceLogLines(t, offset)
+ if len(lines) > 0 || time.Now().After(deadline) {
+ return lines
+ }
+ time.Sleep(20 * time.Millisecond)
+ }
+}
+
+// traceLogErrorFields returns the error field of every appended log line.
+func traceLogErrorFields(t *testing.T, lines []string) []string {
+ t.Helper()
+ var values []string
+ for _, line := range lines {
+ fields := map[string]any{}
+ if err := json.Unmarshal([]byte(line), &fields); err != nil {
+ continue
+ }
+ if value, ok := fields["error"]; ok {
+ values = append(values, fmt.Sprint(value))
+ }
+ }
+ return values
+}
+
+func TestSanitizeTransportError(t *testing.T) {
+ if sanitizeTransportError(nil) != nil {
+ t.Fatal("expected no error for a nil input")
+ }
+ inner := errors.New("dial tcp 127.0.0.1:1: connect: connection refused")
+ urlErr := &url.Error{Op: "Get", URL: "http://127.0.0.1:1/api/traces?service=secret-service", Err: inner}
+ sanitized := sanitizeTransportError(urlErr)
+ if sanitized != inner {
+ t.Fatalf("expected the inner error, got %v", sanitized)
+ }
+ if strings.Contains(sanitized.Error(), "secret-service") {
+ t.Fatalf("expected the request url to be removed, got %q", sanitized.Error())
+ }
+ plain := errors.New("boom")
+ if sanitizeTransportError(plain) != plain {
+ t.Fatal("expected a plain error to be kept as it is")
+ }
+ // A wrapped transport error must be unwrapped as well.
+ wrapped := fmt.Errorf("jaeger trace query failed: %w", urlErr)
+ if sanitizeTransportError(wrapped) != inner {
+ t.Fatalf("expected the inner error of a wrapped failure, got %v", sanitizeTransportError(wrapped))
+ }
+}
+
+func TestJaegerProviderSearchFailureDoesNotLeakQueryPayload(t *testing.T) {
+ provider := newSearchProvider(t, "http://127.0.0.1:1")
+ offset := traceLogOffset(t)
+ if _, err := provider.SearchTraces(context.Background(), TraceSearchQuery{
+ Service: "secret-service",
+ Operation: "secret-operation",
+ Tags: map[string]string{"secret.key": "secret-value"},
+ }); err == nil {
+ t.Fatal("expected a transport failure")
+ }
+ lines := traceLogSince(t, offset)
+ if len(lines) == 0 {
+ t.Fatal("expected the failure to be logged")
+ }
+ logged := strings.Join(lines, "\n")
+ // The query payload must never reach the log. The service name itself is
+ // logged on purpose because it is the only anchor of the failed lookup, but
+ // it must not appear as part of the request URL.
+ for _, forbidden := range []string{
+ "http://127.0.0.1:1/api/traces",
+ "service=secret-service",
+ "operation=secret-operation",
+ "tags=",
+ "secret-operation",
+ "secret-value",
+ "%7B",
+ } {
+ if strings.Contains(logged, forbidden) {
+ t.Fatalf("expected the log to omit %q, got %s", forbidden, logged)
+ }
+ }
+ // The recorded error must not repeat the request URL either.
+ recordedErrors := traceLogErrorFields(t, lines)
+ if len(recordedErrors) == 0 {
+ t.Fatal("expected the failure to record the error")
+ }
+ for _, value := range recordedErrors {
+ for _, forbidden := range []string{"/api/traces", "secret-service", "secret-operation", "secret-value"} {
+ if strings.Contains(value, forbidden) {
+ t.Fatalf("expected the error to omit %q, got %q", forbidden, value)
+ }
+ }
+ }
+}
+
+func TestJaegerProviderTraceFailureDoesNotLeakTraceID(t *testing.T) {
+ provider := newSearchProvider(t, "http://127.0.0.1:1")
+ offset := traceLogOffset(t)
+ if _, err := provider.GetTraceByID(context.Background(), testTraceID); err == nil {
+ t.Fatal("expected a transport failure")
+ }
+ lines := traceLogSince(t, offset)
+ if len(lines) == 0 {
+ t.Fatal("expected the failure to be logged")
+ }
+ logged := strings.Join(lines, "\n")
+ // The trace id must be masked, the raw identifier must not be logged.
+ if strings.Contains(logged, testTraceID) {
+ t.Fatalf("expected the trace id to be masked, got %s", logged)
+ }
+ if !strings.Contains(logged, maskedTraceID(testTraceID)) {
+ t.Fatalf("expected the masked trace id in the log, got %s", logged)
+ }
+ for _, value := range traceLogErrorFields(t, lines) {
+ for _, forbidden := range []string{testTraceID, "/api/traces"} {
+ if strings.Contains(value, forbidden) {
+ t.Fatalf("expected the error to omit %q, got %q", forbidden, value)
+ }
+ }
+ }
+}
+
+// The tool result reaches the model context, so it keeps the transport detail of
+// the upstream failure: the model has to be able to answer which lookup failed
+// and why. The request URL must not appear in it, neither in the tool result nor
+// in the failure log: it repeats the configured endpoint and carries the
+// retrieval conditions, and an endpoint may be configured with a credential.
+func TestJaegerProviderToolResultKeepsTransportDetailButNotTheURL(t *testing.T) {
+ provider := newSearchProvider(t, "http://127.0.0.1:1")
+ offset := traceLogOffset(t)
+ _, err := provider.GetTraceByID(context.Background(), testTraceID)
+ if err == nil {
+ t.Fatal("expected a transport failure")
+ }
+ // The trace identifier may reappear in the tool result: the model built it
+ // itself and it has to be able to report the failed lookup back to the user.
+ if !strings.Contains(err.Error(), testTraceID) {
+ t.Fatalf("expected the tool result to keep the trace identifier: %s", err.Error())
+ }
+ // The request URL must not: it repeats the endpoint path and the query
+ // string. The "dial tcp host:port" form of the inner error stays, it names
+ // the host that refused the connection without repeating the payload.
+ if strings.Contains(err.Error(), "/api/traces/") {
+ t.Fatalf("expected the tool result to omit the request URL: %s", err.Error())
+ }
+ if !strings.Contains(err.Error(), "connection refused") {
+ t.Fatalf("expected the tool result to keep the diagnostic detail: %s", err.Error())
+ }
+
+ // The log must not contain the request URL and must not repeat the raw trace
+ // identifier.
+ lines := traceLogSince(t, offset)
+ if len(lines) == 0 {
+ t.Fatal("expected the failure to be logged")
+ }
+ logged := strings.Join(lines, "\n")
+ if strings.Contains(logged, testTraceID) {
+ t.Fatalf("expected the log to mask the trace identifier, got %s", logged)
+ }
+ if strings.Contains(logged, "/api/traces") {
+ t.Fatalf("expected the log to omit the request URL, got %s", logged)
+ }
+ for _, value := range traceLogErrorFields(t, lines) {
+ for _, forbidden := range []string{testTraceID, "/api/traces"} {
+ if strings.Contains(value, forbidden) {
+ t.Fatalf("expected the error to omit %q, got %q", forbidden, value)
+ }
+ }
+ }
+}
diff --git a/pkg/mcp/tools/trace/jaeger_test.go b/pkg/mcp/tools/trace/jaeger_test.go
index 8132b611b..5668bada2 100644
--- a/pkg/mcp/tools/trace/jaeger_test.go
+++ b/pkg/mcp/tools/trace/jaeger_test.go
@@ -21,6 +21,7 @@ import (
"context"
"net/http"
"net/http/httptest"
+ "strings"
"testing"
observabilitycfg "github.com/apache/dubbo-admin/pkg/config/observability"
@@ -97,3 +98,32 @@ func TestNormalizeJaegerTraceTruncatesSpans(t *testing.T) {
t.Fatalf("expected trace truncation, got %#v", result)
}
}
+
+func TestNewJaegerProviderRejectsEndpointWithQuery(t *testing.T) {
+ // A credential in the endpoint query would be carried into every request URL
+ // and into the transport error of a failed lookup, which reaches the tool
+ // result and the model context. Credentials belong in bearerToken.
+ const endpoint = "https://jaeger.example?api_key=super-secret"
+ _, err := newJaegerProvider(observabilitycfg.TraceProviderConfig{
+ Name: "jaeger-main", Type: observabilitycfg.TraceProviderJaeger, Endpoint: endpoint,
+ })
+ if err == nil {
+ t.Fatal("expected an endpoint with a query string to be rejected")
+ }
+ // The rejection message must not repeat the endpoint, because the endpoint
+ // is exactly what holds the secret.
+ if strings.Contains(err.Error(), "super-secret") {
+ t.Fatalf("expected the rejection to omit the endpoint: %s", err.Error())
+ }
+}
+
+func TestNewJaegerProviderRejectsInvalidEndpoint(t *testing.T) {
+ for _, endpoint := range []string{"jaeger.example:16686", "ftp://jaeger.example", "https://", ""} {
+ _, err := newJaegerProvider(observabilitycfg.TraceProviderConfig{
+ Name: "jaeger-main", Type: observabilitycfg.TraceProviderJaeger, Endpoint: endpoint,
+ })
+ if err == nil {
+ t.Fatalf("expected the endpoint %q to be rejected", endpoint)
+ }
+ }
+}
diff --git a/pkg/mcp/tools/trace/model.go b/pkg/mcp/tools/trace/model.go
index 78d07ce60..170a52e28 100644
--- a/pkg/mcp/tools/trace/model.go
+++ b/pkg/mcp/tools/trace/model.go
@@ -17,6 +17,8 @@
package trace
+import "time"
+
// Trace is a backend-independent, bounded representation of a distributed trace.
type Trace struct {
TraceID string `json:"traceId"`
@@ -53,3 +55,161 @@ type Event struct {
Timestamp string `json:"timestamp"`
Attributes map[string]string `json:"attributes,omitempty"`
}
+
+// TraceSearchQuery bounds a provider-side trace lookup.
+type TraceSearchQuery struct {
+ Service string
+ Operation string
+ TraceID string
+ Tags map[string]string
+ StartTime time.Time
+ EndTime time.Time
+ Limit int
+ MinDuration time.Duration
+}
+
+// TraceSummary is a bounded per-trace overview produced by trace search.
+type TraceSummary struct {
+ TraceID string `json:"traceId"`
+ RootService string `json:"rootService,omitempty"`
+ RootOperation string `json:"rootOperation,omitempty"`
+ StartTime string `json:"startTime"`
+ DurationMicros int64 `json:"durationMicros"`
+ SpanCount int `json:"spanCount"`
+ ErrorSpanCount int `json:"errorSpanCount"`
+ ServiceCount int `json:"serviceCount"`
+ HasError bool `json:"hasError"`
+ TraceKind string `json:"traceKind,omitempty"` // "genai" when the trace carries gen_ai.* attributes
+}
+
+// TraceSearchResult aggregates the outcome of a trace search.
+type TraceSearchResult struct {
+ Traces []TraceSummary `json:"traces"`
+ Total int `json:"total"`
+ Truncated bool `json:"truncated"`
+ SourceEngine string `json:"sourceEngine"`
+}
+
+// CriticalPathEntry is one hop of the latency-determining root-to-leaf chain.
+type CriticalPathEntry struct {
+ SpanID string `json:"spanId"`
+ ServiceName string `json:"serviceName"`
+ OperationName string `json:"operationName"`
+ StartTime string `json:"startTime"`
+ DurationMicros int64 `json:"durationMicros"`
+ SelfMicros int64 `json:"selfMicros"`
+ WaitMicros int64 `json:"waitMicros"`
+}
+
+// ServiceLatency aggregates the spans contributed by one service.
+type ServiceLatency struct {
+ ServiceName string `json:"serviceName"`
+ SpanCount int `json:"spanCount"`
+ ErrorSpanCount int `json:"errorSpanCount"`
+ TotalMicros int64 `json:"totalMicros"`
+ MaxMicros int64 `json:"maxMicros"`
+ AvgMicros int64 `json:"avgMicros"`
+ SharePercent float64 `json:"sharePercent"`
+}
+
+// SlowSpan is a span ranked by the time it consumed on its own.
+type SlowSpan struct {
+ SpanID string `json:"spanId"`
+ ServiceName string `json:"serviceName"`
+ OperationName string `json:"operationName"`
+ DurationMicros int64 `json:"durationMicros"`
+ SharePercent float64 `json:"sharePercent"`
+ OnCriticalPath bool `json:"onCriticalPath"`
+}
+
+// ErrorSpan describes a failed span together with its root-cause verdict.
+type ErrorSpan struct {
+ SpanID string `json:"spanId"`
+ ServiceName string `json:"serviceName"`
+ OperationName string `json:"operationName"`
+ ErrorType string `json:"errorType,omitempty"`
+ ErrorMessage string `json:"errorMessage,omitempty"`
+ Status string `json:"status"`
+ RootCause bool `json:"rootCause"`
+}
+
+// DependencyEdge is a cross-service call observed inside a single trace.
+type DependencyEdge struct {
+ CallerService string `json:"callerService"`
+ CalleeService string `json:"calleeService"`
+ SpanCount int `json:"spanCount"`
+ TotalMicros int64 `json:"totalMicros"`
+ ErrorCount int `json:"errorCount"`
+}
+
+// GenAIModelCall describes one generative model invocation observed in a trace.
+type GenAIModelCall struct {
+ SpanID string `json:"spanId"`
+ ServiceName string `json:"serviceName"`
+ OperationName string `json:"operationName"`
+ Operation string `json:"operation,omitempty"`
+ ProviderName string `json:"providerName,omitempty"`
+ RequestModel string `json:"requestModel,omitempty"`
+ ResponseModel string `json:"responseModel,omitempty"`
+ InputTokens int `json:"inputTokens,omitempty"`
+ OutputTokens int `json:"outputTokens,omitempty"`
+ TotalTokens int `json:"totalTokens,omitempty"`
+ DurationMicros int64 `json:"durationMicros"`
+ Error bool `json:"error"`
+}
+
+// GenAIToolCall describes one tool invocation observed in a trace.
+type GenAIToolCall struct {
+ SpanID string `json:"spanId"`
+ ServiceName string `json:"serviceName"`
+ ToolName string `json:"toolName,omitempty"`
+ ToolCallID string `json:"toolCallId,omitempty"`
+ DurationMicros int64 `json:"durationMicros"`
+ Error bool `json:"error"`
+}
+
+// GenAIAgent describes one agent-scoped span observed in a trace.
+type GenAIAgent struct {
+ SpanID string `json:"spanId"`
+ ServiceName string `json:"serviceName"`
+ AgentName string `json:"agentName,omitempty"`
+ AgentID string `json:"agentId,omitempty"`
+ DurationMicros int64 `json:"durationMicros"`
+ Error bool `json:"error"`
+}
+
+// GenAISummary aggregates the generative AI footprint of a trace.
+type GenAISummary struct {
+ Present bool `json:"present"`
+ ConversationIDs []string `json:"conversationIds,omitempty"`
+ Providers []string `json:"providers,omitempty"`
+ Models []string `json:"models,omitempty"`
+ InputTokens int `json:"inputTokens"`
+ OutputTokens int `json:"outputTokens"`
+ TotalTokens int `json:"totalTokens"`
+ ModelCallCount int `json:"modelCallCount"`
+ ToolCallCount int `json:"toolCallCount"`
+ AgentCount int `json:"agentCount"`
+ ModelCalls []GenAIModelCall `json:"modelCalls,omitempty"`
+ ToolCalls []GenAIToolCall `json:"toolCalls,omitempty"`
+ Agents []GenAIAgent `json:"agents,omitempty"`
+ Truncated bool `json:"truncated"`
+}
+
+// TraceAnalysis is the structured diagnosis of a single trace.
+type TraceAnalysis struct {
+ TraceID string `json:"traceId"`
+ Summary string `json:"summary"`
+ StartTime string `json:"startTime"`
+ DurationMicros int64 `json:"durationMicros"`
+ SpanCount int `json:"spanCount"`
+ ServiceCount int `json:"serviceCount"`
+ ErrorSpanCount int `json:"errorSpanCount"`
+ CriticalPath []CriticalPathEntry `json:"criticalPath,omitempty"`
+ ServiceBreakdown []ServiceLatency `json:"serviceBreakdown,omitempty"`
+ SlowSpans []SlowSpan `json:"slowSpans,omitempty"`
+ ErrorSpans []ErrorSpan `json:"errorSpans,omitempty"`
+ Dependencies []DependencyEdge `json:"dependencies,omitempty"`
+ GenAI *GenAISummary `json:"genai,omitempty"`
+ Truncated bool `json:"truncated"`
+}
diff --git a/pkg/mcp/tools/trace/provider.go b/pkg/mcp/tools/trace/provider.go
index 863256af2..decf86c3a 100644
--- a/pkg/mcp/tools/trace/provider.go
+++ b/pkg/mcp/tools/trace/provider.go
@@ -27,6 +27,9 @@ import (
// Provider hides backend wire formats behind a stable trace lookup contract.
type Provider interface {
GetTraceByID(ctx context.Context, traceID string) (*Trace, error)
+ // SearchTraces discovers candidate traces without knowing a trace identifier up
+ // front. Implementations must keep the result bounded.
+ SearchTraces(ctx context.Context, query TraceSearchQuery) (*TraceSearchResult, error)
}
func newProvider(cfg observabilitycfg.TraceProviderConfig) (Provider, error) {
diff --git a/pkg/mcp/tools/trace/search.go b/pkg/mcp/tools/trace/search.go
new file mode 100644
index 000000000..cca6b565a
--- /dev/null
+++ b/pkg/mcp/tools/trace/search.go
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package trace
+
+import (
+ "fmt"
+ "strings"
+ "time"
+
+ consolectx "github.com/apache/dubbo-admin/pkg/console/context"
+ "github.com/apache/dubbo-admin/pkg/mcp/common"
+)
+
+const (
+ // defaultSearchLimit is the number of traces requested from the provider when the
+ // caller does not ask for a specific amount.
+ defaultSearchLimit = 20
+ // maxSearchLimit bounds the limit that is forwarded to the provider.
+ maxSearchLimit = 100
+ // maxSearchTraceSummaries bounds how many per-trace summaries are kept locally.
+ // The provider may return more traces than the agent can consume.
+ maxSearchTraceSummaries = 100
+)
+
+// SearchTraces discovers candidate traces for a service so the agent can start an
+// investigation without asking the user for a trace identifier.
+func SearchTraces(ctx consolectx.Context, args map[string]any) (*common.ToolResult, error) {
+ query, err := buildTraceSearchQuery(args)
+ if err != nil {
+ return common.ErrorResult(err), nil
+ }
+ providerConfig, err := providerConfigFromContext(ctx, common.NewArgsHelper(args).GetString("provider", ""))
+ if err != nil {
+ return common.ErrorResult(err), nil
+ }
+ provider, err := newProvider(providerConfig)
+ if err != nil {
+ return common.ErrorResult(err), nil
+ }
+ result, err := provider.SearchTraces(traceRequestContext(ctx), query)
+ if err != nil {
+ return common.ErrorResult(err), nil
+ }
+ return common.JsonResult(result)
+}
+
+// SearchProperties returns the MCP input schema properties for trace search.
+func SearchProperties() map[string]common.PropertyDef {
+ return map[string]common.PropertyDef{
+ "service": {Type: "string", Description: "Service name used as the trace search anchor"},
+ "operation": {Type: "string", Description: "Optional root operation name filter"},
+ "traceId": {Type: "string", Description: "Optional Jaeger 64-bit or 128-bit hexadecimal TraceID anchor"},
+ "tags": {Type: "string", Description: "Optional tag filter formatted as key=value pairs separated by commas"},
+ "startTime": {Type: "string", Description: "Optional RFC3339 lower bound of the search window"},
+ "endTime": {Type: "string", Description: "Optional RFC3339 upper bound of the search window"},
+ "minDurationMs": {Type: "integer", Description: "Optional minimum trace duration in milliseconds"},
+ "limit": {Type: "integer", Description: "Maximum number of traces to request", Default: defaultSearchLimit},
+ "provider": {Type: "string", Description: "Optional trace provider name; defaults to defaultProvider"},
+ }
+}
+
+// buildTraceSearchQuery validates the tool arguments and converts them into the
+// provider side query.
+func buildTraceSearchQuery(args map[string]any) (TraceSearchQuery, error) {
+ helper := common.NewArgsHelper(args)
+ service := strings.TrimSpace(helper.GetString("service", ""))
+ if service == "" {
+ return TraceSearchQuery{}, fmt.Errorf("service is required for trace search")
+ }
+ query := TraceSearchQuery{Service: service}
+ query.Operation = strings.TrimSpace(helper.GetString("operation", ""))
+
+ if traceID := strings.TrimSpace(helper.GetString("traceId", "")); traceID != "" {
+ if !traceIDPattern.MatchString(traceID) {
+ return TraceSearchQuery{}, fmt.Errorf("traceId must be a 16 or 32 character hexadecimal value")
+ }
+ query.TraceID = traceID
+ }
+
+ tags, err := parseSearchTags(helper.GetString("tags", ""))
+ if err != nil {
+ return TraceSearchQuery{}, err
+ }
+ query.Tags = tags
+
+ startTime, err := parseSearchTime(helper.GetString("startTime", ""), "startTime")
+ if err != nil {
+ return TraceSearchQuery{}, err
+ }
+ query.StartTime = startTime
+
+ endTime, err := parseSearchTime(helper.GetString("endTime", ""), "endTime")
+ if err != nil {
+ return TraceSearchQuery{}, err
+ }
+ query.EndTime = endTime
+ if !query.StartTime.IsZero() && !query.EndTime.IsZero() && query.EndTime.Before(query.StartTime) {
+ return TraceSearchQuery{}, fmt.Errorf("endTime must not be earlier than startTime")
+ }
+
+ minDurationMs := helper.GetInt("minDurationMs", 0)
+ if minDurationMs < 0 {
+ return TraceSearchQuery{}, fmt.Errorf("minDurationMs must not be negative")
+ }
+ query.MinDuration = time.Duration(minDurationMs) * time.Millisecond
+
+ limit := helper.GetInt("limit", defaultSearchLimit)
+ if limit <= 0 {
+ limit = defaultSearchLimit
+ }
+ if limit > maxSearchLimit {
+ limit = maxSearchLimit
+ }
+ query.Limit = limit
+ return query, nil
+}
+
+// parseSearchTags converts the "k=v,k2=v2" filter syntax into a tag map.
+// A repeated key keeps the last value, which matches the object semantics of the
+// provider side tag filter.
+func parseSearchTags(raw string) (map[string]string, error) {
+ trimmed := strings.TrimSpace(raw)
+ if trimmed == "" {
+ return nil, nil
+ }
+ result := map[string]string{}
+ for _, pair := range strings.Split(trimmed, ",") {
+ pair = strings.TrimSpace(pair)
+ if pair == "" {
+ continue
+ }
+ key, value, found := strings.Cut(pair, "=")
+ key = strings.TrimSpace(key)
+ if !found || key == "" {
+ return nil, fmt.Errorf("tags must be formatted as key=value pairs separated by commas")
+ }
+ result[key] = strings.TrimSpace(value)
+ }
+ if len(result) == 0 {
+ return nil, nil
+ }
+ return result, nil
+}
+
+// parseSearchTime converts an RFC3339 timestamp into a time value. An empty input
+// yields the zero time, which the provider treats as "unbounded".
+func parseSearchTime(raw string, name string) (time.Time, error) {
+ trimmed := strings.TrimSpace(raw)
+ if trimmed == "" {
+ return time.Time{}, nil
+ }
+ parsed, err := time.Parse(time.RFC3339, trimmed)
+ if err != nil {
+ return time.Time{}, fmt.Errorf("%s must be an RFC3339 timestamp: %w", name, err)
+ }
+ return parsed, nil
+}
diff --git a/pkg/mcp/tools/trace/search_test.go b/pkg/mcp/tools/trace/search_test.go
new file mode 100644
index 000000000..9b7406659
--- /dev/null
+++ b/pkg/mcp/tools/trace/search_test.go
@@ -0,0 +1,500 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package trace
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "strconv"
+ "strings"
+ "testing"
+ "time"
+
+ observabilitycfg "github.com/apache/dubbo-admin/pkg/config/observability"
+)
+
+const testSearchTraceID = "aabbccddeeff00112233445566778899"
+
+func newSearchProvider(t *testing.T, endpoint string) *jaegerProvider {
+ t.Helper()
+ provider, err := newJaegerProvider(observabilitycfg.TraceProviderConfig{
+ Name: "jaeger-main", Type: observabilitycfg.TraceProviderJaeger, Endpoint: endpoint, BearerToken: "secret", Tenant: "tenant-a",
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ return provider
+}
+
+func TestJaegerProviderSearchTracesBuildsRequest(t *testing.T) {
+ var captured *http.Request
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ captured = r
+ _, _ = w.Write([]byte(`{"data":[{"traceID":"` + testSearchTraceID + `","spans":[{"traceID":"` + testSearchTraceID + `","spanID":"01","operationName":"root","references":[],"startTime":2000000,"duration":3000,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}]}`))
+ }))
+ defer server.Close()
+
+ provider := newSearchProvider(t, server.URL)
+ start := time.Unix(1700000000, 0).UTC()
+ end := time.Unix(1700003600, 0).UTC()
+ result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{
+ Service: "shop-order",
+ Operation: "checkout",
+ TraceID: testSearchTraceID,
+ Tags: map[string]string{"rpc.system": "apache_dubbo"},
+ StartTime: start,
+ EndTime: end,
+ Limit: 17,
+ MinDuration: 250 * time.Millisecond,
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if captured == nil {
+ t.Fatal("expected the provider to issue a request")
+ }
+ if captured.URL.Path != "/api/traces" {
+ t.Fatalf("unexpected path %s", captured.URL.Path)
+ }
+ if captured.Header.Get("Authorization") != "Bearer secret" || captured.Header.Get(tenantHeader) != "tenant-a" {
+ t.Fatalf("missing auth headers: %#v", captured.Header)
+ }
+ query := captured.URL.Query()
+ if query.Get("service") != "shop-order" || query.Get("operation") != "checkout" {
+ t.Fatalf("unexpected service filter: %v", query)
+ }
+ if query.Get("traceID") != testSearchTraceID {
+ t.Fatalf("unexpected trace id filter: %v", query)
+ }
+ if query.Get("tags") != `{"rpc.system":"apache_dubbo"}` {
+ t.Fatalf("unexpected tags filter: %v", query)
+ }
+ if query.Get("start") != "1700000000000000" || query.Get("end") != "1700003600000000" {
+ t.Fatalf("unexpected time window: %v", query)
+ }
+ // Jaeger parses minDuration with time.ParseDuration, so the wire format is a
+ // duration string and not a bare microsecond count.
+ minDuration := query.Get("minDuration")
+ if minDuration != "250ms" {
+ t.Fatalf("unexpected min duration: %v", query)
+ }
+ // The stub does not validate the parameter like Jaeger does, so the value is
+ // parsed here to keep a unit-less microsecond count from slipping through.
+ parsed, err := time.ParseDuration(minDuration)
+ if err != nil {
+ t.Fatalf("min duration %q is not a duration Jaeger can parse: %v", minDuration, err)
+ }
+ if parsed != 250*time.Millisecond {
+ t.Fatalf("unexpected parsed min duration: %v", parsed)
+ }
+ // One trace more than the caller asked for is requested, so a full result
+ // proves that more traces match than were returned.
+ if query.Get("limit") != "18" {
+ t.Fatalf("unexpected limit: %v", query)
+ }
+ if result.Total != 1 || len(result.Traces) != 1 || result.SourceEngine != "jaeger" {
+ t.Fatalf("unexpected search result: %#v", result)
+ }
+ summary := result.Traces[0]
+ if summary.TraceID != testSearchTraceID || summary.RootService != "shop-order" || summary.RootOperation != "root" {
+ t.Fatalf("unexpected summary: %#v", summary)
+ }
+ if summary.SpanCount != 1 || summary.ServiceCount != 1 || summary.ErrorSpanCount != 0 || summary.HasError {
+ t.Fatalf("unexpected summary counters: %#v", summary)
+ }
+ if summary.DurationMicros != 3000 {
+ t.Fatalf("unexpected summary duration: %#v", summary)
+ }
+}
+
+func TestJaegerProviderSearchTracesOmitsEmptyParameters(t *testing.T) {
+ var captured *http.Request
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ captured = r
+ _, _ = w.Write([]byte(`{"data":[]}`))
+ }))
+ defer server.Close()
+
+ provider := newSearchProvider(t, server.URL)
+ result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ query := captured.URL.Query()
+ for _, key := range []string{"operation", "traceID", "tags", "start", "end", "minDuration", "limit"} {
+ if query.Has(key) {
+ t.Fatalf("expected parameter %q to be omitted, got %v", key, query)
+ }
+ }
+ if query.Get("service") != "shop-order" {
+ t.Fatalf("unexpected service filter: %v", query)
+ }
+ if result.Total != 0 || len(result.Traces) != 0 || result.Truncated {
+ t.Fatalf("expected an empty but successful result, got %#v", result)
+ }
+}
+
+func TestJaegerProviderSearchTracesRequiresService(t *testing.T) {
+ provider := newSearchProvider(t, "http://127.0.0.1:1")
+ _, err := provider.SearchTraces(context.Background(), TraceSearchQuery{})
+ if err == nil {
+ t.Fatal("expected a missing service error")
+ }
+ if !strings.Contains(err.Error(), "service") {
+ t.Fatalf("expected the error to name the missing service, got %v", err)
+ }
+}
+
+func TestJaegerProviderSearchTracesSummarizesOrphanRoot(t *testing.T) {
+ // The parent of the span is not part of the trace, so the span is an orphan
+ // and roots the trace on its own. The summary and the analyzer have to agree
+ // on that root.
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(`{"data":[{"traceID":"` + testSearchTraceID + `","spans":[{"traceID":"` + testSearchTraceID + `","spanID":"02","operationName":"orphan","references":[{"refType":"CHILD_OF","traceID":"` + testSearchTraceID + `","spanID":"missing"}],"startTime":2000000,"duration":3000,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}]}`))
+ }))
+ defer server.Close()
+
+ provider := newSearchProvider(t, server.URL)
+ result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(result.Traces) != 1 {
+ t.Fatalf("unexpected number of traces: %#v", result.Traces)
+ }
+ summary := result.Traces[0]
+ if summary.RootService != "shop-order" || summary.RootOperation != "orphan" {
+ t.Fatalf("expected the orphan to be reported as the root: %#v", summary)
+ }
+}
+
+func TestJaegerProviderSearchTracesDetectsGenAIKind(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(`{"data":[{"traceID":"` + testSearchTraceID + `","spans":[{"traceID":"` + testSearchTraceID + `","spanID":"01","operationName":"chat","references":[],"startTime":2000000,"duration":3000,"tags":[{"key":"gen_ai.operation.name","type":"string","value":"chat"}],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"dubbo-admin-sre-agent","tags":[]}}}]}`))
+ }))
+ defer server.Close()
+
+ provider := newSearchProvider(t, server.URL)
+ result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "dubbo-admin-sre-agent"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(result.Traces) != 1 || result.Traces[0].TraceKind != "genai" {
+ t.Fatalf("expected a genai trace kind, got %#v", result.Traces)
+ }
+}
+
+func TestJaegerProviderSearchTracesSortsByStartTime(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ // The traces are returned out of order and use timestamps whose RFC3339
+ // rendering would sort differently than the numeric microsecond value.
+ _, _ = w.Write([]byte(`{"data":[
+ {"traceID":"cccccccccccccccccccccccccccccccc","spans":[{"spanID":"01","operationName":"late","references":[],"startTime":9000000,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}},
+ {"traceID":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","spans":[{"spanID":"01","operationName":"early","references":[],"startTime":1000000,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}},
+ {"traceID":"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","spans":[{"spanID":"01","operationName":"middle","references":[],"startTime":5000000,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}
+ ]}`))
+ }))
+ defer server.Close()
+
+ provider := newSearchProvider(t, server.URL)
+ result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ expected := []string{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "cccccccccccccccccccccccccccccccc"}
+ if len(result.Traces) != len(expected) {
+ t.Fatalf("unexpected number of traces: %#v", result.Traces)
+ }
+ for i, traceID := range expected {
+ if result.Traces[i].TraceID != traceID {
+ t.Fatalf("unexpected trace order: %v", result.Traces)
+ }
+ }
+}
+
+func TestJaegerProviderSearchTracesTruncatesSummaries(t *testing.T) {
+ spans := make([]string, 0, maxSearchTraceSummaries+5)
+ for i := 0; i < maxSearchTraceSummaries+5; i++ {
+ spans = append(spans, `{"traceID":"trace`+strconv.Itoa(i)+`","spans":[{"spanID":"01","operationName":"op","references":[],"startTime":`+strconv.Itoa(1000000+i)+`,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}`)
+ }
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(`{"data":[` + strings.Join(spans, ",") + `]}`))
+ }))
+ defer server.Close()
+
+ provider := newSearchProvider(t, server.URL)
+ result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.Truncated {
+ t.Fatal("expected the result to be marked as truncated")
+ }
+ if len(result.Traces) != maxSearchTraceSummaries {
+ t.Fatalf("unexpected number of summaries: %d", len(result.Traces))
+ }
+ if result.Total != maxSearchTraceSummaries+5 {
+ t.Fatalf("expected the backend total to be preserved, got %d", result.Total)
+ }
+}
+
+func TestJaegerProviderSearchTracesReportsUpstreamFailure(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ w.WriteHeader(http.StatusInternalServerError)
+ _, _ = w.Write([]byte("upstream exploded"))
+ }))
+ defer server.Close()
+
+ provider := newSearchProvider(t, server.URL)
+ result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"})
+ if err == nil {
+ t.Fatal("expected an upstream error")
+ }
+ if result != nil {
+ t.Fatalf("expected no result on failure, got %#v", result)
+ }
+ if !strings.Contains(err.Error(), "500") {
+ t.Fatalf("expected the status code in the error, got %v", err)
+ }
+}
+
+func TestJaegerProviderSearchTracesReportsEnvelopeErrors(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(`{"data":[],"errors":[{"code":500,"msg":"query rejected"}]}`))
+ }))
+ defer server.Close()
+
+ provider := newSearchProvider(t, server.URL)
+ if _, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"}); err == nil {
+ t.Fatal("expected an envelope error")
+ }
+}
+
+func TestBuildTraceSearchQuery(t *testing.T) {
+ query, err := buildTraceSearchQuery(map[string]any{
+ "service": "shop-order",
+ "operation": "checkout",
+ "traceId": testSearchTraceID,
+ "tags": "rpc.system=apache_dubbo, http.status_code=500",
+ "startTime": "2026-10-24T10:00:00Z",
+ "endTime": "2026-10-24T11:00:00Z",
+ "minDurationMs": 250,
+ "limit": 33,
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if query.Service != "shop-order" || query.Operation != "checkout" || query.TraceID != testSearchTraceID {
+ t.Fatalf("unexpected query: %#v", query)
+ }
+ if len(query.Tags) != 2 || query.Tags["rpc.system"] != "apache_dubbo" || query.Tags["http.status_code"] != "500" {
+ t.Fatalf("unexpected tags: %#v", query.Tags)
+ }
+ if query.StartTime.UTC().Format(time.RFC3339) != "2026-10-24T10:00:00Z" {
+ t.Fatalf("unexpected start time: %v", query.StartTime)
+ }
+ if query.EndTime.UTC().Format(time.RFC3339) != "2026-10-24T11:00:00Z" {
+ t.Fatalf("unexpected end time: %v", query.EndTime)
+ }
+ if query.MinDuration != 250*time.Millisecond {
+ t.Fatalf("unexpected min duration: %v", query.MinDuration)
+ }
+ if query.Limit != 33 {
+ t.Fatalf("unexpected limit: %d", query.Limit)
+ }
+}
+
+func TestBuildTraceSearchQueryDefaultsAndLimits(t *testing.T) {
+ query, err := buildTraceSearchQuery(map[string]any{"service": "shop-order"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if query.Limit != defaultSearchLimit {
+ t.Fatalf("unexpected default limit: %d", query.Limit)
+ }
+ if !query.StartTime.IsZero() || !query.EndTime.IsZero() || query.MinDuration != 0 || query.Tags != nil {
+ t.Fatalf("expected empty optional filters, got %#v", query)
+ }
+
+ query, err = buildTraceSearchQuery(map[string]any{"service": "shop-order", "limit": float64(1000)})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if query.Limit != maxSearchLimit {
+ t.Fatalf("expected the limit to be capped, got %d", query.Limit)
+ }
+
+ query, err = buildTraceSearchQuery(map[string]any{"service": "shop-order", "limit": float64(-5)})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if query.Limit != defaultSearchLimit {
+ t.Fatalf("expected a negative limit to fall back, got %d", query.Limit)
+ }
+}
+
+func TestBuildTraceSearchQueryRejectsInvalidArguments(t *testing.T) {
+ tests := []struct {
+ name string
+ args map[string]any
+ }{
+ {name: "missing service", args: map[string]any{}},
+ {name: "blank service", args: map[string]any{"service": " "}},
+ {name: "invalid trace id", args: map[string]any{"service": "shop-order", "traceId": "not-hex"}},
+ {name: "malformed tags", args: map[string]any{"service": "shop-order", "tags": "rpc.system"}},
+ {name: "tags without key", args: map[string]any{"service": "shop-order", "tags": "=value"}},
+ {name: "invalid start time", args: map[string]any{"service": "shop-order", "startTime": "2026-10-24"}},
+ {name: "invalid end time", args: map[string]any{"service": "shop-order", "endTime": "yesterday"}},
+ {name: "reversed window", args: map[string]any{"service": "shop-order", "startTime": "2026-10-24T11:00:00Z", "endTime": "2026-10-24T10:00:00Z"}},
+ {name: "negative duration", args: map[string]any{"service": "shop-order", "minDurationMs": float64(-1)}},
+ }
+ for _, test := range tests {
+ t.Run(test.name, func(t *testing.T) {
+ if _, err := buildTraceSearchQuery(test.args); err == nil {
+ t.Fatal("expected a validation error")
+ }
+ })
+ }
+}
+
+func TestParseSearchTags(t *testing.T) {
+ tags, err := parseSearchTags("")
+ if err != nil || tags != nil {
+ t.Fatalf("expected no tags, got %#v (%v)", tags, err)
+ }
+ tags, err = parseSearchTags(" a = 1 , b=2 ")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if tags["a"] != "1" || tags["b"] != "2" {
+ t.Fatalf("unexpected tags: %#v", tags)
+ }
+ tags, err = parseSearchTags("a=1,a=2")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if tags["a"] != "2" {
+ t.Fatalf("expected the last value to win, got %#v", tags)
+ }
+ tags, err = parseSearchTags("a=b=c")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if tags["a"] != "b=c" {
+ t.Fatalf("unexpected tags: %#v", tags)
+ }
+ if _, err := parseSearchTags("broken"); err == nil {
+ t.Fatal("expected a malformed tag error")
+ }
+}
+
+func TestSearchPropertiesExposeRequiredParameters(t *testing.T) {
+ properties := SearchProperties()
+ for _, name := range []string{"service", "operation", "traceId", "tags", "startTime", "endTime", "minDurationMs", "limit", "provider"} {
+ if _, ok := properties[name]; !ok {
+ t.Fatalf("search properties are missing %q", name)
+ }
+ }
+ if properties["minDurationMs"].Type != "integer" {
+ t.Fatalf("unexpected minDurationMs type: %s", properties["minDurationMs"].Type)
+ }
+}
+
+func TestSearchTracesToolResultIsSerializable(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(`{"data":[{"traceID":"` + testSearchTraceID + `","spans":[{"traceID":"` + testSearchTraceID + `","spanID":"01","operationName":"chat","references":[],"startTime":2000000,"duration":3000,"tags":[{"key":"gen_ai.operation.name","type":"string","value":"chat"}],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}]}`))
+ }))
+ defer server.Close()
+
+ // The handler must produce a payload that survives a round trip.
+ result, err := SearchTraces(newTraceToolTestContext(server.URL), map[string]any{"service": "shop-order"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if result.IsError {
+ t.Fatalf("unexpected error result: %s", result.Content[0].Text)
+ }
+ var payload TraceSearchResult
+ if err := json.Unmarshal([]byte(result.Content[0].Text), &payload); err != nil {
+ t.Fatal(err)
+ }
+ if payload.Total != 1 || len(payload.Traces) != 1 {
+ t.Fatalf("unexpected payload: %#v", payload)
+ }
+ if payload.Traces[0].TraceID != testSearchTraceID {
+ t.Fatalf("unexpected summary: %#v", payload.Traces[0])
+ }
+ if !strings.Contains(result.Content[0].Text, `"traceId":"`+testSearchTraceID+`"`) {
+ t.Fatalf("unexpected payload: %s", result.Content[0].Text)
+ }
+ if !strings.Contains(result.Content[0].Text, `"traceKind":"genai"`) {
+ t.Fatalf("expected the genai kind in the payload: %s", result.Content[0].Text)
+ }
+ // The tool result envelope must be serializable as well.
+ if _, err := json.Marshal(result); err != nil {
+ t.Fatal(err)
+ }
+}
+
+func TestJaegerProviderSearchTracesReportsTruncationAtTheBackendLimit(t *testing.T) {
+ // The provider is asked for one trace more than the caller wants, so a result
+ // that reached the limit proves that more traces match than were returned.
+ // Without the extra trace a full result would look complete.
+ available := 30
+ traces := make([]string, 0, available)
+ for i := 0; i < available; i++ {
+ traces = append(traces, `{"traceID":"trace`+strconv.Itoa(i)+`","spans":[{"spanID":"01","operationName":"op","references":[],"startTime":`+strconv.Itoa(1000000+i)+`,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}`)
+ }
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
+ if limit > len(traces) {
+ limit = len(traces)
+ }
+ _, _ = w.Write([]byte(`{"data":[` + strings.Join(traces[:limit], ",") + `]}`))
+ }))
+ defer server.Close()
+ provider := newSearchProvider(t, server.URL)
+
+ // The backend has fewer traces than the caller asked for, so the result is
+ // complete even though the request asked for one trace more.
+ result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order", Limit: 50})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if result.Truncated {
+ t.Fatal("did not expect the result to be truncated")
+ }
+ if len(result.Traces) != available || result.Total != available {
+ t.Fatalf("unexpected result: %d traces, total %d", len(result.Traces), result.Total)
+ }
+
+ // The backend reached the requested limit, so more traces match than were
+ // returned and the agent has to be told about it.
+ result, err = provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order", Limit: 20})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.Truncated {
+ t.Fatal("expected the result to be marked as truncated")
+ }
+ if len(result.Traces) != 20 || result.Total != 20 {
+ t.Fatalf("unexpected result: %d traces, total %d", len(result.Traces), result.Total)
+ }
+}
diff --git a/pkg/mcp/tools/trace/tools.go b/pkg/mcp/tools/trace/tools.go
index d59318759..7659e5f32 100644
--- a/pkg/mcp/tools/trace/tools.go
+++ b/pkg/mcp/tools/trace/tools.go
@@ -18,7 +18,6 @@
package trace
import (
- "context"
"fmt"
"regexp"
@@ -44,11 +43,7 @@ func GetTraceByID(ctx consolectx.Context, args map[string]any) (*common.ToolResu
if err != nil {
return common.ErrorResult(err), nil
}
- requestCtx := context.Background()
- if ctx.AppContext() != nil {
- requestCtx = ctx.AppContext()
- }
- result, err := provider.GetTraceByID(requestCtx, traceID)
+ result, err := provider.GetTraceByID(traceRequestContext(ctx), traceID)
if err != nil {
return common.ErrorResult(err), nil
}
diff --git a/pkg/mcp/tools/trace/tools_test.go b/pkg/mcp/tools/trace/tools_test.go
new file mode 100644
index 000000000..0707c649e
--- /dev/null
+++ b/pkg/mcp/tools/trace/tools_test.go
@@ -0,0 +1,231 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package trace
+
+import (
+ "context"
+ "encoding/json"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+
+ "github.com/apache/dubbo-admin/pkg/config/app"
+ "github.com/apache/dubbo-admin/pkg/config/observability"
+ consolectx "github.com/apache/dubbo-admin/pkg/console/context"
+ "github.com/apache/dubbo-admin/pkg/console/counter"
+ "github.com/apache/dubbo-admin/pkg/core/lock"
+ "github.com/apache/dubbo-admin/pkg/core/manager"
+ "github.com/apache/dubbo-admin/pkg/core/versioning"
+)
+
+type traceToolTestContext struct {
+ config app.AdminConfig
+}
+
+func (c *traceToolTestContext) RuleVersioning() *versioning.Service { return nil }
+
+func (c *traceToolTestContext) ResourceManager() manager.ResourceManager { return nil }
+
+func (c *traceToolTestContext) CounterManager() counter.CounterManager { return nil }
+
+func (c *traceToolTestContext) Config() app.AdminConfig { return c.config }
+
+func (c *traceToolTestContext) AppContext() context.Context { return context.Background() }
+
+func (c *traceToolTestContext) LockManager() lock.Lock { return nil }
+
+func newTraceToolTestContext(endpoint string) consolectx.Context {
+ return &traceToolTestContext{
+ config: app.AdminConfig{
+ Observability: &observability.Config{
+ Tracing: &observability.TracingConfig{
+ DefaultProvider: "jaeger-main",
+ Providers: []observability.TraceProviderConfig{{
+ Name: "jaeger-main",
+ Type: observability.TraceProviderJaeger,
+ Endpoint: endpoint,
+ }},
+ },
+ },
+ },
+ }
+}
+
+const traceToolTestPayload = `{"data":[{"traceID":"` + testSearchTraceID + `","spans":[
+ {"traceID":"` + testSearchTraceID + `","spanID":"01","operationName":"checkout","references":[],"startTime":2000000,"duration":100000,"tags":[],"logs":[],"processID":"p1"},
+ {"traceID":"` + testSearchTraceID + `","spanID":"02","operationName":"charge","references":[{"refType":"CHILD_OF","traceID":"` + testSearchTraceID + `","spanID":"01"}],"startTime":2010000,"duration":50000,"tags":[{"key":"error","type":"bool","value":true},{"key":"error.type","type":"string","value":"timeout"}],"logs":[],"processID":"p2"}
+],"processes":{"p1":{"serviceName":"order-service","tags":[]},"p2":{"serviceName":"payment-service","tags":[]}}}]}`
+
+func TestSearchTracesToolReturnsSummaries(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(traceToolTestPayload))
+ }))
+ defer server.Close()
+
+ result, err := SearchTraces(newTraceToolTestContext(server.URL), map[string]any{"service": "order-service"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if result.IsError {
+ t.Fatalf("unexpected error result: %s", result.Content[0].Text)
+ }
+ var payload TraceSearchResult
+ if err := json.Unmarshal([]byte(result.Content[0].Text), &payload); err != nil {
+ t.Fatal(err)
+ }
+ if payload.Total != 1 || len(payload.Traces) != 1 {
+ t.Fatalf("unexpected payload: %#v", payload)
+ }
+ if payload.Traces[0].TraceID != testSearchTraceID || !payload.Traces[0].HasError {
+ t.Fatalf("unexpected summary: %#v", payload.Traces[0])
+ }
+ if payload.Traces[0].ErrorSpanCount != 1 || payload.Traces[0].SpanCount != 2 {
+ t.Fatalf("unexpected summary counters: %#v", payload.Traces[0])
+ }
+}
+
+func TestSearchTracesToolRejectsMissingService(t *testing.T) {
+ result, err := SearchTraces(newTraceToolTestContext("http://127.0.0.1:1"), map[string]any{})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.IsError {
+ t.Fatal("expected an error result")
+ }
+ if !strings.Contains(result.Content[0].Text, "service is required") {
+ t.Fatalf("unexpected error message: %s", result.Content[0].Text)
+ }
+}
+
+func TestSearchTracesToolRequiresConfiguredProvider(t *testing.T) {
+ result, err := SearchTraces(&traceToolTestContext{}, map[string]any{"service": "order-service"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.IsError {
+ t.Fatal("expected an error result")
+ }
+ if !strings.Contains(result.Content[0].Text, "trace provider is not configured") {
+ t.Fatalf("unexpected error message: %s", result.Content[0].Text)
+ }
+}
+
+func TestSearchTracesToolReportsUnknownProvider(t *testing.T) {
+ result, err := SearchTraces(newTraceToolTestContext("http://127.0.0.1:1"), map[string]any{
+ "service": "order-service",
+ "provider": "missing",
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.IsError {
+ t.Fatal("expected an error result")
+ }
+ if !strings.Contains(result.Content[0].Text, "missing") {
+ t.Fatalf("unexpected error message: %s", result.Content[0].Text)
+ }
+}
+
+func TestSearchTracesToolReturnsEmptyResultWithoutError(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(`{"data":[]}`))
+ }))
+ defer server.Close()
+
+ result, err := SearchTraces(newTraceToolTestContext(server.URL), map[string]any{"service": "unknown-service"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if result.IsError {
+ t.Fatalf("unexpected error result: %s", result.Content[0].Text)
+ }
+ var payload TraceSearchResult
+ if err := json.Unmarshal([]byte(result.Content[0].Text), &payload); err != nil {
+ t.Fatal(err)
+ }
+ if payload.Total != 0 || len(payload.Traces) != 0 {
+ t.Fatalf("unexpected payload: %#v", payload)
+ }
+}
+
+func TestAnalyzeTraceToolReturnsDiagnosis(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(traceToolTestPayload))
+ }))
+ defer server.Close()
+
+ result, err := AnalyzeTrace(newTraceToolTestContext(server.URL), map[string]any{"traceId": testSearchTraceID})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if result.IsError {
+ t.Fatalf("unexpected error result: %s", result.Content[0].Text)
+ }
+ var payload TraceAnalysis
+ if err := json.Unmarshal([]byte(result.Content[0].Text), &payload); err != nil {
+ t.Fatal(err)
+ }
+ if payload.TraceID != testSearchTraceID || payload.SpanCount != 2 {
+ t.Fatalf("unexpected payload: %#v", payload)
+ }
+ if len(payload.CriticalPath) != 2 || payload.CriticalPath[1].ServiceName != "payment-service" {
+ t.Fatalf("unexpected critical path: %#v", payload.CriticalPath)
+ }
+ if payload.ErrorSpanCount != 1 || len(payload.ErrorSpans) != 1 {
+ t.Fatalf("unexpected error spans: %#v", payload.ErrorSpans)
+ }
+ if !payload.ErrorSpans[0].RootCause || payload.ErrorSpans[0].ErrorType != "timeout" {
+ t.Fatalf("unexpected error span: %#v", payload.ErrorSpans[0])
+ }
+ if len(payload.Dependencies) != 1 || payload.Dependencies[0].CallerService != "order-service" {
+ t.Fatalf("unexpected dependencies: %#v", payload.Dependencies)
+ }
+ if payload.Summary == "" {
+ t.Fatal("expected a summary")
+ }
+}
+
+func TestAnalyzeTraceToolRejectsInvalidTraceID(t *testing.T) {
+ result, err := AnalyzeTrace(newTraceToolTestContext("http://127.0.0.1:1"), map[string]any{"traceId": "not-a-trace-id"})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.IsError {
+ t.Fatal("expected an error result")
+ }
+ if !strings.Contains(result.Content[0].Text, "traceId must be a 16 or 32 character hexadecimal value") {
+ t.Fatalf("unexpected error message: %s", result.Content[0].Text)
+ }
+}
+
+func TestAnalyzeTraceToolReportsUpstreamFailure(t *testing.T) {
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+ w.WriteHeader(http.StatusInternalServerError)
+ _, _ = w.Write([]byte("upstream exploded"))
+ }))
+ defer server.Close()
+
+ result, err := AnalyzeTrace(newTraceToolTestContext(server.URL), map[string]any{"traceId": testSearchTraceID})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if !result.IsError {
+ t.Fatal("expected an error result")
+ }
+}