diff --git a/docs/ai-trace-observability-design.md b/docs/ai-trace-observability-design.md new file mode 100644 index 000000000..24a96e4e9 --- /dev/null +++ b/docs/ai-trace-observability-design.md @@ -0,0 +1,685 @@ +# AI 助手 Trace 观测加强设计 + +> **实施状态**:Phase 1 已在本分支实现完成(工作区未提交)。 +> +> | 交付物 | 状态 | +> | -- | -- | +> | `pkg/mcp/tools/trace/search.go` + `Provider.SearchTraces` + Jaeger v1 查询 API 实现 | 已实现 | +> | `pkg/mcp/tools/trace/analyzer.go`(关键路径 / self time / 服务维度 / 错误归因 / 依赖边) | 已实现 | +> | `pkg/mcp/tools/trace/genai.go`(`gen_ai.*` 识别与归类) | 已实现 | +> | `pkg/mcp/register.go` 注册 `search_traces`、`analyze_trace` | 已实现 | +> | `ai/component/hooks/tracing.go` 对齐语义约定 | 已实现(`gen_ai.agent.name`、span 改名、`error.type` 兜底、`gen_ai.client.operation.exception` 事件) | +| `gen_ai.provider.name` 取值映射(T2) | 已实现(`gemini` → `gcp.gemini`,其余按合法自定义值) | +| `capture_reasoning` 独立策略(T3) | 已实现(与 `capture_content` 正交,默认 `none`) | +> | `server.address` / `server.port`、`gen_ai.response.model` | 未做,见 6.4 与 14 节待确认事项 | +> | Phase 2(metric / trace / log 联动)、Phase 3(跨 trace 聚合、多 provider) | 未开始 | +> +> 下文 2.1 与 6.2、6.4 描述的是 **Phase 1 实施前** 的基线状态,用于说明改造动机与差距来源。 + +## 1. 背景 + +2026-10-24 周会([Discussion #1612](https://github.com/apache/dubbo-admin/discussions/1612))的 **AI assistant** 章节记录了如下待办: + +| task | owner | 等级 | 进度 | issue/discussion | pr | +| -- | -- | -- | -- | -- | -- | +| trace的观测加强 | luoyi | P0 | 待开始 | - | - | + +该条目属于 **AI assistant** 章节,而不是 Admin 控制台章节,因此交付物是 **Agent 可用的观测能力**,不是给人看的控制台页面。 + +对照 2026 路线图 [Discussion #1386](https://github.com/apache/dubbo-admin/discussions/1386): + +- **AI / SRE Agent** 第 7 项「设计Hook系统,接入otel trace」——Agent 自身已通过 `ai/component/hooks/tracing.go` 接入 OTel,但缺少把 trace 数据**回喂给 Agent 自己**的能力。 +- **可观测能力** 第 3 项「基于Grafana,探索metric,trace,log的联动」——这是**给人**的 Grafana 联动,与本设计是互补关系。 + +trace 之所以是 AI 助手当前最大的盲区,是因为 AI 侧的观测能力存在一个结构性缺口: + +| 数据类型 | Agent 侧已有工具 | 能否自主发现问题 | +| -- | -- | -- | +| 集群/实例/服务拓扑 | `get_cluster_info`、`search_services`、`get_service_instances` 等 | 能 | +| 日志 | `search_logs`、`analyze_error_logs`、`get_log_capabilities` | 能 | +| 指标 | `query_prometheus`、`query_prometheus_range` | 能 | +| **链路** | **只有 `get_trace_by_id`** | **不能** | + +`get_trace_by_id`(`pkg/mcp/tools/trace/tools.go`)要求调用方提供 `traceId`,而 `traceId` 是人肉从日志、告警或 Grafana 里捞出来的。Agent 能自己查日志和指标,却无法自己「找出」一条失败的 trace。因此,只要用户的问题停留在「最近 shop-order 有没有失败的请求」「这个接口为什么变慢」这一层,Agent 就必须先把问题退回给用户索要 traceId,观测能力形同虚设。 + +## 2. 现状 + +### 2.1 现有 trace 能力 + +`pkg/mcp/tools/trace/` 当前包含: + +| 文件 | 内容 | +| -- | -- | +| `provider.go` | `Provider` 接口,当前只有 `GetTraceByID(ctx, traceID) (*Trace, error)` 一个方法;`newProvider` 按 `TraceProviderConfig.Type` 分发,`jaeger` 是唯一支持的类型 | +| `tools.go` | `GetTraceByID` handler;`traceIDPattern` 校验 16/32 位十六进制;`providerConfigFromContext` 解析 provider;`Properties()` 返回 MCP input schema | +| `jaeger.go` | Jaeger v1 `/api/traces/{traceID}` 查询;响应归一化与有界裁剪 | +| `model.go` | 后端无关的 `Trace` / `Span` / `Event` 类型(以及本次新增的搜索/分析契约类型) | +| `jaeger_test.go` | 4 个单测:查询与归一化、TraceID 校验、not found、span 裁剪 | + +`pkg/mcp/register.go` 只注册了一个 trace 工具 `get_trace_by_id`。`pkg/config/observability/tracing.go` 定义了 `TracingConfig` / `TraceProviderConfig`,`TraceProviderConfig` 带有 `BearerToken`、`Tenant` 两个认证字段,`TracingConfig.Sanitize()` 会把 `BearerToken` 替换为 `config.SanitizedValue`。 + +### 2.2 邻近工作流及其边界 + +以下工作流**已经存在或正在被别人推进**,本设计不重复实现,只做衔接: + +| 编号 | 标题 | 层次 | 与本设计的关系 | +| -- | -- | -- | -- | +| [#1606](https://github.com/apache/dubbo-admin/issues/1606) / PR [#1607](https://github.com/apache/dubbo-admin/pull/1607)(开放中) | Correlate Jaeger traces with Dubbo RPC metrics | OpenTelemetry Collector OTTL transform + Grafana `tracesToMetrics` | **互补**。它让**人**在 Grafana 里能由 trace 跳到 metric(`dubbo.application` → `application_name`、`dubbo.metric.interface` → `interface`、`rpc.method` → `method`),但它明确不新增 Admin API、Vue 页面和 spanmetrics pipeline,**Agent 依然无法自己发现或分析 trace** | +| [#1537](https://github.com/apache/dubbo-admin/issues/1537) / [#1524](https://github.com/apache/dubbo-admin/issues/1524) | 日志观测与 metric-trace-log 关联 | Grafana 面板 + 关联 | 归属他人。本设计只复用其**数据维度**,不新增 Grafana 面板 | +| PR [#1499](https://github.com/apache/dubbo-admin/pull/1499) | PromQL 与 trace 诊断工具 | MCP 工具 | 已合并,提供 `query_prometheus`、`query_prometheus_range`、`get_trace_by_id`。本设计是它的延伸,**不替换** `get_trace_by_id` | +| PR [#1536](https://github.com/apache/dubbo-admin/pull/1536) | agent hooks 与 OpenTelemetry tracing | Agent 侧 | 已合并,`ai/component/hooks/tracing.go` 已能产出 `gen_ai.*` 属性。本设计负责**对齐语义约定**并让 Agent 能读回自己的 trace | + +### 2.3 明确不属于本设计的部分 + +以下内容属于上述其他 issue 的范围,本设计**不做**: + +- 不新增 Admin HTTP API(`pkg/console/` 下无任何新增路由)。 +- 不新增或修改 Vue 页面(`ui-vue3/`)。 +- 不新增或修改 OpenTelemetry Collector pipeline(OTTL transform、spanmetrics connector 等)。 +- 不修改 Grafana datasource / dashboard 配置。 +- 不替换或下线已有的 `get_trace_by_id`。 + +本设计的唯一产出面是:**`pkg/mcp/tools/trace/` 下的新 MCP 工具** + **`ai/component/hooks/tracing.go` 的属性对齐**。 + +## 3. 目标与非目标 + +### 3.1 目标 + +- 让 Agent 能**自主发现** trace:从「服务 + 时间窗 + 条件」出发搜出候选 trace,而不需要人工提供 traceId。 +- 让 Agent 能**结构化诊断**一条 trace:关键路径、self time、服务耗时占比、错误归因、依赖边。 +- 识别 trace 内部的 **GenAI span**(模型调用、工具调用、Agent span),让 Agent 能回答「这次 LLM 调用花了多少 token、失败在哪一步」。 +- 全流程**有界**:沿用 `jaeger.go` 已有的裁剪语义,响应体积、span 数量、属性数量都有上限,超限时显式标记 `truncated`。 +- 让 `search_traces` → `analyze_trace` → `search_logs` / `query_prometheus` 形成一条 Agent 可自主执行的调查链路。 + +### 3.2 非目标 + +- 不做实时流式 trace 订阅(tail / watch)。 +- 不做跨 trace 的聚合统计(错误率趋势、慢调用分布),这属于 Phase 3。 +- 不做 trace 写入 / 采样策略配置。 +- 不在 MCP 工具输出中回显 prompt / completion 正文、tool 参数体。 +- 不做跨 Provider 的统一搜索语义(当前只有 Jaeger)。 + +## 4. 方案总览 + +```mermaid +flowchart TB + subgraph AI["Dubbo Admin AI Agent - ai/"] + REACT["ReActAgent
决策与工具编排"] + HOOK["hooks/tracing.go
GenAI 语义约定对齐"] + OTLP["OTLP Exporter
gRPC / HTTP"] + end + + subgraph MCP["MCP Server (pkg/mcp)"] + ST["search_traces
发现"] + AT["analyze_trace
诊断"] + GT["get_trace_by_id
原始明细"] + SL["search_logs
Loki"] + QP["query_prometheus
Prometheus"] + end + + subgraph PROV["Provider 抽象 pkg/mcp/tools/trace/provider.go"] + JA["jaegerProvider
Jaeger v1 Query API"] + end + + subgraph BACK["后端"] + JG["Jaeger / Trace Backend"] + LK["Loki"] + PM["Prometheus"] + end + + REACT --> ST + REACT --> AT + REACT --> GT + REACT --> SL + REACT --> QP + ST --> PROV + AT --> PROV + GT --> PROV + PROV --> JA + JA --> JG + SL --> LK + QP --> PM + HOOK --> OTLP + OTLP --> JG +``` + +三条并行的观测面: + +1. **发现面**:`search_traces` 在 Provider 侧检索,把「一段时间内某服务的失败 trace」变成 Agent 可直接消费的结构化列表。 +2. **诊断面**:`analyze_trace` 拉取单条 trace 的 span 明细,在本地做关键路径与错误归因计算,返回结论而不是原始 span 列表。 +3. **旁路面**:`search_logs`(可按 `traceId` 过滤)与 `query_prometheus` 用于把诊断结论落到具体日志行和指标上,这条链路已由 PR #1499 与 #1537 铺好。 + +Agent 自身的 `ai/component/hooks/tracing.go` 通过 OTLP 导出到同一个 Trace Backend,因此 Admin AI 自己的交互也会出现在 `search_traces` 的结果里(`traceKind == "genai"`),这使得「AI 刚才这次对话慢在哪」成为可回答的问题。 + +## 5. 关键设计 + +### 5.1 先发现,后分析 + +这是整个设计的核心判断:**只提供 `analyze_trace` 而不提供 `search_traces` 是无效的**。 + +一个只持有 traceId 的 Agent 无法开始任何调查——它无法回答「有哪些失败的 trace」「这类失败有多频繁」「是不是新出现的」。真实的问题形态是「最近 shop-order 是不是有问题」,Agent 必须先从「服务 + 时间窗」收敛到一批候选 trace,再逐条深入。 + +因此两个工具的职责严格分离: + +| 工具 | 输入锚点 | 输出体量 | 用途 | +| -- | -- | -- | -- | +| `search_traces` | service(必需)+ 时间窗 + 过滤条件 | 每条 trace 一行摘要,列表有上限 | 缩小范围、发现异常、拿到 traceId | +| `analyze_trace` | traceId(必需) | 单条 trace 的结构化诊断 | 定位原因、量化耗时、归因错误 | + +典型 Agent 调用序列: + +```mermaid +sequenceDiagram + participant U as 用户 + participant A as AI Agent + participant ST as search_traces + participant AT as analyze_trace + participant SL as search_logs + participant QP as query_prometheus + + U->>A: 最近一小时 shop-order 有失败的请求吗 + A->>ST: service=shop-order, startTime=..., endTime=..., limit=20 + ST-->>A: 12 条候选,其中 3 条 hasError=true + A->>AT: traceId=<失败 trace> + AT-->>A: criticalPath + errorSpans(rootCause) + serviceBreakdown + A->>SL: traceId=<失败 trace> + SL-->>A: 异常日志行 + A->>QP: query=histogram_quantile(... dubbo_application=...) + QP-->>A: p99 序列 + A-->>U: 结论:3 条失败均由 payment-service 超时引起 +``` + +### 5.2 有界响应与数据最小化 + +`jaeger.go` 已经确立了一整套裁剪常量,这些语义在 `search` 与 `analysis` 中必须被继承,而不是另起一套: + +| 常量 | 值 | 作用 | +| -- | -- | -- | +| `traceQueryTimeout` | 15s | 单次 Provider 请求超时 | +| `maxTraceResponseBytes` | 8 MiB | Provider 响应体上限 | +| `maxTraceErrorBodyBytes` | 4096 | 上游错误体回显上限 | +| `maxTraceSpans` | 500 | 单条 trace 参与分析的 span 上限 | +| `maxTraceRoots` | 50 | 根 span 记录上限 | +| `maxTraceServices` | 100 | 服务名列表上限 | +| `maxSpanAttributes` | 32 | 单 span 属性数上限 | +| `maxSpanEvents` | 10 | 单 span 事件数上限 | +| `maxEventAttributes` | 16 | 单事件属性数上限 | +| `maxAttributeValueBytes` | 512 | 单属性值字节上限(UTF-8 安全截断) | + +在此基础上新增两组上限: + +- **搜索侧**:`maxSearchLimit`(默认 20,上限 100)限制 `TraceSearchResult.Traces` 的长度,`maxSearchTraceSummaries`(100)限制本地保留的摘要数。Provider 侧按调用方条数再多取一条用于探测是否还有更多匹配,命中该上限即置 `Truncated = true`;否则一个刚好装满的结果会被误读为"已看全"。 +- **分析侧**:`maxCriticalPath`(默认 32)、`maxServiceBreakdown`(默认 50)、`maxSlowSpans`(默认 10)、`maxErrorSpans`(默认 20)、`maxDependencyEdges`(默认 50)、`maxGenAICalls`(默认 20)。超出部分丢弃,**只有在真正丢失分析数据时**才在 `TraceAnalysis.Truncated` 上置位:`maxCriticalPath` 截断的是仍在延续的链路,属于丢数据;`maxSlowSpans` 截断的是排名结果,不丢分析数据,因此不置位。 + +数据最小化上有一条硬规则:**分析器只读取元数据属性,不把 `gen_ai.input.messages`、`gen_ai.output.messages`、`gen_ai.tool.call.arguments`、`gen_ai.tool.call.result` 复制进工具输出**。这些属性在 semconv 中是 `Opt-In` 级别,本身就可能包含 PII;分析器只消费 `gen_ai.request.model`、`gen_ai.usage.*` 这类聚合量。 + +### 5.3 关键路径与 self time + +#### 5.3.1 span 树的构建 + +输入是 `Trace.Spans`(已由 `normalizeJaegerTrace` 归一化)。构建过程: + +1. 建立 `map[spanID]*Span`,遍历 `Trace.Spans`。 +2. 父指针取 `Span.ParentSpanID`。`jaeger.go` 的 `parentSpanID(references)` 优先取 `refType == "CHILD_OF"` 的引用,没有则退回第一个引用;两者都为空即为根 span。因此**根 span = `ParentSpanID` 为空的 span**,与 `Trace.RootSpanIDs` 一致。 +3. `ParentSpanID` 指向的 span 不在集合内(被 `maxTraceSpans` 裁掉,或采样缺失)时,该 span 退化为**孤儿根**。孤儿根不参与父链计算,但其自身耗时仍计入 `ServiceBreakdown` 与 `SlowSpans`。 +4. 解析 `Span.StartTime`(RFC3339Nano)为 `time.Time`。解析失败的 span 退化为叶子节点并跳过 self time 计算,同时把 `TraceAnalysis.Truncated` 置位——时间基准不可信时继续计算会产出误导性的百分比。 + +#### 5.3.2 self time 的计算 + +定义:对 span `S`,令 `S` 的子集合为 `C(S)`,`S` 的时间区间为 `[S.start, S.end]`。 + +``` +covered(S) = | ⋃_{c ∈ C(S)} [ max(c.start, S.start), min(c.end, S.end] ] | +self(S) = S.duration - covered(S) +wait(S) = covered(S) +``` + +`covered` 是**子 span 时间区间的并集长度**,而不是子 span duration 之和。这一点是必须的:并发调用(例如同时发起 3 个下游 RPC)的子 span 在时间上重叠,简单求和会让 `self` 变成负数、并让 `ServiceBreakdown.SharePercent` 严重失真。实现上采用区间合并:按 `start` 升序排序子区间,线性扫描合并重叠段,再累加合并后的总长度,复杂度 `O(n log n)`。 + +边界处理: + +- 子 span 完全在父 span 之外(时钟漂移或采样错乱)→ 裁剪到父区间后长度归零,不计入 `covered`。 +- 子 span 区间为空或 duration 为 0 → 不计入。 +- `self` 计算结果小于 0(父区间被裁剪得更短)→ 置 0,并在该 span 上标记异常。 + +#### 5.3.3 关键路径的选择 + +关键路径定义为:**决定端到端时延的那条根→叶链路**。 + +算法: + +1. 在所有根 span 中取 `end` 最大者作为关键路径起点(`end` 相同则取 duration 最大者)。若存在多个根,其余根的子树仍参与 `ServiceBreakdown`、`SlowSpans`、`ErrorSpans`、`Dependencies` 计算,但不进入 `criticalPath`。 +2. 从当前节点出发,在其子节点中选取 `end` 最大者(`end` 相同则取 duration 最大者)作为下一跳。理由:`end` 最大的子 span 是父 span 内**最后完成**的活动,它决定了父 span 的完成时刻。 +3. 重复直到到达叶子(无子节点)。整条链路即为 `TraceAnalysis.CriticalPath`。 +4. 每一跳记录 `CriticalPathEntry`:`spanId`、`serviceName`、`operationName`、`startTime`、`durationMicros`、`selfMicros`、`waitMicros`。 + +这样得到的链路满足:链路上每一跳的 `end` 都不晚于其父节点的 `end`。在 span 列表未被裁剪时,链尾的 `end` 等于 trace 的墙钟终点;列表被 `maxTraceSpans` 裁剪时,关键路径只在保留下来的 span 上计算,链尾可能早于 provider 给出的墙钟终点(此时 `durationMicros` 取 provider 的完整包络,见 §5.2)。 + +链路上的 `selfMicros` 之和是**这条关键路径上**的不可压缩耗时,不是整条 trace 的不可压缩耗时:不在关键路径上的并行分支同样有 self time,不计入这个和。它是 trace 实际工作量的下界,不是上界。 + +`SlowSpan` 则按 **self time 降序**取前 N(默认 10),并用 `onCriticalPath` 标记该 span 是否落在关键路径上。self time 排名与 critical path 互补:前者回答「谁自己最慢」,后者回答「请求是怎么被拖长的」。 + +> 注:`ServiceLatency.SharePercent` 的语义是「该服务的 span duration 之和占 trace 墙钟时长的比例」,分母是墙钟、分子是各 span 时长之和,因此在并发场景下可能超过 100%。这是刻意的「工作量占比」语义而非「时间占比」,调用方需要按此理解,不能把它当成互斥的时间切片。 + +### 5.4 错误归因与根因检测 + +**错误 span 判定**:`Span.Error == true` 或 `Span.Status == "ERROR"`。该状态来自 `jaeger.go` 的 `spanStatus(tags)`,它识别三类 tag:`error=true|1`、`otel.status_code=error`、`status.code=error`;`status.code=ok` 映射为 `Status == "OK"`。 + +**error.type 分类**(按优先级取第一个可用的): + +| 优先级 | 来源 | 说明 | +| -- | -- | -- | +| 1 | span 属性 `error.type` | OTel 稳定属性,semconv 中为 `Conditionally Required`,值域如 `timeout`、`_OTHER` | +| 2 | 事件 `gen_ai.client.operation.exception` 的属性 `exception.type` | semconv 的 GenAI 专用异常事件。本项目服务于 **Agent 循环的 GenAI 执行**,故排在传统异常事件之前 | +| 3 | 事件 `exception` 的属性 `exception.type` | 标准 OTel 异常事件,面向传统 RPC 服务的异常面 | +| 4 | span 属性 `exception.type` | 采集器把异常直接落在 span 属性/标签上时的兜底,事件仍是主来源 | +| 5 | span 属性 `error.message` | **不是 OTel 标准属性**(error registry 只定义 `error.type`),仅在没有任何类型化来源时降级使用 | +| 6 | 常量 `unknown` | 全部缺失 | + +`ErrorSpan.ErrorMessage` 按 `gen_ai.client.operation.exception` 事件 → 标准 `exception` 事件 → span 属性 `exception.message` → span 属性 `error.message` 的顺序取值,并受长度上限约束(错误类型 96 字符、错误描述 256 字符)。`error.message` 因为不是标准属性,排在最后。 + +> 这里刻意区分了两类错误面:**GenAI 执行异常**(模型调用、工具调用失败,走 `gen_ai.client.operation.exception`)与**传统 API 服务异常**(Dubbo RPC 侧的普通异常,走标准 `exception` 事件)。本设计以前者为重点,但分析 Dubbo 链路时后者同样必须能读,故两者都保留、按 GenAI 优先排序。 + +**token 口径**:`gen_ai.usage.total_tokens` **不是**规范属性,规范只定义了 `gen_ai.usage.input_tokens` 与 `gen_ai.usage.output_tokens`,「总量」是派生概念。因此 `GenAISummary.TotalTokens` **恒等于** `InputTokens + OutputTokens`;若某个非规范 provider 上报了 `gen_ai.usage.total_tokens`,它只作为单次调用明细 `GenAIModelCall.TotalTokens` 的参考值出现,**不参与汇总**(此时明细之和与汇总之和可能不等,属预期)。 + +**根因检测**:在错误 span 构成的子树中,**没有任何后代 span 也失败**的 span 被标记 `RootCause = true`。反之,一个错误 span 如果其子树中还有失败 span,说明它只是**错误传播**的中间环节,不是根因。`ErrorSpans` 按「最深失败优先」排序,让 Agent 先看到最内层的失败。 + +这套规则能覆盖 Dubbo 场景中最常见的两种形态:下游超时导致上游 RPC 报错(根因 = 下游 span),以及序列化失败(根因 = 唯一失败的 span)。 + +### 5.5 服务依赖边 + +依赖边直接由父子关系导出:对每个 span `c`(其父 `p` 存在于同一 trace 内),产生一条 `p.ServiceName → c.ServiceName` 的边。 + +聚合规则: + +- 按 `(callerService, calleeService)` 去重,`spanCount` 为该对出现的次数。 +- `totalMicros` 为该对下所有子 span 的 duration 之和。 +- `errorCount` 为该对下 `Error == true` 的子 span 数量。 +- 按 `totalMicros` 降序排列。 + +父子关系天然刻画了**同一条 trace 内的**调用拓扑,是 Dubbo 场景下最可靠的服务依赖信号。跨 trace 的全局拓扑不在本设计范围(属于 roadmap 的「服务拓扑/应用拓扑」条目)。 + +## 6. GenAI 语义约定对齐 + +### 6.1 规范来源与状态 + +权威来源是 OpenTelemetry **semantic conventions for generative AI systems**,本设计对齐的是该仓库的只读 checkout(commit `6fd0d76`),文档位于 `docs/gen-ai/`: + +| 规范文件 | 内容 | +| -- | -- | +| `docs/gen-ai/README.md` | 索引 | +| `docs/gen-ai/gen-ai-spans.md` | 模型类 span(chat / embeddings / retrieval / fetch_response / memory / execute_tool) | +| `docs/gen-ai/gen-ai-agent-spans.md` | Agent 类 span(create_agent / invoke_agent / invoke_workflow / plan) | +| `docs/gen-ai/client-inference.md` | 推理(chat)span 与事件 | +| `docs/gen-ai/gen-ai-events.md` | GenAI 事件(`gen_ai.evaluation.result`) | +| `docs/gen-ai/gen-ai-exceptions.md` | GenAI 异常事件 | +| `docs/gen-ai/gen-ai-token-metrics.md` | 推理 token 指标 | + +**文档状态**:上述所有文件的标题下方均为 `**Status**: [Development][DocumentStatus]`,即**开发中、尚未稳定**。所有 `gen_ai.*` 属性在 registry 中的 Stability 均为 `Development`;例外是通用属性 `error.type`、`exception.message`、`exception.type`、`exception.stacktrace`、`server.address`、`server.port` 为 `Stable`。 + +这是一份快速演进中的文档。本设计必须**钉住规范版本**(记录 commit `6fd0d76`),并在每次升级规范时重新核对属性名与枚举值,而不是把属性名写死为「永久正确」。 + +### 6.2 属性对照表 + +下表列出 `analyze_trace` 侧识别的属性,以及 Agent 侧 `ai/component/hooks/tracing.go` 需要发出的属性。 + +| 属性 | 分析器用途 | 规范文件 | Agent 侧现状 | Agent 侧动作 | +| -- | -- | -- | -- | -- | +| `gen_ai.operation.name` | 判定 span 类别(模型调用 / 工具调用 / Agent) | `gen-ai-spans.md`、`gen-ai-agent-spans.md` | 已发:`invoke_agent`、`chat`、`execute_tool` | 保持 | +| `gen_ai.provider.name` | 归类 provider(`openai`、`anthropic`、`aws.bedrock`、`azure.ai.openai`、`gcp.vertex_ai` …) | `gen-ai-spans.md` | 已发,但由 model 字符串按 `/` 切分推导 | **改为白名单映射**,见 6.4 | +| `gen_ai.request.model` | 模型名(模型调用摘要) | `client-inference.md` | 已发(取 model 字符串 `/` 之后部分) | 保持 | +| `gen_ai.response.model` | 实际响应模型 | `gen-ai-spans.md` | 未发 | 可选补充 | +| `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` | token 聚合 | `gen-ai-spans.md`(span 属性,Recommended) | 已发(`EventModelCallEnd`) | 保持 | +| `gen_ai.conversation.id` | 会话归组 | `gen-ai-agent-spans.md` | 已发(取 `SessionID`) | 保持 | +| `gen_ai.agent.name` | Agent 名称 | `gen-ai-agent-spans.md`(invoke_agent 上 Conditionally Required) | **未发** | **Phase 1 必须补** | +| `gen_ai.agent.id` | Agent 稳定标识 | `gen-ai-agent-spans.md` | 未发 | 有稳定标识时补 | +| `gen_ai.tool.name` | 工具名 | `gen-ai-spans.md`(execute_tool 上 **Required**) | 已发 | 保持 | +| `gen_ai.tool.call.id` | 工具调用标识 | `gen-ai-spans.md`(Recommended) | 已发 | 保持 | +| `gen_ai.tool.call.arguments` / `gen_ai.tool.call.result` | 工具入参/结果 | `gen-ai-spans.md`(**Opt-In**) | 已发,受 capture policy 控制 | 保持,分析器**不读取** | +| `gen_ai.input.messages` / `gen_ai.output.messages` | 提示词/回复 | `gen-ai-spans.md`(**Opt-In**) | 已发,受 capture policy 控制 | 保持,分析器**不读取** | +| `error.type` | 错误分类 | `gen-ai-spans.md`(Conditionally Required) | 已发(`state.ErrorType` 非空时) | 保持 | +| `exception.type` / `exception.message` | 异常类型与消息 | `gen-ai-exceptions.md` | 由 `span.RecordError` 产生的标准 `exception` 事件 | 见 6.4 | + +### 6.3 genai trace kind 的判定 + +`TraceSummary.TraceKind` 的语义(见 `model.go`):**当 trace 中任一 span 携带 `gen_ai.*` 属性时,值为 `"genai"`,否则为空字符串**。 + +实现上,`pkg/mcp/tools/trace/genai.go` 提供统一识别函数: + +- `hasGenAIAttributes(span)`:span 的 `Attributes` 中存在任意以 `gen_ai.` 为前缀的键。 +- 分类映射: + - `gen_ai.operation.name ∈ {chat, generate_content, text_completion, embeddings}` → **模型调用**(`GenAIModelCall`)。 + - `gen_ai.operation.name == execute_tool` → **工具调用**(`GenAIToolCall`)。 + - `gen_ai.operation.name ∈ {invoke_agent, create_agent, invoke_workflow, plan}` → **Agent span**(`GenAIAgent`)。 + +之所以必须识别 LLM span,是因为在 Agent 场景里,**「失败」经常发生在模型调用或工具调用上,而不是 Dubbo RPC 上**。一条 `traceKind == "genai"` 的 trace 里,最有价值的诊断信息是:调了哪个模型、花了多少 token、哪个工具返回了错误、异常事件是什么。这些信息在通用的 RPC 视角下完全不可见。 + +### 6.4 与当前 Agent hook 的不一致点 + +`ai/component/hooks/tracing.go` 已由 PR #1536 合并,其属性与当前规范存在以下差距,本设计需要在 Phase 1 对齐: + +1. **`gen_ai.provider.name` 的取值不规范**。当前 `providerName(model)` 直接按 `/` 切分 model 字符串。规范枚举值是 `openai`、`anthropic`、`aws.bedrock`、`azure.ai.inference`、`azure.ai.openai`、`gcp.gemini`、`gcp.gen_ai`、`gcp.vertex_ai`、`deepseek`、`groq`、`mistral_ai`、`moonshot_ai`、`perplexity`、`x_ai`、`cohere`、`ibm.watsonx.ai` 等。切分得到的可能是 `azure` 而不是 `azure.ai.openai`。需要一张显式映射表,无法识别时按规范允许「自定义值」,但不应产生 `azure` 这种半截值。 +2. **缺少 `gen_ai.agent.name`**。规范在 `invoke_agent` span 上将其列为 Conditionally Required。缺了它,Admin 自己的 Agent span 在 trace 里就没有身份,分析器只能识别出「有一个 Agent span」,无法命名。当前 span 名 `invoke_agent` 正是规范中 `gen_ai.agent.name` 不可用时的合法回退形式,所以补上 `gen_ai.agent.name` 后 span 名应升级为 `invoke_agent {gen_ai.agent.name}`。 +3. **异常事件不是 GenAI 专用事件**。当前用 `span.RecordError(errors.New(state.Error))` 产生标准 `exception` 事件,其 `exception.type` 会是 `*errors.errorString` 这类包装类型,而 `gen-ai-exceptions.md` 明确指出「如果记录的异常类型是无意义的包装类型,应使用内层异常的类型」,并定义了 `gen_ai.client.operation.exception` 事件(severity SHOULD 为 WARN)。对齐后,模型调用与工具调用的失败应同时产生 `gen_ai.client.operation.exception` 事件并带上真实的 `exception.type`。 +4. **`agent.*` 系列属性是自定义扩展**。`agent.interaction.id`、`agent.iteration`、`agent.stage`、`agent.degraded`、`agent.fallback.used`、`agent.fallback.reason` **不在 semconv registry 中**,是 Dubbo Admin 私有的命名空间。它们本身合理,但必须在设计里明确标注为自定义属性,避免被误认为符合规范。 + > **已确认(2026-10-25)**:原 `agent.usage.total_tokens` **已移除**。规范只定义 `gen_ai.usage.input_tokens` 与 `gen_ai.usage.output_tokens`,总量为两者之和;成本核算由企业侧 API 网关承担,Admin 只需给出「模型名 + token 数」。token 口径完全对齐 GenAI 规范,Dubbo Admin 不再维护第二套。 +5. **缺少 `server.address` / `server.port`**。规范列为 Recommended / Conditionally Required。补齐后,分析器可以指出模型调用打到了哪个 provider 端点,对「换了网关导致模型调用变慢」这类问题有价值。 +6. **token 指标口径未对齐**。规范在 span 属性层面用 `gen_ai.usage.input_tokens`(当前 hook 已符合),但在**指标**层面用的是 `gen_ai.client.inference.usage.input_tokens` 等 counter,并额外要求 `gen_ai.token.modality`(`text`/`image`/`audio`,无法判定时为 `unknown`)。当前 hook 不产出 modality,因此如果将来要按 semconv 聚合 token 指标,需要补 `gen_ai.token.modality=unknown`。 +7. **`gen_ai.system` 的历史重命名**。`gen_ai.system` 在早期规范版本中已被 `gen_ai.provider.name` 取代;在当前 checkout 的 `docs/registry/attributes/gen-ai.md` 中**已完全不存在 `gen_ai.system`**,仅保留 `gen_ai.provider.name`。本设计的 Agent hook 已经使用新名,无需兼容旧名;但阅读历史资料时需要留意。 +8. **缺少 `gen_ai.main_agent.*`**。当前规范在 registry 中新增了 `gen_ai.main_agent.id` / `gen_ai.main_agent.name` / `gen_ai.main_agent.description`,用于标识进程内的顶层 Agent 实体。Admin AI 自身就是顶层 Agent,但 hook 未发出这些属性,规范同时要求「不得由进程内临时标识生成 `gen_ai.main_agent.id`」,因此只有在具备稳定标识时才应补齐。这属于可选增强,不阻塞 Phase 1。 + +## 7. 数据契约 + +### 7.1 Go 类型 + +以下类型定义在 `pkg/mcp/tools/trace/model.go`,JSON tag 即 MCP 工具返回给 Agent 的字段名。 + +| 类型 | 字段(JSON) | 说明 | +| -- | -- | -- | +| `TraceSearchQuery` | — | Provider 侧检索入参(进程内结构,不直接序列化) | +| | `Service` | 必填。根服务名,Provider 无法脱离服务锚点检索 | +| | `Operation` | 可选,限定根操作名 | +| | `TraceID` | 可选,锚定单条 trace | +| | `Tags` | 可选,`map[string]string`,如 `rpc.system=apache_dubbo` | +| | `StartTime` / `EndTime` | `time.Time`,零值表示不限 | +| | `Limit` | 限制返回条数,非正值表示 Provider 默认 | +| | `MinDuration` | `time.Duration`,过滤短于阈值的 trace | +| `TraceSearchResult` | `traces` / `total` / `truncated` / `sourceEngine` | 搜索结果聚合 | +| `TraceSummary` | `traceId` | trace 标识 | +| | `rootService` / `rootOperation` | 根 span 的服务名与操作名 | +| | `startTime` / `durationMicros` | trace 墙钟包络 | +| | `spanCount` / `errorSpanCount` / `serviceCount` | 规模指标 | +| | `hasError` | 是否存在错误 span | +| | `traceKind` | `"genai"` 或空串 | +| `TraceAnalysis` | `traceId` / `summary` | 标识与一句话结论 | +| | `startTime` / `durationMicros` / `spanCount` / `serviceCount` / `errorSpanCount` | 规模指标 | +| | `criticalPath` | `[]CriticalPathEntry` | +| | `serviceBreakdown` | `[]ServiceLatency` | +| | `slowSpans` | `[]SlowSpan` | +| | `errorSpans` | `[]ErrorSpan` | +| | `dependencies` | `[]DependencyEdge` | +| | `genai` | `*GenAISummary`,无 GenAI 属性时为 `nil` | +| | `truncated` | 分析是否基于裁剪后的 span 计算 | +| `CriticalPathEntry` | `spanId` / `serviceName` / `operationName` / `startTime` / `durationMicros` | 关键路径一跳 | +| | `selfMicros` / `waitMicros` | self time 与等待子 span 的时间 | +| `ServiceLatency` | `serviceName` / `spanCount` / `errorSpanCount` | 服务的 span 规模 | +| | `totalMicros` / `maxMicros` / `avgMicros` | 服务的耗时统计 | +| | `sharePercent` | 占 trace 墙钟时长的比例(可为 > 100,见 5.3.3) | +| `SlowSpan` | `spanId` / `serviceName` / `operationName` / `durationMicros` | 慢 span 标识 | +| | `sharePercent` / `onCriticalPath` | 占比与是否在关键路径上 | +| `ErrorSpan` | `spanId` / `serviceName` / `operationName` | 错误 span 标识 | +| | `errorType` / `errorMessage` | 错误分类与描述 | +| | `status` | 归一化 span 状态,如 `ERROR` | +| | `rootCause` | 是否为根因(无失败后代) | +| `DependencyEdge` | `callerService` / `calleeService` | 调用方与被调方 | +| | `spanCount` / `totalMicros` / `errorCount` | 边的聚合统计 | +| `GenAISummary` | `present` | 是否观察到 GenAI 属性 | +| | `conversationIds` / `providers` / `models` | 去重后的标识集合 | +| | `inputTokens` / `outputTokens` / `totalTokens` | token 聚合 | +| | `modelCallCount` / `toolCallCount` / `agentCount` | 调用构成 | +| | `modelCalls` / `toolCalls` / `agents` | 逐调用明细 | +| `GenAIModelCall` | `spanId` / `serviceName` / `operationName` / `operation` | 模型调用标识与 `gen_ai.operation.name` | +| | `providerName` / `requestModel` / `responseModel` | provider 与模型 | +| | `inputTokens` / `outputTokens` / `totalTokens` / `durationMicros` / `error` | 用量、耗时与失败标记 | +| `GenAIToolCall` | `spanId` / `serviceName` / `toolName` / `toolCallId` / `durationMicros` / `error` | 工具调用明细 | +| `GenAIAgent` | `spanId` / `serviceName` / `agentName` / `agentId` / `durationMicros` / `error` | Agent span 明细 | + +### 7.2 MCP 工具签名 + +| 工具 | 参数 | 类型 | 必填 | 说明 | +| -- | -- | -- | -- | -- | +| `search_traces` | `service` | string | **是** | 检索的服务锚点 | +| | `operation` | string | 否 | 根操作名 | +| | `traceId` | string | 否 | 锚定单条 trace(16/32 位十六进制) | +| | `tags` | string | 否 | `k=v,k2=v2` 形式的标签过滤 | +| | `startTime` | string | 否 | RFC3339 起始时间 | +| | `endTime` | string | 否 | RFC3339 结束时间 | +| | `minDurationMs` | integer | 否 | 最小耗时(毫秒) | +| | `limit` | integer | 否 | 返回条数上限,默认 20 | +| | `provider` | string | 否 | Provider 名,默认 `defaultProvider` | +| `analyze_trace` | `traceId` | string | **是** | 16/32 位十六进制 trace 标识 | +| | `provider` | string | 否 | Provider 名,默认 `defaultProvider` | +| `get_trace_by_id` | `traceId` | string | **是** | 同上(已存在,本设计不改) | +| | `provider` | string | 否 | 同上(已存在) | + +## 8. MCP 工具清单 + +| 工具 | 用途 | 必需参数 | 典型 Agent 流程 | +| -- | -- | -- | -- | +| `search_traces` | 按服务与时间窗发现候选 trace,返回逐条摘要 | `service` | 「最近一小时 X 服务有没有失败的请求」→ 先搜出候选,筛 `hasError=true` | +| `analyze_trace` | 对单条 trace 做结构化诊断:关键路径、耗时分解、错误归因、依赖边、GenAI 摘要 | `traceId` | 「这条 trace 为什么慢 / 为什么失败」→ 定位到具体 span 与根因 | +| `get_trace_by_id` | 返回原始 trace 明细(服务、span 关系、耗时、错误状态) | `traceId` | 需要逐 span 细节时兜底,已有工具,不改动 | + +`search_traces` 与 `analyze_trace` 在 `pkg/mcp/register.go` 中注册,Description 与 InputSchema 沿用现有风格(`get_trace_by_id` 的 Description 为英文,工具名与参数名保持英文)。 + +## 9. 错误与边界 + +所有错误都以 `common.ErrorResult(err)` 返回,即 `ToolResult{IsError: true, Content: [{Type: "text", Text: err.Error()}]}`,**handler 永远返回 `nil` 作为 Go error,不 panic**。这与 `tools.go` 现有的 `GetTraceByID` 行为一致。 + +| 场景 | 判定 | 对 Agent 的呈现 | +| -- | -- | -- | +| Provider 未配置 | `ctx.Config().Observability.Tracing == nil` | `isError=true`,「trace provider is not configured」 | +| 默认 Provider 未配置 | `tracing.Default()` 不存在 | `isError=true`,「default trace provider is not configured」 | +| 指定 Provider 不存在 | `tracing.Get(name)` 不存在 | `isError=true`,「trace provider %q is not configured」 | +| Provider 类型不支持 | `newProvider` 命中 default 分支 | `isError=true`,「unsupported trace provider type: %s」 | +| Trace 不存在 | Jaeger 返回 404 或 `data` 为空 | `isError=true`,「trace %s was not found」 | +| TraceID 非法 | 不匹配 `^(?:[0-9a-fA-F]{16}\|[0-9a-fA-F]{32})$` | `isError=true`,「traceId must be a 16 or 32 character hexadecimal value」 | +| 上游报错 | 非 2xx,或 `errors[]` 非空 | `isError=true`,包含状态码与截断后的错误体(≤ `maxTraceErrorBodyBytes`) | +| 上游响应体过大 | 超过 `maxTraceResponseBytes` | `isError=true`,「jaeger response exceeds %d bytes」 | +| 响应不一致 | `data` 长度 ≠ 1 或 traceId 不匹配 | `isError=true`,「jaeger returned an inconsistent response for trace %s」 | +| 搜索结果为空 | 正常返回 | **`isError=false`**,`traces: []`、`total: 0`;这是合法业务结果而非错误 | +| 搜索被裁剪 | Provider 命中条数上限(按调用方条数再多取一条探测) | `isError=false`,`truncated: true` | +| 分析基于裁剪数据 | 底层 `Trace.Truncated` 为真 | `isError=false`,`TraceAnalysis.Truncated = true`,Agent 需在结论中注明数据不完整 | +| 参数非法 | `startTime`/`endTime` 非 RFC3339、`tags` 格式错误、`minDurationMs` 为负 | `isError=true`,指明具体参数 | + +所有上游失败都会经 `jaegerProvider.logFailure` 记录结构化日志,字段为 `provider`、`engine`、`trace_id`(掩码)、`elapsed_ms`、`error_type`,可选 `upstream_status`。 + +## 10. 安全与隐私 + +| 方面 | 措施 | +| -- | -- | +| 凭据不外泄 | `TraceProviderConfig.BearerToken` 只用于出站请求的 `Authorization: Bearer ` 头,不进入任何工具输出;`TracingConfig.Sanitize()` 在配置展示时把 `BearerToken` 替换为 `config.SanitizedValue`;`get_observability_capabilities` 只返回 provider 的 `name` 与 `type` | +| 租户隔离 | `TraceProviderConfig.Tenant` 通过 `X-Scope-OrgID` 头(常量 `tenantHeader`)传给后端,不出现在响应中 | +| TraceID 掩码 | 日志中的 trace_id 经 `maskedTraceID()` 处理,仅保留末 8 位并加 `...` 前缀 | +| 传输错误净化 | Go 的 `http.Client` 返回 `*url.Error`,其 message 内嵌完整请求 URL(含 query string)。**失败日志**与**工具返回**中的传输错误都经 `sanitizeTransportError()` 解包为内层错误,只保留 `dial tcp ...` 这类诊断信息,避免检索条件、`tags` 值与被配置为 endpoint 的凭据进入日志或模型上下文。为杜绝凭据经 URL 泄漏,`newJaegerProvider` 拒绝带 query 或 fragment 的 endpoint,凭据只允许走 `bearerToken`。工具结果中**上游响应体**的业务敏感数据目前仍**没有过滤机制**,见 14.2 | +| 服务名 | `service` 既是检索锚点也是链路内的服务标识。它是基础设施元数据(与 OTel 的 resource 属性 `service.name` 同类),不是用户数据也不是凭据;设计文档 §9 明确要求失败日志记录服务名。真正需要防范的是 `tags` 过滤值、span 属性里的业务字段以及内容类属性 | +| 错误体回显 | 上游错误体读取上限 `maxTraceErrorBodyBytes = 4096`,避免把超长 HTML 错误页灌进模型上下文 | +| 属性值截断 | 单属性值上限 512 字节,UTF-8 安全截断(`truncateUTF8`) | +| 内容不外泄 | 分析器不读取也不输出 `gen_ai.input.messages`、`gen_ai.output.messages`、`gen_ai.tool.call.arguments`、`gen_ai.tool.call.result` | +| Agent 侧内容捕获 | `ai/component/hooks/config.go` 提供 `CaptureNone` / `CaptureTruncated` / `CaptureFull` 三档(`ai/component/hooks/component.go`)。`none` 完全不发出内容属性;`truncated` 在 `truncatedContentLimit = 4096` 字节内截断,单个 JSON 字符串再截到 512 字符;`full` 原样输出。默认配置为 `CaptureNone` | +| 提示注入 | trace 属性、错误消息、span 名都是**不可信数据**,只能作为当前轮观察数据传给模型,不得作为指令 | + +其中内容捕获策略与分析器口径是配合的:`CaptureNone` 下 Agent 自己的 trace 里根本不含提示词正文,分析器拿到的只有模型名与 token 数——这正是我们想要的最小数据集。 + +## 11. 测试策略 + +### 11.1 单元测试矩阵 + +| 文件 | 覆盖点 | +| -- | -- | +| `pkg/mcp/tools/trace/search_test.go` | `tags` 字符串解析(`k=v,k2=v2`、空串、缺 `=`、重复键、含空格);`startTime`/`endTime` 的 RFC3339 解析与非法值;`minDurationMs` → `MinDuration` 换算;`limit` 默认值与上限裁剪;`traceId` 格式校验;provider 未配置 / 指定 provider 不存在;空结果返回 `total=0` 且 `isError=false`;`traceKind` 判定(含含 `gen_ai.*` 与不含两种用例);`truncated` 标记 | +| `pkg/mcp/tools/trace/analyzer_test.go` | span 树构建(正常父子、孤儿 span、`ParentSpanID` 悬空);**self time**:单子 span、两个**重叠**并发子 span(验证并集而非求和)、三层嵌套、子 span 越界、负值归零;**critical path**:多分支取 `end` 最大者、多根取 `end` 最大根、叶子终止;`serviceBreakdown` 的 `totalMicros`/`maxMicros`/`avgMicros`/`sharePercent`;`slowSpans` 按 self time 排序与 `onCriticalPath` 标记;**错误归因**:`error.type` 优先、`exception.type` 次之、`error.message` 兜底、全缺失时为 `unknown`;**根因检测**:最深层失败 `rootCause=true`、传播型错误 `rootCause=false`;**依赖边**:去重、`spanCount`/`totalMicros`/`errorCount` 聚合、按 `totalMicros` 排序;`truncated` 传播;时间解析失败时降级不 panic | +| `pkg/mcp/tools/trace/genai_test.go` | `gen_ai.operation.name` 到模型调用/工具调用/Agent 的分类;`gen_ai.provider.name`/`gen_ai.request.model`/`gen_ai.response.model` 提取;`gen_ai.usage.input_tokens`/`output_tokens` 解析与求和;`gen_ai.conversation.id` 去重;`gen_ai.agent.name`/`gen_ai.agent.id` 提取;无 `gen_ai.*` 属性时 `GenAI` 为 `nil`;`error.type` 分类 | +| `pkg/mcp/tools/trace/jaeger_test.go` | 保持现有 4 个用例不变(回归保护) | + +### 11.2 Agent 侧 hook 测试 + +`ai/component/hooks/tracing_test.go` 补充属性断言: + +- span 名:`chat {model}`、`execute_tool {tool}`、`invoke_agent {agent.name}`(补齐 `gen_ai.agent.name` 后)。 +- span kind:模型调用为 `CLIENT`,Agent / 工具调用为 `INTERNAL`。 +- 属性存在性与取值:`gen_ai.operation.name`、`gen_ai.provider.name`(映射表命中与未命中两种)、`gen_ai.request.model`、`gen_ai.conversation.id`、`gen_ai.usage.input_tokens`、`gen_ai.usage.output_tokens`、`gen_ai.tool.name`、`gen_ai.tool.call.id`、`error.type`。 +- 异常路径:产生 `gen_ai.client.operation.exception` 事件且 `exception.type` 为内层真实类型。 +- 内容捕获策略三档:`none` 不产生内容属性;`truncated` 长度受限;`full` 原样。 + +### 11.3 集成测试 + +- 用 `httptest` 起一个假 Jaeger:校验 `search_traces` 实际发出的查询参数(`service`/`operation`/`tags`/`start`/`end`/`limit`),以及 `analyze_trace` 对完整 trace 的端到端诊断输出。 +- MCP Server 侧:`tools/list` 能看到 `search_traces` 与 `analyze_trace`;调用参数缺 `service` 时返回 `isError=true`。 + +## 12. 分阶段实施 + +### Phase 1:P0 核心(本次 P0 任务的主体) + +交付物: + +- `pkg/mcp/tools/trace/search.go`:Provider 侧检索能力(`Provider` 接口新增 `SearchTraces` 方法,Jaeger 实现走 v1 查询 API),MCP 工具 `search_traces`。 +- `pkg/mcp/tools/trace/analyzer.go`:本地诊断计算与 MCP 工具 `analyze_trace`。 +- `pkg/mcp/tools/trace/genai.go`:GenAI 属性识别与归类。 +- `pkg/mcp/register.go`:注册两个新工具。 +- `ai/component/hooks/tracing.go`:按 6.4 对齐语义约定(provider 映射表、`gen_ai.agent.name`、GenAI 异常事件、`server.address`/`server.port`)。 +- 上述单元测试与集成测试。 + +退出标准: + +- `go test ./pkg/mcp/tools/trace/...` 全绿;`ai` 模块 `go test ./...` 全绿。 +- MCP `tools/list` 出现 `search_traces` 与 `analyze_trace`。 +- 在一个接入真实 Jaeger 的环境中,Agent 能在**零人工 traceId** 的前提下回答「最近一小时 X 服务的失败 trace 有哪些」。 +- 现有 `get_trace_by_id` 行为无回归。 + +### Phase 2:metric / trace / log 联动 + +交付物: + +- 在 `analyze_trace` 的输出中补齐可直接用于 PromQL 的维度(服务名、接口名),与 PR #1607 引入的 `dubbo.application` / `dubbo.metric.interface` 标签对齐。 +- 在 Agent 的工作流层串联 `search_logs`(按 `traceId` 过滤)与 `query_prometheus`,形成「trace → 日志行 → 指标序列」的完整调查闭环。 +- 与 #1537 的 metric-trace-log 关联方案对齐标签口径,避免两套命名。 + +退出标准: + +- Agent 能对单条失败 trace 给出「错误 span + 对应日志行 + 相关指标序列」的联合结论。 +- PromQL 中使用的标签名与 #1607 / #1537 的口径一致。 + +### Phase 3:纵深能力 + +交付物: + +- Collector 侧 spanmetrics / exemplars 打通(与 #1607 互补,不重复造)。 +- 多 Provider 支持(如 Tempo),把 `Provider` 接口的检索能力抽象为可插拔。 +- 跨 trace 聚合:按服务统计错误率与慢调用分布,支撑「最近一小时错误率是上升还是下降」这类问题。 + +退出标准: + +- Provider 接口的实现数量 ≥ 2,且新增 Provider 不需要改动 MCP 工具层。 +- Agent 能给出跨 trace 的时间趋势结论。 + +## 13. 度量与验收 + +### 13.1 核心验收场景 + +| 场景 | 问题 | 期望 | +| -- | -- | -- | +| 发现 | 「最近一小时 shop-order 有失败的请求吗」 | Agent 自主调用 `search_traces`,**零人工 traceId**,返回失败 trace 列表 | +| 定位慢 | 「这条 trace 为什么慢」 | 返回 `criticalPath` 与 top `slowSpans`,指出瓶颈服务与占比 | +| 归因失败 | 「这条 trace 为什么失败」 | 返回 `errorSpans`,其中 `rootCause=true` 的那条带 `errorType` | +| 拓扑 | 「这条 trace 经过了哪些服务」 | 返回 `dependencies` 调用边 | +| LLM 可观测 | 「这次 AI 对话花了多少 token、哪一步失败」 | 返回 `genai` 摘要,含 `models`、`totalTokens`、`modelCalls`、`toolCalls` | + +### 13.2 可实际执行的检查 + +```bash +# 单元测试 +go test ./pkg/mcp/tools/trace/... +(cd ai && go test ./component/hooks/...) + +# 工具已注册(MCP Server 运行后调用 tools/list) +# 期望能看到 "search_traces" 与 "analyze_trace" + +# 空结果必须是正常结果而不是错误 +# 调用 search_traces {service: "<不存在的服务>", startTime: ""} +# 期望 isError=false 且 traces=[] 、total=0 +``` + +### 13.3 效果指标 + +- 观测类问题中,**不需要用户提供 traceId** 的比例(目标:trace 相关问题 100% 由 Agent 自主发现)。 +- `search_traces` → `analyze_trace` 的平均调用轮数(目标:定位单条 trace ≤ 2 轮工具调用)。 +- 工具返回中被裁剪(`truncated=true`)的比例,用于反向调整 `maxTraceSpans` 等上限。 + +## 14. 风险与待确认事项 + +### 14.1 结论状态(2026-10-25 已确认) + +| # | 事项 | 状态 | +| -- | -- | -- | +| 1 | Jaeger v1 查询 API 表达能力有限 | 保留为已知限制;如需更强检索后端(Jaeger gRPC query / Tempo search)在 Phase 3 一并考虑 | +| 2 | `service` 参数语义需在部署环境验证 | 保留为部署验证项,需真实 Jaeger 环境确认 | +| 3 | 是否暴露 Console API | **本期不做**,记为 To Do T1(控制台已有 Grafana trace 面板,再做一套会造成页面与日志冗余) | +| 4 | 采样缺口 | **结论**:采样是客户端行为,缺的是「数据不完整」这一信息;`search_traces` 返回需携带采样信息,记为 To Do T5 | +| 5 | token 指标是否对齐 GenAI token metric | **结论**:span 属性层已对齐;指标层(`gen_ai.client.inference.usage.*` + `gen_ai.token.modality`)本期不做——Admin 不产出 token 指标,成本由企业网关核算 | +| 6 | 规范版本钉死 | **结论**:以 `6fd0d76` 为锁定基准 commit,升级规范时必须重跑对齐检查 | +| 7 | `gen_ai.provider.name` 映射表 | **结论并已实现**:不做大而全映射表。按实际配置落地别名表(`gemini` → `gcp.gemini`),16 个 well-known 值落成参考集合但**从不过滤**枚举外的值——规范允许自定义值。模型名整串透传 | +| 8 | 多租户下的 trace 可见性 | 保留为部署确认项 | +| 9 | 多 provider 检索优先级 | **结论**:做成配置参数 `tracing.searchProviders`,留空时行为不变。记为 To Do T4(Phase 3) | +| 10 | `capture_content` 与大模型思考过程 | **结论并已实现**:拆成两个独立参数,新增 `capture_reasoning`(默认 `none`);思考过程按规范以 `reasoning` part 承载在 `gen_ai.output.messages` 中,已补上 `convertGenAIMessages` 的转换与 JSON 层策略过滤 | +| 11 | GenAI 异常事件的 severity | **结论**:Go SDK 的 span 事件无 severity 字段且 `EventOption` 不可在包外实现,先以 span status = Error 兜底;如后端需要按 severity 过滤,再以事件属性承载 severity number。记为 To Do T6 | +| 12 | 统一收集平台选型 | **结论**:Admin 只保证出口符合 OTLP 标准协议,不指定 URL、后端类型或厂商 SDK;企业可用 Collector 或代理统一接入 | +| 13 | 存储与脱敏 | **结论**:开发阶段全量提供,企业侧过滤逻辑自行处理;Admin 只需保证「出口标准、属性可过滤、不绑后端、敏感面可控」四项能力 | + +### 14.2 To Do 清单(本期不做,已记录) + +- [ ] **T1** trace 检索暴露为 Console API —— 控制台已有 Grafana trace 面板,再做一套会造成页面与日志冗余;等 Agent 侧能力稳定后评估。 +- [x] **T2** `gen_ai.provider.name` 取值映射 —— **已完成**。按 `ai/component/models/models.yaml` 的实际配置落地别名表:`gemini` → `gcp.gemini`(规范依据:base_url 为 `generativelanguage.googleapis.com`,规范对 `gcp.gemini` 的描述即「Also known as the AI Studio API」)。`dashscope`、`siliconflow` 无规范枚举项,按规范「otherwise, a custom value MAY be used」作为合法自定义值原样上报。模型名(第一个 `/` 之后的全部内容,含斜杠)整串透传。 +- [x] **T3** `capture_reasoning` 独立配置参数 + `gen_ai.output.messages` 的 reasoning part 转换 —— **已完成**。新增 `capture_reasoning`(`none`/`truncated`/`full`,默认 `none`),与 `capture_content` 正交;`convertGenAIMessages` 补上 `type: "reasoning"` 的 part 转换;策略在 JSON 层生效,先于整体截断执行。 +- [ ] **T4** 多 provider 检索合并的配置参数 `tracing.searchProviders`(留空时行为不变),Phase 3。 +- [ ] **T5** 检索结果携带采样信息,让 Agent 在返回 0 条时提示「当前采样率 X,结论仅供参考」,Phase 2。 +- [ ] **T6** GenAI 异常事件的 severity 表达(以事件属性承载 severity number),按需启用。 +- [ ] **T7** **工具返回的敏感内容过滤机制尚未实现**。传输错误已经净化:请求 URL(endpoint 路径与全部检索条件)不会进入工具返回,只有 `dial tcp ...` 这类诊断信息与模型自选的 traceId / service 锚点保留。**仍未净化**的是上游响应体与 span 内容——工具入参/返回值可能包含用户标识与业务数据。生产环境需要按部署场景决定过滤/脱敏口径(例如按工具白名单、按属性名过滤、或在 Collector 侧统一处理)。机制未实现,代码中已留 TODO 标记。 + +### 14.3 风险 + +| 风险 | 影响 | 缓解 | +| -- | -- | -- | +| `maxTraceSpans = 500` 在超大 trace 上裁剪子树 | 关键路径断裂、self time 失真 | 明确标记 `truncated`;延迟分母改用 provider 的完整包络,裁剪不再让占比虚高,但关键路径本身仍只在保留的 span 上计算;Phase 3 引入采样式采样或按服务子树裁剪 | +| 错误根因误判(中间层也是 ERROR) | 结论指向错误的服务 | 根因规则基于「无失败后代」;同时保留全部 `errorSpans` 供 Agent 复核 | +| 规范持续演进导致属性漂移 | 分析器识别不到 GenAI span | 钉住 commit;识别逻辑对未知 `gen_ai.operation.name` 做宽松归类(未知值仍标记为 GenAI span) | +| 与 #1607 的标签口径不一致 | Phase 2 联动失败 | Phase 1 结束前与 #1607 的 `dubbo.application` / `dubbo.metric.interface` 对齐命名 | +| 工具输出膨胀 | 挤占模型上下文 | 全部列表有上限,超限置 `truncated`;分析器不输出原始 span 列表 | + +## 15. 关联链接 + +### 15.1 讨论与 issue + +- [2026-10-24 Weekly(Discussion #1612)](https://github.com/apache/dubbo-admin/discussions/1612) —— 本 P0 任务的来源 +- [2026 路线图(Discussion #1386)](https://github.com/apache/dubbo-admin/discussions/1386) —— AI/SRE-Agent 第 7 项「设计Hook系统,接入otel trace」;可观测能力第 3 项「基于Grafana,探索metric,trace,log的联动」 +- [Issue #1606:Correlate Jaeger traces with Dubbo RPC metrics](https://github.com/apache/dubbo-admin/issues/1606) +- [PR #1607:feat(observability): correlate Jaeger traces with Dubbo RPC metrics](https://github.com/apache/dubbo-admin/pull/1607) +- [Issue #1537:[Proposal] Add log observability and metric-trace-log correlation](https://github.com/apache/dubbo-admin/issues/1537) +- [Issue #1524:[OSPP] Observability optimization in Admin](https://github.com/apache/dubbo-admin/issues/1524) +- [PR #1499:feat: add PromQL and trace diagnosis tools](https://github.com/apache/dubbo-admin/pull/1499) +- [PR #1536:feat(ai): add agent hooks and OpenTelemetry tracing](https://github.com/apache/dubbo-admin/pull/1536) + +### 15.2 规范 + +- OpenTelemetry semantic conventions for generative AI systems(只读 checkout,commit `6fd0d76`) + - `docs/gen-ai/README.md` + - `docs/gen-ai/gen-ai-spans.md` + - `docs/gen-ai/gen-ai-agent-spans.md` + - `docs/gen-ai/client-inference.md` + - `docs/gen-ai/gen-ai-events.md` + - `docs/gen-ai/gen-ai-exceptions.md` + - `docs/gen-ai/gen-ai-token-metrics.md` + - `docs/registry/attributes/gen-ai.md` +- [Document Status 定义](https://opentelemetry.io/docs/specs/otel/document-status) + +### 15.3 仓库内相关文件 + +| 路径 | 作用 | +| -- | -- | +| `pkg/mcp/tools/trace/provider.go` | `Provider` 接口与 Provider 工厂 | +| `pkg/mcp/tools/trace/tools.go` | `get_trace_by_id` handler、provider 解析、input schema | +| `pkg/mcp/tools/trace/jaeger.go` | Jaeger v1 Provider、归一化与有界裁剪 | +| `pkg/mcp/tools/trace/model.go` | `Trace`/`Span`/`Event` 与搜索、分析契约类型 | +| `pkg/mcp/tools/trace/search.go` | 新增:`search_traces` | +| `pkg/mcp/tools/trace/analyzer.go` | 新增:`analyze_trace` | +| `pkg/mcp/tools/trace/genai.go` | 新增:GenAI 属性识别 | +| `pkg/mcp/register.go` | MCP 工具注册 | +| `pkg/mcp/common/types.go`、`pkg/mcp/common/utils.go` | `ToolDef`、`PropertyDef`、`ToolResult`、`ArgsHelper`、`JsonResult`、`ErrorResult` | +| `pkg/config/observability/tracing.go` | `TracingConfig` / `TraceProviderConfig` | +| `pkg/mcp/tools/log/tools.go` | `search_logs`(支持 `traceId` 过滤) | +| `pkg/mcp/tools/metrics/prometheus.go` | `query_prometheus` / `query_prometheus_range` 与查询上限 | +| `pkg/mcp/tools/observability.go` | `get_observability_capabilities` | +| `ai/component/hooks/tracing.go` | Agent 侧 OTel hook | +| `ai/component/hooks/component.go` | OTLP exporter 与内容捕获策略常量 | +| `ai/component/hooks/config.go` | hook 配置与默认 `CaptureNone` | +| `docs/ai-context-management-design.md` | 同系列设计文档(上下文管理) | diff --git a/pkg/mcp/DEVELOPMENT.md b/pkg/mcp/DEVELOPMENT.md index 79f4121de..8db393f00 100644 --- a/pkg/mcp/DEVELOPMENT.md +++ b/pkg/mcp/DEVELOPMENT.md @@ -417,6 +417,8 @@ Dubbo Admin MCP 提供以下只读诊断工具: | `query_prometheus` | Prometheus | 执行即时 PromQL 查询 | | `query_prometheus_range` | Prometheus | 执行最长 24 小时的区间 PromQL 查询 | | `get_trace_by_id` | Jaeger | 按 TraceID 查询规范化调用链 | +| `search_traces` | Jaeger | 按服务、时间窗与过滤条件发现候选链路,返回逐条摘要 | +| `analyze_trace` | Jaeger | 对单条链路做结构化诊断:关键路径、耗时分解、错误归因、依赖边、GenAI 用量 | | `get_observability_capabilities` | 本地配置 | 查询数据源和查询限制 | Prometheus 工具复用 `observability.prometheus`。区间查询的 `step` 不得低于 15 秒,返回结果最多包含 100 个时间序列和 5000 个采样点。结果被裁剪时,响应中的 `truncated` 为 `true`。 @@ -436,3 +438,5 @@ observability: ``` 诊断工具不会记录完整 PromQL、Bearer Token 或完整 trace attributes。工具只读,不提供流量、配置或部署变更能力。 + +`search_traces` 必须提供 `service` 锚点,默认请求 20 条、最多 100 条,本地最多保留 100 条摘要;后端命中条数上限时 `truncated` 为 `true`。`analyze_trace` 最多输出 32 跳关键路径、10 个慢 span、20 个错误 span、20 个 GenAI 明细,并给出一句话结论。分析器只读取 `gen_ai.*` 元数据属性,不回显 prompt、completion 与工具入参等 Opt-In 内容。 diff --git a/pkg/mcp/register.go b/pkg/mcp/register.go index 2d8856a58..8733ecbd6 100644 --- a/pkg/mcp/register.go +++ b/pkg/mcp/register.go @@ -406,6 +406,28 @@ func RegisterTools(server *Server) { Handler: tracetools.GetTraceByID, }) + server.RegisterTool(&common.ToolDef{ + Name: "search_traces", + Description: "Search distributed traces by service, time window, and optional filters, returning bounded per-trace summaries to discover candidate traces", + InputSchema: common.InputSchema{ + Type: "object", + Properties: tracetools.SearchProperties(), + Required: []string{"service"}, + }, + Handler: tracetools.SearchTraces, + }) + + server.RegisterTool(&common.ToolDef{ + Name: "analyze_trace", + Description: "Diagnose a distributed trace by TraceID and return the critical path, service latency breakdown, slow spans, error root causes, dependency edges, and GenAI usage", + InputSchema: common.InputSchema{ + Type: "object", + Properties: tracetools.AnalyzeProperties(), + Required: []string{"traceId"}, + }, + Handler: tracetools.AnalyzeTrace, + }) + server.RegisterTool(&common.ToolDef{ Name: "get_observability_capabilities", Description: "Report Prometheus and trace providers, capabilities, and enforced query limits", diff --git a/pkg/mcp/register_test.go b/pkg/mcp/register_test.go index 5d423b657..af644b77a 100644 --- a/pkg/mcp/register_test.go +++ b/pkg/mcp/register_test.go @@ -67,6 +67,8 @@ func TestRegisterObservabilityTools(t *testing.T) { {name: "query_prometheus", required: []string{"query"}}, {name: "query_prometheus_range", required: []string{"query", "startTime", "endTime", "step"}}, {name: "get_trace_by_id", required: []string{"traceId"}}, + {name: "search_traces", required: []string{"service"}}, + {name: "analyze_trace", required: []string{"traceId"}}, {name: "get_observability_capabilities"}, } for _, test := range tests { diff --git a/pkg/mcp/tools/trace/analyzer.go b/pkg/mcp/tools/trace/analyzer.go new file mode 100644 index 000000000..0a097eab0 --- /dev/null +++ b/pkg/mcp/tools/trace/analyzer.go @@ -0,0 +1,831 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package trace + +import ( + "context" + "fmt" + "math" + "sort" + "strconv" + "strings" + "time" + "unicode/utf8" + + consolectx "github.com/apache/dubbo-admin/pkg/console/context" + "github.com/apache/dubbo-admin/pkg/mcp/common" +) + +const ( + // Bounded output limits. They keep the tool payload small enough to stay inside + // the model context window while still describing the most relevant spans. + maxCriticalPath = 32 + maxServiceBreakdown = 50 + maxSlowSpans = 10 + maxErrorSpans = 20 + maxDependencyEdges = 50 + maxGenAICalls = 20 + + maxAnalysisSummaryChars = 300 + maxErrorTypeChars = 96 + maxErrorMessageChars = 256 +) + +const ( + errorTypeAttribute = "error.type" + errorMessageAttribute = "error.message" + exceptionTypeAttribute = "exception.type" + exceptionMessageAttribute = "exception.message" + spanEventNameAttribute = "event" + standardExceptionEvent = "exception" + genAIExceptionEvent = "gen_ai.client.operation.exception" +) + +// analysisSpan is the analyzer's internal view of a span with a resolved timeline. +type analysisSpan struct { + span *Span + position int + startMicros int64 + endMicros int64 + timeValid bool + selfMicros int64 + waitMicros int64 + parent *analysisSpan + children []*analysisSpan +} + +// timelineInterval is a half-open interval on the absolute microsecond timeline. +type timelineInterval struct { + start int64 + end int64 +} + +// AnalyzeTrace diagnoses a single distributed trace and returns a structured report. +func AnalyzeTrace(ctx consolectx.Context, args map[string]any) (*common.ToolResult, error) { + helper := common.NewArgsHelper(args) + traceID := helper.GetString("traceId", "") + if !traceIDPattern.MatchString(traceID) { + return common.ErrorResult(fmt.Errorf("traceId must be a 16 or 32 character hexadecimal value")), nil + } + providerConfig, err := providerConfigFromContext(ctx, helper.GetString("provider", "")) + if err != nil { + return common.ErrorResult(err), nil + } + provider, err := newProvider(providerConfig) + if err != nil { + return common.ErrorResult(err), nil + } + trace, err := provider.GetTraceByID(traceRequestContext(ctx), traceID) + if err != nil { + return common.ErrorResult(err), nil + } + return common.JsonResult(BuildTraceAnalysis(trace)) +} + +// AnalyzeProperties returns the MCP input schema properties for trace diagnosis. +func AnalyzeProperties() map[string]common.PropertyDef { + return map[string]common.PropertyDef{ + "traceId": {Type: "string", Description: "Jaeger 64-bit or 128-bit hexadecimal TraceID"}, + "provider": {Type: "string", Description: "Optional trace provider name; defaults to defaultProvider"}, + } +} + +// BuildTraceAnalysis turns a normalized trace into a structured diagnosis. +// It is a pure function so the diagnosis rules can be verified without a backend. +func BuildTraceAnalysis(trace *Trace) *TraceAnalysis { + analysis := &TraceAnalysis{} + if trace == nil { + analysis.Summary = "no trace data available" + return analysis + } + analysis.TraceID = trace.TraceID + analysis.Truncated = trace.Truncated + + nodes := buildAnalysisTree(trace.Spans) + computeSelfTimes(nodes, analysis) + criticalPath, criticalPathSet := buildCriticalPath(nodes, analysis) + analysis.CriticalPath = criticalPath + + wallClock, traceStart := analysisEnvelope(nodes, trace) + analysis.DurationMicros = wallClock + analysis.StartTime = traceStart + analysis.SpanCount = maxInt(trace.SpanCount, len(trace.Spans)) + + breakdown := buildServiceBreakdown(nodes, wallClock, analysis) + analysis.ServiceBreakdown = breakdown + analysis.ServiceCount = maxInt(trace.ServiceCount, len(breakdown)) + + slowSpans := buildSlowSpans(nodes, wallClock, criticalPathSet) + analysis.SlowSpans = make([]SlowSpan, 0, len(slowSpans)) + for _, entry := range slowSpans { + analysis.SlowSpans = append(analysis.SlowSpans, entry.slow) + } + + errorSpans, errorCount := buildErrorSpans(nodes, analysis) + analysis.ErrorSpans = errorSpans + // The provider already counted errors over the complete trace; that count must not + // be hidden by the bounded error span list. + analysis.ErrorSpanCount = maxInt(trace.ErrorSpanCount, errorCount) + + analysis.Dependencies = buildDependencyEdges(nodes, analysis) + + if genai := BuildGenAISummary(trace.Spans); genai.Present { + analysis.GenAI = genai + } + + analysis.Summary = buildAnalysisSummary(analysis, slowSpans) + return analysis +} + +// buildAnalysisTree links the spans into a parent/child forest. A span whose parent +// is missing from the trace degrades into an orphan root: it still contributes to the +// aggregate statistics but it cannot extend a parent chain. +func buildAnalysisTree(spans []Span) []*analysisSpan { + nodes := make([]*analysisSpan, 0, len(spans)) + index := make(map[string]*analysisSpan, len(spans)) + for i := range spans { + span := &spans[i] + node := &analysisSpan{span: span, position: i} + if parsed, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(span.StartTime)); err == nil { + node.timeValid = true + node.startMicros = parsed.UnixMicro() + node.endMicros = node.startMicros + span.DurationMicros + } else { + node.endMicros = span.DurationMicros + } + nodes = append(nodes, node) + if span.SpanID != "" { + index[span.SpanID] = node + } + } + for _, node := range nodes { + parentID := node.span.ParentSpanID + if parentID == "" { + continue + } + parent, ok := index[parentID] + if !ok || parent == node { + continue + } + node.parent = parent + parent.children = append(parent.children, node) + } + return nodes +} + +// computeSelfTimes derives the exclusive time of every span. The covered time is the +// union of the clipped child intervals, so concurrent children are not counted twice. +func computeSelfTimes(nodes []*analysisSpan, analysis *TraceAnalysis) { + for _, node := range nodes { + if !node.timeValid { + // Without a trustworthy start time the span degrades to a leaf. It + // cannot be placed on the timeline, so it must not claim any time + // of its own: counting its full duration here would let the parent + // account for the same time a second time. + node.selfMicros = 0 + node.waitMicros = 0 + analysis.Truncated = true + continue + } + intervals := make([]timelineInterval, 0, len(node.children)) + for _, child := range node.children { + if !child.timeValid { + analysis.Truncated = true + continue + } + start := maxInt64(child.startMicros, node.startMicros) + end := minInt64(child.endMicros, node.endMicros) + if end <= start { + continue + } + intervals = append(intervals, timelineInterval{start: start, end: end}) + } + covered := unionLength(intervals) + node.waitMicros = covered + node.selfMicros = node.span.DurationMicros - covered + if node.selfMicros < 0 { + node.selfMicros = 0 + } + } +} + +// unionLength returns the length of the union of the given intervals. +func unionLength(intervals []timelineInterval) int64 { + if len(intervals) == 0 { + return 0 + } + sort.Slice(intervals, func(i, j int) bool { + if intervals[i].start != intervals[j].start { + return intervals[i].start < intervals[j].start + } + return intervals[i].end < intervals[j].end + }) + total := int64(0) + current := intervals[0] + for _, next := range intervals[1:] { + if next.start > current.end { + total += current.end - current.start + current = next + continue + } + if next.end > current.end { + current.end = next.end + } + } + return total + current.end - current.start +} + +// buildCriticalPath walks the latency determining root-to-leaf chain. +// The second return value holds the analyzer internal positions of the spans on the +// path. The position is used instead of the business span id because a span id can +// be empty or duplicated, which would make two different spans overwrite each other. +func buildCriticalPath(nodes []*analysisSpan, analysis *TraceAnalysis) ([]CriticalPathEntry, map[int]struct{}) { + roots := make([]*analysisSpan, 0, len(nodes)) + for _, node := range nodes { + if node.parent == nil { + roots = append(roots, node) + } + } + entries := make([]CriticalPathEntry, 0, maxCriticalPath) + onPath := map[int]struct{}{} + visited := map[int]struct{}{} + current := selectCriticalRoot(roots) + for current != nil { + if len(entries) >= maxCriticalPath { + // The budget is exhausted while the chain still continues, so the + // remaining hops are dropped and the analysis is incomplete. + analysis.Truncated = true + break + } + if _, seen := visited[current.position]; seen { + break + } + visited[current.position] = struct{}{} + onPath[current.position] = struct{}{} + entries = append(entries, CriticalPathEntry{ + SpanID: current.span.SpanID, + ServiceName: current.span.ServiceName, + OperationName: current.span.OperationName, + StartTime: current.span.StartTime, + DurationMicros: current.span.DurationMicros, + SelfMicros: current.selfMicros, + WaitMicros: current.waitMicros, + }) + current = selectCriticalChild(current) + } + return entries, onPath +} + +// selectCriticalRoot picks the root that finishes last, because that root determines +// the end of the request. +func selectCriticalRoot(roots []*analysisSpan) *analysisSpan { + var best *analysisSpan + for _, root := range roots { + if !root.timeValid { + continue + } + if best == nil || betterCriticalRoot(root, best) { + best = root + } + } + if best != nil { + return best + } + for _, root := range roots { + if best == nil || betterCriticalRoot(root, best) { + best = root + } + } + return best +} + +func betterCriticalRoot(candidate, current *analysisSpan) bool { + if candidate.endMicros != current.endMicros { + return candidate.endMicros > current.endMicros + } + if candidate.span.DurationMicros != current.span.DurationMicros { + return candidate.span.DurationMicros > current.span.DurationMicros + } + // The start time is derived from the end time and the duration, so it is + // already equal here and cannot be used as a tie breaker. + return candidate.span.SpanID < current.span.SpanID +} + +// selectCriticalChild picks the child that finishes last, because it is the activity +// that determined when the parent completed. +func selectCriticalChild(node *analysisSpan) *analysisSpan { + var best *analysisSpan + for _, child := range node.children { + if !child.timeValid { + // A span without a trustworthy timestamp degrades to a leaf and is not + // allowed to extend the chain. + continue + } + if best == nil || betterCriticalChild(child, best) { + best = child + } + } + return best +} + +func betterCriticalChild(candidate, current *analysisSpan) bool { + if candidate.endMicros != current.endMicros { + return candidate.endMicros > current.endMicros + } + if candidate.span.DurationMicros != current.span.DurationMicros { + return candidate.span.DurationMicros > current.span.DurationMicros + } + return candidate.span.SpanID < current.span.SpanID +} + +// analysisEnvelope returns the wall clock duration and the start timestamp of the trace. +func analysisEnvelope(nodes []*analysisSpan, trace *Trace) (int64, string) { + var minStart, maxEnd int64 + hasStart := false + startText := trace.StartTime + for _, node := range nodes { + if !node.timeValid { + continue + } + if !hasStart || node.startMicros < minStart { + minStart = node.startMicros + startText = node.span.StartTime + hasStart = true + } + if node.endMicros > maxEnd { + maxEnd = node.endMicros + } + } + if !hasStart { + return trace.DurationMicros, trace.StartTime + } + // The provider envelope covers every raw span, the node envelope only the + // retained ones. A clipped span list would make the node envelope shorter + // than the trace and inflate every latency percentage that uses it as the + // denominator, so the provider value is preferred whenever it is available. + if trace.DurationMicros > 0 { + return trace.DurationMicros, startText + } + return maxEnd - minStart, startText +} + +// buildServiceBreakdown aggregates the spans per service. +func buildServiceBreakdown(nodes []*analysisSpan, wallClock int64, analysis *TraceAnalysis) []ServiceLatency { + aggregates := map[string]*ServiceLatency{} + for _, node := range nodes { + span := node.span + entry, ok := aggregates[span.ServiceName] + if !ok { + entry = &ServiceLatency{ServiceName: span.ServiceName} + aggregates[span.ServiceName] = entry + } + entry.SpanCount++ + if spanFailed(span) { + entry.ErrorSpanCount++ + } + entry.TotalMicros += span.DurationMicros + if span.DurationMicros > entry.MaxMicros { + entry.MaxMicros = span.DurationMicros + } + } + result := make([]ServiceLatency, 0, len(aggregates)) + for _, entry := range aggregates { + if entry.SpanCount > 0 { + entry.AvgMicros = entry.TotalMicros / int64(entry.SpanCount) + } + // The share is the service workload relative to the wall clock of the trace. + // Concurrent spans can push it above 100 percent on purpose. + entry.SharePercent = percentOf(entry.TotalMicros, wallClock) + result = append(result, *entry) + } + sort.SliceStable(result, func(i, j int) bool { + if result[i].TotalMicros != result[j].TotalMicros { + return result[i].TotalMicros > result[j].TotalMicros + } + return result[i].ServiceName < result[j].ServiceName + }) + if len(result) > maxServiceBreakdown { + result = result[:maxServiceBreakdown] + analysis.Truncated = true + } + return result +} + +// slowSpanEntry pairs a slow span with the analyzer internal identity of the span +// it was derived from. The position is used instead of the business span id so +// that spans without an id, or with duplicated ids, cannot overwrite each other. +type slowSpanEntry struct { + slow SlowSpan + position int + selfMicros int64 +} + +// buildSlowSpans ranks the spans by the time they consumed on their own. +// The exclusive time is returned alongside the report so the summary can quote it +// even for spans that are not part of the critical path. +func buildSlowSpans(nodes []*analysisSpan, wallClock int64, criticalPath map[int]struct{}) []slowSpanEntry { + candidates := make([]slowSpanEntry, 0, len(nodes)) + for _, node := range nodes { + span := node.span + _, onPath := criticalPath[node.position] + candidates = append(candidates, slowSpanEntry{ + slow: SlowSpan{ + SpanID: span.SpanID, + ServiceName: span.ServiceName, + OperationName: span.OperationName, + DurationMicros: span.DurationMicros, + SharePercent: percentOf(node.selfMicros, wallClock), + OnCriticalPath: onPath, + }, + position: node.position, + selfMicros: node.selfMicros, + }) + } + sort.SliceStable(candidates, func(i, j int) bool { + if candidates[i].selfMicros != candidates[j].selfMicros { + return candidates[i].selfMicros > candidates[j].selfMicros + } + return candidates[i].position < candidates[j].position + }) + // The list is a ranking, so cutting it to the top entries does not lose any + // analysis data. The truncated flag stays untouched: it would tell the agent + // that the diagnosis is based on incomplete data, and a ranking is complete + // even when only its best entries are reported. + if len(candidates) > maxSlowSpans { + candidates = candidates[:maxSlowSpans] + } + return candidates +} + +// errorSpanEntry pairs an error span with the analyzer node it was derived from. +// The node is kept so the ordering can use the start time of the span without +// indexing the table by the business span id. +type errorSpanEntry struct { + span ErrorSpan + node *analysisSpan +} + +// buildErrorSpans collects the failed spans and marks the root causes. +func buildErrorSpans(nodes []*analysisSpan, analysis *TraceAnalysis) ([]ErrorSpan, int) { + failed := make([]*analysisSpan, 0, len(nodes)) + for _, node := range nodes { + if spanFailed(node.span) { + failed = append(failed, node) + } + } + entries := make([]errorSpanEntry, 0, len(failed)) + for _, node := range failed { + span := node.span + errorType, errorMessage := errorDetails(span) + entries = append(entries, errorSpanEntry{ + span: ErrorSpan{ + SpanID: span.SpanID, + ServiceName: span.ServiceName, + OperationName: span.OperationName, + ErrorType: errorType, + ErrorMessage: errorMessage, + Status: errorStatus(span), + RootCause: !hasFailedDescendant(node), + }, + node: node, + }) + } + // Root causes come first so the agent sees the deepest failure before the + // ancestors that only propagate it. + sort.SliceStable(entries, func(i, j int) bool { + left, right := entries[i], entries[j] + if left.span.RootCause != right.span.RootCause { + return left.span.RootCause + } + if left.node.startMicros != right.node.startMicros { + return left.node.startMicros < right.node.startMicros + } + return left.node.position < right.node.position + }) + count := len(entries) + if len(entries) > maxErrorSpans { + entries = entries[:maxErrorSpans] + analysis.Truncated = true + } + result := make([]ErrorSpan, 0, len(entries)) + for _, entry := range entries { + result = append(result, entry.span) + } + return result, count +} + +// errorDetails resolves the error classification and the human readable description. +// +// The sources are ordered "generative AI first, standard second, non standard +// last": +// - The span attribute error.type stays the leading source because the OpenTelemetry +// error registry defines it as the stable error type of a span. +// - The generative AI exception event comes before the standard exception event +// because this project serves the generative AI execution of the agent loop, so +// the typed error of the model call is the more relevant classification. +// - error.message is only the last resort because it is not an OpenTelemetry +// standard attribute: the registry only defines error.type, and the stable +// attribute for the exception text is exception.message. error.message is a +// habit of the Java ecosystem and of some collectors, so it is used only when +// no typed error is available, and it then degrades into the error type. +func errorDetails(span *Span) (string, string) { + errorType := "" + if value, ok := lookupAttributeOK(span.Attributes, errorTypeAttribute); ok { + errorType = strings.TrimSpace(value) + } + exceptionType, exceptionMessage := exceptionDetails(span) + if errorType == "" { + errorType = exceptionType + } + errorMessage := exceptionMessage + if errorMessage == "" { + if value, ok := lookupAttributeOK(span.Attributes, errorMessageAttribute); ok { + errorMessage = strings.TrimSpace(value) + } + } + if errorType == "" && errorMessage != "" { + // Degraded classification: without a typed error the message is the only + // signal left, so it degrades into the error type. + errorType = errorMessage + } + if errorType == "" { + errorType = "unknown" + } + return truncateRunes(errorType, maxErrorTypeChars), truncateRunes(errorMessage, maxErrorMessageChars) +} + +// exceptionDetails reads the exception type and message from the span events. +// The generative AI exception event wins over the standard exception event, +// because this project serves the generative AI execution of the agent loop and +// the generative AI event carries the typed error of the model call. Collectors +// also report the exception as a plain span attribute, so the attributes are read +// as a fallback once the events have been exhausted. +func exceptionDetails(span *Span) (string, string) { + exceptionType := "" + exceptionMessage := "" + collect := func(attributes map[string]string) { + if exceptionType == "" { + if value, ok := lookupAttributeOK(attributes, exceptionTypeAttribute); ok { + exceptionType = strings.TrimSpace(value) + } + } + if exceptionMessage == "" { + if value, ok := lookupAttributeOK(attributes, exceptionMessageAttribute); ok { + exceptionMessage = strings.TrimSpace(value) + } + } + } + for _, event := range span.Events { + if strings.EqualFold(eventName(event), genAIExceptionEvent) { + collect(event.Attributes) + } + } + for _, event := range span.Events { + if strings.EqualFold(eventName(event), standardExceptionEvent) { + collect(event.Attributes) + } + } + // The span attributes are the last resort: the two documented exception + // events stay the primary source, and no other event is consulted. + collect(span.Attributes) + return exceptionType, exceptionMessage +} + +func eventName(event Event) string { + if value, ok := lookupAttributeOK(event.Attributes, spanEventNameAttribute); ok { + return value + } + return "" +} + +// errorStatus returns the normalized status of a failed span. +func errorStatus(span *Span) string { + if spanFailed(span) { + return "ERROR" + } + if span.Status == "" { + return "UNSET" + } + return span.Status +} + +// hasFailedDescendant reports whether a failed span only propagates an error that +// happened deeper in the trace. +func hasFailedDescendant(node *analysisSpan) bool { + visited := map[int]struct{}{} + stack := make([]*analysisSpan, 0, len(node.children)) + stack = append(stack, node.children...) + for len(stack) > 0 { + current := stack[len(stack)-1] + stack = stack[:len(stack)-1] + if _, seen := visited[current.position]; seen { + continue + } + visited[current.position] = struct{}{} + if spanFailed(current.span) { + return true + } + stack = append(stack, current.children...) + } + return false +} + +// buildDependencyEdges derives the cross service call graph of the trace. +func buildDependencyEdges(nodes []*analysisSpan, analysis *TraceAnalysis) []DependencyEdge { + aggregates := map[string]*DependencyEdge{} + for _, node := range nodes { + if node.parent == nil { + continue + } + caller := node.parent.span.ServiceName + callee := node.span.ServiceName + if caller == callee { + continue + } + key := caller + "\x00" + callee + edge, ok := aggregates[key] + if !ok { + edge = &DependencyEdge{CallerService: caller, CalleeService: callee} + aggregates[key] = edge + } + edge.SpanCount++ + edge.TotalMicros += node.span.DurationMicros + if spanFailed(node.span) { + edge.ErrorCount++ + } + } + result := make([]DependencyEdge, 0, len(aggregates)) + for _, edge := range aggregates { + result = append(result, *edge) + } + sort.SliceStable(result, func(i, j int) bool { + if result[i].TotalMicros != result[j].TotalMicros { + return result[i].TotalMicros > result[j].TotalMicros + } + if result[i].CallerService != result[j].CallerService { + return result[i].CallerService < result[j].CallerService + } + return result[i].CalleeService < result[j].CalleeService + }) + if len(result) > maxDependencyEdges { + result = result[:maxDependencyEdges] + analysis.Truncated = true + } + return result +} + +// buildAnalysisSummary renders the one line conclusion of the diagnosis. +func buildAnalysisSummary(analysis *TraceAnalysis, slowSpans []slowSpanEntry) string { + base := fmt.Sprintf("trace %d %s, %d %s, %s, %d %s", + analysis.ServiceCount, pluralize(analysis.ServiceCount, "service", "services"), + analysis.SpanCount, pluralize(analysis.SpanCount, "span", "spans"), + formatMicros(analysis.DurationMicros), + analysis.ErrorSpanCount, pluralize(analysis.ErrorSpanCount, "error", "errors")) + + var slowPart string + if len(slowSpans) > 0 { + slowest := slowSpans[0] + slowPart = fmt.Sprintf("; slowest: %s/%s self %s of %s (%.0f%% of trace)", + sanitizeSummaryText(slowest.slow.ServiceName), sanitizeSummaryText(slowest.slow.OperationName), + formatMicros(slowest.selfMicros), formatMicros(slowest.slow.DurationMicros), slowest.slow.SharePercent) + } + var rootPart string + for _, span := range analysis.ErrorSpans { + if !span.RootCause { + continue + } + rootPart = fmt.Sprintf("; root cause: %s/%s %s %s", + sanitizeSummaryText(span.ServiceName), sanitizeSummaryText(span.OperationName), + span.Status, sanitizeSummaryText(span.ErrorType)) + break + } + for _, candidate := range []string{base + slowPart + rootPart, base + rootPart, base} { + // The budget counts runes, not bytes: a Chinese service name uses three + // bytes per character and would otherwise push the diagnosis out of the + // summary even though it still fits. + if utf8.RuneCountInString(candidate) <= maxAnalysisSummaryChars { + return candidate + } + } + return truncateRunes(base+slowPart+rootPart, maxAnalysisSummaryChars) +} + +// pluralize returns the singular or the plural form depending on the count. +func pluralize(count int, singular, plural string) string { + if count == 1 { + return singular + } + return plural +} + +// lookupAttributeOK resolves an attribute value, tolerating casing and padding +// differences introduced by different collectors. +func lookupAttributeOK(attributes map[string]string, name string) (string, bool) { + if value, ok := attributes[name]; ok { + return value, true + } + bestKey := "" + bestValue := "" + found := false + for key, value := range attributes { + if !strings.EqualFold(strings.TrimSpace(key), name) { + continue + } + if !found || key < bestKey { + bestKey, bestValue, found = key, value, true + } + } + return bestValue, found +} + +// percentOf computes a share with two decimals. A zero denominator yields zero +// instead of a division by zero. +func percentOf(value, total int64) float64 { + if total <= 0 { + return 0 + } + return math.Round(float64(value)/float64(total)*100*100) / 100 +} + +// formatMicros renders a microsecond duration in a compact human readable form. +func formatMicros(micros int64) string { + switch { + case micros < 1000: + return strconv.FormatInt(micros, 10) + "us" + case micros < 1000*1000: + return trimDecimal(float64(micros)/1000, 1) + "ms" + default: + return trimDecimal(float64(micros)/(1000*1000), 2) + "s" + } +} + +func trimDecimal(value float64, decimals int) string { + formatted := strconv.FormatFloat(value, 'f', decimals, 64) + if strings.Contains(formatted, ".") { + formatted = strings.TrimRight(formatted, "0") + formatted = strings.TrimSuffix(formatted, ".") + } + return formatted +} + +// sanitizeSummaryText keeps the one line summary free of line breaks. +func sanitizeSummaryText(value string) string { + replaced := strings.NewReplacer("\r", " ", "\n", " ", "\t", " ").Replace(value) + return strings.TrimSpace(replaced) +} + +// truncateRunes limits a string by rune count so multi byte characters stay intact. +func truncateRunes(value string, limit int) string { + if limit <= 0 { + return "" + } + runes := []rune(value) + if len(runes) <= limit { + return value + } + return string(runes[:limit]) +} + +func maxInt(a, b int) int { + if a > b { + return a + } + return b +} + +func maxInt64(a, b int64) int64 { + if a > b { + return a + } + return b +} + +func minInt64(a, b int64) int64 { + if a < b { + return a + } + return b +} + +// traceRequestContext derives the outbound request context from the console context. +func traceRequestContext(ctx consolectx.Context) context.Context { + if ctx == nil || ctx.AppContext() == nil { + return context.Background() + } + return ctx.AppContext() +} diff --git a/pkg/mcp/tools/trace/analyzer_test.go b/pkg/mcp/tools/trace/analyzer_test.go new file mode 100644 index 000000000..0a872141f --- /dev/null +++ b/pkg/mcp/tools/trace/analyzer_test.go @@ -0,0 +1,841 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package trace + +import ( + "fmt" + "strings" + "testing" + "time" + "unicode/utf8" +) + +// analysisBase is the reference point of the test timeline. +const analysisBase = "2026-10-24T10:00:00.000000000Z" + +func spanAt(offsetMicros int64, spanID, parentID, service, operation string, durationMicros int64) Span { + return Span{ + SpanID: spanID, + ParentSpanID: parentID, + ServiceName: service, + OperationName: operation, + StartTime: shiftTime(analysisBase, offsetMicros), + DurationMicros: durationMicros, + } +} + +func shiftTime(base string, offsetMicros int64) string { + parsed, err := time.Parse(time.RFC3339Nano, base) + if err != nil { + panic(err) + } + return parsed.Add(time.Duration(offsetMicros) * time.Microsecond).UTC().Format(time.RFC3339Nano) +} + +func TestBuildTraceAnalysisLinearChain(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "GET /checkout", 100000), + spanAt(10000, "02", "01", "order-service", "checkout", 80000), + spanAt(20000, "03", "02", "payment-service", "charge", 50000), + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.TraceID != testTraceID { + t.Fatalf("unexpected trace id: %s", analysis.TraceID) + } + if analysis.SpanCount != 3 || analysis.ServiceCount != 3 { + t.Fatalf("unexpected counters: %#v", analysis) + } + if analysis.DurationMicros != 100000 { + t.Fatalf("unexpected duration: %d", analysis.DurationMicros) + } + if len(analysis.CriticalPath) != 3 { + t.Fatalf("unexpected critical path: %#v", analysis.CriticalPath) + } + if analysis.CriticalPath[0].SpanID != "01" || analysis.CriticalPath[2].SpanID != "03" { + t.Fatalf("unexpected critical path order: %#v", analysis.CriticalPath) + } + // The root waits for the child for 80ms, the child waits for its own child for 50ms. + if analysis.CriticalPath[0].SelfMicros != 20000 || analysis.CriticalPath[0].WaitMicros != 80000 { + t.Fatalf("unexpected root self time: %#v", analysis.CriticalPath[0]) + } + if analysis.CriticalPath[1].SelfMicros != 30000 || analysis.CriticalPath[1].WaitMicros != 50000 { + t.Fatalf("unexpected child self time: %#v", analysis.CriticalPath[1]) + } + if analysis.CriticalPath[2].SelfMicros != 50000 || analysis.CriticalPath[2].WaitMicros != 0 { + t.Fatalf("unexpected leaf self time: %#v", analysis.CriticalPath[2]) + } +} + +func TestBuildTraceAnalysisCriticalPathFollowsSlowBranch(t *testing.T) { + // The root fans out into a fast branch and a slow branch. The critical path has + // to follow the branch that finishes last. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "GET /checkout", 200000), + spanAt(1000, "02", "01", "order-service", "fast", 10000), + spanAt(2000, "03", "01", "payment-service", "slow", 150000), + }, + } + analysis := BuildTraceAnalysis(trace) + if len(analysis.CriticalPath) != 2 { + t.Fatalf("unexpected critical path: %#v", analysis.CriticalPath) + } + if analysis.CriticalPath[1].SpanID != "03" || analysis.CriticalPath[1].ServiceName != "payment-service" { + t.Fatalf("expected the critical path to follow the slow branch: %#v", analysis.CriticalPath) + } +} + +func TestBuildTraceAnalysisCriticalPathPrefersLatestRoot(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "early", 10000), + spanAt(5000, "02", "", "frontend", "late", 60000), + }, + } + analysis := BuildTraceAnalysis(trace) + if len(analysis.CriticalPath) != 1 || analysis.CriticalPath[0].SpanID != "02" { + t.Fatalf("expected the latest root to start the critical path: %#v", analysis.CriticalPath) + } +} + +func TestBuildTraceAnalysisSelfTimeUsesUnionOfChildren(t *testing.T) { + // Two concurrent children overlap, so the covered time is the union and not the + // sum of both durations. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + spanAt(10000, "02", "01", "order-service", "a", 50000), + spanAt(20000, "03", "01", "order-service", "b", 50000), + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.CriticalPath[0].WaitMicros != 60000 { + t.Fatalf("expected the union of the concurrent children, got %d", analysis.CriticalPath[0].WaitMicros) + } + if analysis.CriticalPath[0].SelfMicros != 40000 { + t.Fatalf("unexpected self time: %d", analysis.CriticalPath[0].SelfMicros) + } +} + +func TestBuildTraceAnalysisSelfTimeClipsChildrenToParent(t *testing.T) { + // The child reaches beyond the parent, so only the overlap counts. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + spanAt(50000, "02", "01", "order-service", "child", 900000), + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.CriticalPath[0].WaitMicros != 50000 { + t.Fatalf("expected the child to be clipped, got %d", analysis.CriticalPath[0].WaitMicros) + } + if analysis.CriticalPath[0].SelfMicros != 50000 { + t.Fatalf("unexpected self time: %d", analysis.CriticalPath[0].SelfMicros) + } +} + +func TestBuildTraceAnalysisServiceBreakdown(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + spanAt(10000, "02", "01", "order-service", "a", 60000), + spanAt(20000, "03", "01", "order-service", "b", 30000), + }, + } + analysis := BuildTraceAnalysis(trace) + if len(analysis.ServiceBreakdown) != 2 { + t.Fatalf("unexpected breakdown: %#v", analysis.ServiceBreakdown) + } + order := analysis.ServiceBreakdown[0] + if order.ServiceName != "frontend" || order.TotalMicros != 100000 || order.SpanCount != 1 { + t.Fatalf("unexpected breakdown entry: %#v", order) + } + if order.MaxMicros != 100000 || order.AvgMicros != 100000 { + t.Fatalf("unexpected breakdown statistics: %#v", order) + } + if order.SharePercent != 100 { + t.Fatalf("unexpected share: %v", order.SharePercent) + } + second := analysis.ServiceBreakdown[1] + if second.ServiceName != "order-service" || second.TotalMicros != 90000 || second.SpanCount != 2 { + t.Fatalf("unexpected breakdown entry: %#v", second) + } + if second.MaxMicros != 60000 || second.AvgMicros != 45000 { + t.Fatalf("unexpected breakdown statistics: %#v", second) + } + // The service workload is 90ms against a 100ms wall clock, concurrent spans may + // push the share above 100 percent but never below the real ratio. + if second.SharePercent != 90 { + t.Fatalf("unexpected share: %v", second.SharePercent) + } +} + +func TestBuildTraceAnalysisServiceBreakdownSortsByTotalTime(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + spanAt(10000, "02", "01", "alpha-service", "a", 50000), + spanAt(20000, "03", "01", "beta-service", "b", 50000), + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.ServiceBreakdown[1].ServiceName != "alpha-service" { + t.Fatalf("expected equal totals to be sorted by service name: %#v", analysis.ServiceBreakdown) + } +} + +func TestBuildTraceAnalysisSlowSpansUseSelfTime(t *testing.T) { + // The early branch consumes the most time on its own, but it is not the branch + // that finishes last, so it is not part of the critical path. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 500000), + spanAt(0, "02", "01", "order-service", "early-branch", 400000), + spanAt(100000, "03", "01", "payment-service", "late-branch", 380000), + }, + } + analysis := BuildTraceAnalysis(trace) + if len(analysis.SlowSpans) != 3 { + t.Fatalf("unexpected slow spans: %#v", analysis.SlowSpans) + } + if analysis.SlowSpans[0].SpanID != "02" { + t.Fatalf("expected the branch with the largest self time to rank first: %#v", analysis.SlowSpans) + } + if analysis.SlowSpans[0].DurationMicros != 400000 { + t.Fatalf("expected the total duration in the slow span: %#v", analysis.SlowSpans[0]) + } + if analysis.SlowSpans[0].SharePercent != 80 { + t.Fatalf("unexpected share: %v", analysis.SlowSpans[0].SharePercent) + } + if analysis.SlowSpans[0].OnCriticalPath { + t.Fatal("did not expect the early branch to be part of the critical path") + } + if !analysis.SlowSpans[1].OnCriticalPath { + t.Fatal("expected the late branch to be part of the critical path") + } + if len(analysis.CriticalPath) != 2 || analysis.CriticalPath[1].SpanID != "03" { + t.Fatalf("unexpected critical path: %#v", analysis.CriticalPath) + } +} + +func TestBuildTraceAnalysisDependencyEdges(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + spanAt(10000, "02", "01", "order-service", "a", 40000), + spanAt(20000, "03", "01", "order-service", "b", 30000), + spanAt(30000, "04", "02", "payment-service", "charge", 20000), + }, + } + analysis := BuildTraceAnalysis(trace) + if len(analysis.Dependencies) != 2 { + t.Fatalf("unexpected dependencies: %#v", analysis.Dependencies) + } + edges := map[string]DependencyEdge{} + for _, edge := range analysis.Dependencies { + edges[edge.CallerService+"->"+edge.CalleeService] = edge + } + frontend := edges["frontend->order-service"] + if frontend.SpanCount != 2 || frontend.TotalMicros != 70000 || frontend.ErrorCount != 0 { + t.Fatalf("unexpected edge: %#v", frontend) + } + payment := edges["order-service->payment-service"] + if payment.SpanCount != 1 || payment.TotalMicros != 20000 { + t.Fatalf("unexpected edge: %#v", payment) + } + // The edge with the largest total time comes first. + if analysis.Dependencies[0].CallerService != "frontend" { + t.Fatalf("unexpected dependency order: %#v", analysis.Dependencies) + } +} + +func TestBuildTraceAnalysisErrorAttributionAndRootCause(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + spanAt(10000, "02", "01", "order-service", "checkout", 80000), + spanAt(20000, "03", "02", "payment-service", "charge", 50000), + }, + } + trace.Spans[0].Error = true + trace.Spans[0].Status = "ERROR" + trace.Spans[0].Attributes = map[string]string{"error.type": "_OTHER"} + trace.Spans[1].Error = true + trace.Spans[1].Status = "ERROR" + trace.Spans[1].Attributes = map[string]string{"error.type": "timeout", "error.message": "payment gateway timeout"} + trace.Spans[2].Error = true + trace.Spans[2].Status = "ERROR" + + analysis := BuildTraceAnalysis(trace) + if analysis.ErrorSpanCount != 3 { + t.Fatalf("unexpected error count: %d", analysis.ErrorSpanCount) + } + if len(analysis.ErrorSpans) != 3 { + t.Fatalf("unexpected error spans: %#v", analysis.ErrorSpans) + } + // The deepest failure is the root cause, the ancestors only propagate it. + if !analysis.ErrorSpans[0].RootCause || analysis.ErrorSpans[0].ServiceName != "payment-service" { + t.Fatalf("expected the deepest failure to be the root cause: %#v", analysis.ErrorSpans) + } + for _, span := range analysis.ErrorSpans[1:] { + if span.RootCause { + t.Fatalf("expected the propagated failures to not be root causes: %#v", analysis.ErrorSpans) + } + } + byService := map[string]ErrorSpan{} + for _, span := range analysis.ErrorSpans { + byService[span.ServiceName] = span + } + order := byService["order-service"] + if order.ErrorType != "timeout" { + t.Fatalf("unexpected error type: %#v", order) + } + if order.ErrorMessage != "payment gateway timeout" { + t.Fatalf("unexpected error message: %#v", order) + } + if byService["frontend"].ErrorType != "_OTHER" { + t.Fatalf("unexpected error type: %#v", byService["frontend"]) + } + if byService["payment-service"].ErrorType != "unknown" { + t.Fatalf("expected a fallback error type: %#v", byService["payment-service"]) + } +} + +func TestBuildTraceAnalysisReadsExceptionEvent(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "payment-service", "charge", 50000), + }, + } + trace.Spans[0].Error = true + trace.Spans[0].Status = "ERROR" + trace.Spans[0].Events = []Event{{ + Timestamp: analysisBase, + Attributes: map[string]string{ + "event": "exception", + "exception.type": "java.net.SocketTimeoutException", + "exception.message": "Read timed out", + }, + }} + analysis := BuildTraceAnalysis(trace) + if analysis.ErrorSpans[0].ErrorType != "java.net.SocketTimeoutException" { + t.Fatalf("unexpected error type: %#v", analysis.ErrorSpans[0]) + } + if analysis.ErrorSpans[0].ErrorMessage != "Read timed out" { + t.Fatalf("unexpected error message: %#v", analysis.ErrorSpans[0]) + } +} + +func TestBuildTraceAnalysisPrefersGenAIExceptionEventOverErrorMessage(t *testing.T) { + // error.message is not an OpenTelemetry standard attribute, so the typed + // exception of the generative AI event has to win over it. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{spanAt(0, "01", "", "dubbo-admin-sre-agent", "chat gpt-4o", 50000)}, + } + trace.Spans[0].Error = true + trace.Spans[0].Status = "ERROR" + trace.Spans[0].Attributes = map[string]string{"error.message": "upstream timeout"} + trace.Spans[0].Events = []Event{{ + Timestamp: analysisBase, + Attributes: map[string]string{ + "event": "gen_ai.client.operation.exception", + "exception.type": "openai.APIConnectionError", + "exception.message": "connection reset by peer", + }, + }} + analysis := BuildTraceAnalysis(trace) + if analysis.ErrorSpans[0].ErrorMessage != "connection reset by peer" { + t.Fatalf("expected the generative AI exception message to win: %#v", analysis.ErrorSpans[0]) + } + if analysis.ErrorSpans[0].ErrorType != "openai.APIConnectionError" { + t.Fatalf("unexpected error type: %#v", analysis.ErrorSpans[0]) + } +} + +func TestBuildTraceAnalysisPrefersGenAIExceptionEventOverStandardExceptionEvent(t *testing.T) { + // Both events describe the same failure. The generative AI one wins because + // this project serves the generative AI execution of the agent loop. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{spanAt(0, "01", "", "dubbo-admin-sre-agent", "chat gpt-4o", 50000)}, + } + trace.Spans[0].Error = true + trace.Spans[0].Status = "ERROR" + trace.Spans[0].Events = []Event{ + { + Timestamp: analysisBase, + Attributes: map[string]string{ + "event": "exception", + "exception.type": "java.lang.IllegalStateException", + "exception.message": "state mismatch", + }, + }, + { + Timestamp: analysisBase, + Attributes: map[string]string{ + "event": "gen_ai.client.operation.exception", + "exception.type": "openai.RateLimitError", + "exception.message": "rate limit exceeded", + }, + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.ErrorSpans[0].ErrorType != "openai.RateLimitError" { + t.Fatalf("expected the generative AI exception type to win: %#v", analysis.ErrorSpans[0]) + } + if analysis.ErrorSpans[0].ErrorMessage != "rate limit exceeded" { + t.Fatalf("expected the generative AI exception message to win: %#v", analysis.ErrorSpans[0]) + } +} + +func TestBuildTraceAnalysisFallsBackToErrorMessageAttribute(t *testing.T) { + // error.message is the only signal left, so it provides the message and + // degrades into the error type as well. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{spanAt(0, "01", "", "payment-service", "charge", 50000)}, + } + trace.Spans[0].Error = true + trace.Spans[0].Status = "ERROR" + trace.Spans[0].Attributes = map[string]string{"error.message": "payment gateway timeout"} + analysis := BuildTraceAnalysis(trace) + if analysis.ErrorSpans[0].ErrorMessage != "payment gateway timeout" { + t.Fatalf("unexpected error message: %#v", analysis.ErrorSpans[0]) + } + if analysis.ErrorSpans[0].ErrorType != "payment gateway timeout" { + t.Fatalf("expected the message to degrade into the error type: %#v", analysis.ErrorSpans[0]) + } +} + +func TestBuildTraceAnalysisCountsErrorsBeyondListLimit(t *testing.T) { + spans := make([]Span, 0, maxErrorSpans+10) + for i := 0; i < maxErrorSpans+10; i++ { + span := spanAt(int64(i)*1000, fmt.Sprintf("%02d", i), "", "payment-service", "charge", 1000) + span.Error = true + span.Status = "ERROR" + spans = append(spans, span) + } + analysis := BuildTraceAnalysis(&Trace{TraceID: testTraceID, Spans: spans}) + if len(analysis.ErrorSpans) != maxErrorSpans { + t.Fatalf("expected the list to be capped, got %d", len(analysis.ErrorSpans)) + } + if analysis.ErrorSpanCount != maxErrorSpans+10 { + t.Fatalf("expected the real error count, got %d", analysis.ErrorSpanCount) + } + if !analysis.Truncated { + t.Fatal("expected the analysis to be marked as truncated") + } +} + +func TestBuildTraceAnalysisHandlesOrphanSpans(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + spanAt(10000, "02", "missing-parent", "order-service", "orphan", 40000), + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.SpanCount != 2 || analysis.ServiceCount != 2 { + t.Fatalf("unexpected counters: %#v", analysis) + } + // The orphan still contributes to the service breakdown. + found := false + for _, entry := range analysis.ServiceBreakdown { + if entry.ServiceName == "order-service" && entry.SpanCount == 1 { + found = true + } + } + if !found { + t.Fatalf("expected the orphan in the breakdown: %#v", analysis.ServiceBreakdown) + } +} + +func TestBuildTraceAnalysisSurvivesCyclicReferences(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "02", "frontend", "a", 100000), + spanAt(1000, "02", "01", "order-service", "b", 50000), + }, + } + analysis := BuildTraceAnalysis(trace) + if len(analysis.CriticalPath) > 2 { + t.Fatalf("expected the cycle to be broken, got %#v", analysis.CriticalPath) + } + if analysis.SpanCount != 2 { + t.Fatalf("unexpected span count: %d", analysis.SpanCount) + } +} + +func TestBuildTraceAnalysisSurvivesMalformedStartTime(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + {SpanID: "02", ParentSpanID: "01", ServiceName: "order-service", OperationName: "broken", StartTime: "not-a-timestamp", DurationMicros: 5000}, + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.SpanCount != 2 { + t.Fatalf("unexpected span count: %d", analysis.SpanCount) + } + if !analysis.Truncated { + t.Fatal("expected the analysis to be marked as truncated") + } +} + +func TestBuildTraceAnalysisSurvivesSelfReference(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "01", "frontend", "self", 100000), + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.SpanCount != 1 { + t.Fatalf("unexpected span count: %d", analysis.SpanCount) + } +} + +func TestBuildTraceAnalysisSummary(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "order-service", "checkout", 250000), + spanAt(10000, "02", "01", "payment-service", "charge", 200000), + }, + } + trace.Spans[1].Error = true + trace.Spans[1].Status = "ERROR" + trace.Spans[1].Attributes = map[string]string{"error.type": "timeout"} + analysis := BuildTraceAnalysis(trace) + if !strings.HasPrefix(analysis.Summary, "trace 2 services, 2 spans, 250ms, 1 error") { + t.Fatalf("unexpected summary: %q", analysis.Summary) + } + // The charge span consumes 200ms on its own, the checkout span only 50ms. + if !strings.Contains(analysis.Summary, "slowest: payment-service/charge self 200ms of 200ms (80% of trace)") { + t.Fatalf("unexpected summary: %q", analysis.Summary) + } + if !strings.Contains(analysis.Summary, "root cause: payment-service/charge ERROR timeout") { + t.Fatalf("unexpected summary: %q", analysis.Summary) + } + if utf8.RuneCountInString(analysis.Summary) > maxAnalysisSummaryChars { + t.Fatalf("expected a bounded summary, got %d characters", utf8.RuneCountInString(analysis.Summary)) + } + if strings.ContainsAny(analysis.Summary, "\r\n") { + t.Fatalf("expected a single line summary, got %q", analysis.Summary) + } +} + +func TestBuildTraceAnalysisSummaryStaysBounded(t *testing.T) { + spans := []Span{ + spanAt(0, "01", "", strings.Repeat("very-long-service-name-", 8), strings.Repeat("very-long-operation-name-", 8), 1000), + } + spans[0].Error = true + spans[0].Status = "ERROR" + analysis := BuildTraceAnalysis(&Trace{TraceID: testTraceID, Spans: spans}) + if utf8.RuneCountInString(analysis.Summary) > maxAnalysisSummaryChars { + t.Fatalf("expected a bounded summary, got %d characters", utf8.RuneCountInString(analysis.Summary)) + } +} + +func TestBuildTraceAnalysisHandlesEmptyTrace(t *testing.T) { + analysis := BuildTraceAnalysis(&Trace{TraceID: testTraceID}) + if analysis.SpanCount != 0 || analysis.ServiceCount != 0 { + t.Fatalf("unexpected counters: %#v", analysis) + } + if analysis.Summary == "" { + t.Fatal("expected a summary even for an empty trace") + } +} + +func TestAnalyzePropertiesExposeRequiredParameters(t *testing.T) { + properties := AnalyzeProperties() + for _, name := range []string{"traceId", "provider"} { + if _, ok := properties[name]; !ok { + t.Fatalf("analyze properties are missing %q", name) + } + } +} + +// criticalPathChain builds a linear chain of spans for the critical path budget. +func criticalPathChain(hops int) []*analysisSpan { + spans := make([]Span, 0, hops) + for i := 0; i < hops; i++ { + parentID := "" + if i > 0 { + parentID = fmt.Sprintf("%02d", i-1) + } + spans = append(spans, spanAt(int64(i)*1000, fmt.Sprintf("%02d", i), parentID, "order-service", "hop", 10000)) + } + return buildAnalysisTree(spans) +} + +func TestBuildCriticalPathStopsAtBudgetWithoutTruncation(t *testing.T) { + // A chain that ends exactly at the budget does not lose any hop, so the + // analysis must not be reported as truncated. + analysis := &TraceAnalysis{} + entries, _ := buildCriticalPath(criticalPathChain(maxCriticalPath), analysis) + if len(entries) != maxCriticalPath { + t.Fatalf("unexpected critical path length: %d", len(entries)) + } + if analysis.Truncated { + t.Fatal("did not expect the analysis to be truncated when the chain ends at the budget") + } + + // One hop more than the budget does lose a hop. + analysis = &TraceAnalysis{} + entries, _ = buildCriticalPath(criticalPathChain(maxCriticalPath+1), analysis) + if len(entries) != maxCriticalPath { + t.Fatalf("unexpected critical path length: %d", len(entries)) + } + if !analysis.Truncated { + t.Fatal("expected the analysis to be truncated when the chain exceeds the budget") + } +} + +func TestBuildTraceAnalysisUsesProviderEnvelopeForClippedSpans(t *testing.T) { + // The provider computes the duration over every raw span and only then clips + // the span list. The analysis has to use the provider envelope as the + // latency denominator, otherwise the reported duration is shorter than the + // one the search reported and every latency percentage is inflated. + trace := &Trace{ + TraceID: testTraceID, + DurationMicros: 500000, + Truncated: true, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "GET /checkout", 100000), + spanAt(10000, "02", "01", "order-service", "checkout", 80000), + }, + } + analysis := BuildTraceAnalysis(trace) + if analysis.DurationMicros != 500000 { + t.Fatalf("duration = %d, want %d", analysis.DurationMicros, 500000) + } + // The service breakdown divides by the envelope, so the frontend service + // covers a fifth of the whole trace and not of the retained spans only. + var frontend *ServiceLatency + for _, entry := range analysis.ServiceBreakdown { + if entry.ServiceName == "frontend" { + frontend = &entry + break + } + } + if frontend == nil { + t.Fatalf("expected a frontend entry in the breakdown: %#v", analysis.ServiceBreakdown) + } + if frontend.SharePercent != 20 { + t.Fatalf("frontend share = %v, want 20", frontend.SharePercent) + } +} + +func TestBuildTraceAnalysisDoesNotReportTruncationForTheSlowSpanRanking(t *testing.T) { + // The slow span list is a ranking: cutting it to the best entries does not + // lose any analysis data. The truncated flag would tell the agent that the + // diagnosis is based on incomplete data, so it must stay unset. + spans := make([]Span, 0, maxSlowSpans+5) + for i := 0; i < maxSlowSpans+5; i++ { + spans = append(spans, spanAt(int64(i)*1000, fmt.Sprintf("%02d", i), "", "frontend", "GET /checkout", 10000)) + } + trace := &Trace{TraceID: testTraceID, Spans: spans} + analysis := BuildTraceAnalysis(trace) + if len(analysis.SlowSpans) != maxSlowSpans { + t.Fatalf("slow spans = %d, want %d", len(analysis.SlowSpans), maxSlowSpans) + } + if analysis.Truncated { + t.Fatal("did not expect the analysis to be truncated by the slow span ranking") + } +} + +func TestBuildTraceAnalysisHandlesSpansWithoutSpanID(t *testing.T) { + // Both spans share an empty span id. An index keyed by the business id would + // let one span overwrite the other and mislabel the critical path. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + {SpanID: "", ParentSpanID: "", ServiceName: "frontend", OperationName: "root", StartTime: analysisBase, DurationMicros: 100000}, + {SpanID: "", ParentSpanID: "", ServiceName: "order-service", OperationName: "orphan", StartTime: shiftTime(analysisBase, 10000), DurationMicros: 50000}, + }, + } + analysis := BuildTraceAnalysis(trace) + // The critical path starts at the root that finishes last. + if len(analysis.CriticalPath) != 1 || analysis.CriticalPath[0].ServiceName != "frontend" { + t.Fatalf("unexpected critical path: %#v", analysis.CriticalPath) + } + // Only the span on the critical path may be marked as such. + onPath := 0 + for _, slow := range analysis.SlowSpans { + if !slow.OnCriticalPath { + continue + } + onPath++ + if slow.ServiceName != "frontend" { + t.Fatalf("expected only the root to be on the critical path: %#v", analysis.SlowSpans) + } + } + if onPath != 1 { + t.Fatalf("expected exactly one span on the critical path: %#v", analysis.SlowSpans) + } + // The summary must quote the self time of the slowest span and not the one of + // another span that happens to share the same identifier. + if !strings.Contains(analysis.Summary, "slowest: frontend/root self 100ms of 100ms") { + t.Fatalf("unexpected summary: %q", analysis.Summary) + } +} + +func TestBuildTraceAnalysisSkipsSelfTimeForUnparsableStartTime(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + {SpanID: "02", ParentSpanID: "01", ServiceName: "order-service", OperationName: "broken", StartTime: "not-a-timestamp", DurationMicros: 50000}, + }, + } + analysis := BuildTraceAnalysis(trace) + if !analysis.Truncated { + t.Fatal("expected the analysis to be marked as truncated") + } + // The span without a timestamp must not claim a time of its own, otherwise + // the parent would account for the same 50ms a second time. + for _, slow := range analysis.SlowSpans { + if slow.SpanID == "02" && slow.SharePercent != 0 { + t.Fatalf("expected no self time for the span without a timestamp: %#v", slow) + } + } + if analysis.CriticalPath[0].SelfMicros != 100000 { + t.Fatalf("unexpected self time of the root: %#v", analysis.CriticalPath[0]) + } +} + +func TestBuildTraceAnalysisReadsExceptionSpanAttributes(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{spanAt(0, "01", "", "payment-service", "charge", 50000)}, + } + trace.Spans[0].Error = true + trace.Spans[0].Status = "ERROR" + trace.Spans[0].Attributes = map[string]string{ + "exception.type": "java.lang.IllegalStateException", + "exception.message": "state mismatch", + } + analysis := BuildTraceAnalysis(trace) + if analysis.ErrorSpans[0].ErrorType != "java.lang.IllegalStateException" { + t.Fatalf("unexpected error type: %#v", analysis.ErrorSpans[0]) + } + if analysis.ErrorSpans[0].ErrorMessage != "state mismatch" { + t.Fatalf("unexpected error message: %#v", analysis.ErrorSpans[0]) + } +} + +func TestBuildTraceAnalysisPrefersExceptionEventOverSpanAttributes(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{spanAt(0, "01", "", "payment-service", "charge", 50000)}, + } + trace.Spans[0].Error = true + trace.Spans[0].Status = "ERROR" + trace.Spans[0].Attributes = map[string]string{ + "exception.type": "java.lang.IllegalStateException", + "exception.message": "state mismatch", + } + trace.Spans[0].Events = []Event{{ + Timestamp: analysisBase, + Attributes: map[string]string{ + "event": "exception", + "exception.type": "java.net.SocketTimeoutException", + "exception.message": "Read timed out", + }, + }} + analysis := BuildTraceAnalysis(trace) + if analysis.ErrorSpans[0].ErrorType != "java.net.SocketTimeoutException" { + t.Fatalf("expected the event to win over the span attribute: %#v", analysis.ErrorSpans[0]) + } + if analysis.ErrorSpans[0].ErrorMessage != "Read timed out" { + t.Fatalf("expected the event to win over the span attribute: %#v", analysis.ErrorSpans[0]) + } +} + +func TestBuildTraceAnalysisSummaryCountsRunes(t *testing.T) { + // A Chinese name uses three bytes per character, so a byte budget would drop + // the diagnosis even though it still fits into the character budget. + longName := strings.Repeat("服", 60) + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", longName, longName, 100000), + spanAt(10000, "02", "01", "order-service", "charge", 50000), + }, + } + analysis := BuildTraceAnalysis(trace) + if !strings.Contains(analysis.Summary, "slowest:") { + t.Fatalf("expected the slowest span in the summary: %q", analysis.Summary) + } + if utf8.RuneCountInString(analysis.Summary) > maxAnalysisSummaryChars { + t.Fatalf("expected a bounded summary, got %d characters", utf8.RuneCountInString(analysis.Summary)) + } +} + +func TestBuildTraceAnalysisSummaryUsesSingularForms(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{spanAt(0, "01", "", "frontend", "root", 100000)}, + } + analysis := BuildTraceAnalysis(trace) + if !strings.HasPrefix(analysis.Summary, "trace 1 service, 1 span, 100ms, 0 errors") { + t.Fatalf("unexpected summary: %q", analysis.Summary) + } +} + +func TestBuildTraceAnalysisServiceShareCanExceedHundred(t *testing.T) { + // Two concurrent children each cover the whole window of the parent, so the + // service workload is twice the wall clock of the trace. The share is a + // workload ratio and not a time slice, so it may exceed 100 percent. + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "frontend", "root", 100000), + spanAt(0, "02", "01", "order-service", "a", 100000), + spanAt(0, "03", "01", "order-service", "b", 100000), + }, + } + analysis := BuildTraceAnalysis(trace) + share := 0.0 + for _, entry := range analysis.ServiceBreakdown { + if entry.ServiceName == "order-service" { + share = entry.SharePercent + } + } + if share <= 100 { + t.Fatalf("expected the concurrent workload to exceed 100 percent, got %v", share) + } +} diff --git a/pkg/mcp/tools/trace/genai.go b/pkg/mcp/tools/trace/genai.go new file mode 100644 index 000000000..742593635 --- /dev/null +++ b/pkg/mcp/tools/trace/genai.go @@ -0,0 +1,423 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package trace + +import ( + "math" + "sort" + "strconv" + "strings" + "time" +) + +// The generative AI semantic conventions are still in development, so the analyzer pins +// the recognized attribute set instead of trusting an open-ended namespace. +// Reference: OpenTelemetry semantic conventions for generative AI systems, commit 6fd0d76. +const genAIPrefix = "gen_ai." + +// Recognized generative AI attributes. +const ( + genAIOperationName = "gen_ai.operation.name" + genAIProviderName = "gen_ai.provider.name" + genAIRequestModel = "gen_ai.request.model" + genAIResponseModel = "gen_ai.response.model" + genAIConversationID = "gen_ai.conversation.id" + genAIAgentName = "gen_ai.agent.name" + genAIAgentID = "gen_ai.agent.id" + genAIAgentDescription = "gen_ai.agent.description" + genAIAgentVersion = "gen_ai.agent.version" + genAIToolName = "gen_ai.tool.name" + genAIToolCallID = "gen_ai.tool.call.id" + genAIToolCallArguments = "gen_ai.tool.call.arguments" + genAIToolCallResult = "gen_ai.tool.call.result" + genAIInputMessages = "gen_ai.input.messages" + genAIOutputMessages = "gen_ai.output.messages" + genAIUsageInputTokens = "gen_ai.usage.input_tokens" + genAIUsageOutputTokens = "gen_ai.usage.output_tokens" + genAIUsageReasoningTokens = "gen_ai.usage.reasoning.output_tokens" + genAIUsageCacheReadTokens = "gen_ai.usage.cache_read.input_tokens" + genAIUsageCacheWriteTokens = "gen_ai.usage.cache_write.input_tokens" + genAIResponseFinishReasons = "gen_ai.response.finish_reasons" + genAIResponseID = "gen_ai.response.id" + genAIDataSourceID = "gen_ai.data_source.id" + genAIOutputType = "gen_ai.output.type" +) + +// genAIUsageTotalTokens is not part of the semantic conventions, the conventions +// only define the input and the output token counters. It is honored when a +// provider emits it anyway, because some vendors already report the aggregate. +// The value stays a reference value in the detail of a single call, the aggregate +// of the trace is always the sum of the input and the output tokens. +const genAIUsageTotalTokens = "gen_ai.usage.total_tokens" + +// genAIContentAttributes hold prompts, completions and tool payloads. They are Opt-In +// in the semantic conventions and may embed PII, so their values are never read. +var genAIContentAttributes = map[string]struct{}{ + genAIInputMessages: {}, + genAIOutputMessages: {}, + genAIToolCallArguments: {}, + genAIToolCallResult: {}, +} + +// genAIRecognizedAttributes is the pinned set of attributes the analyzer is allowed to +// read. An unknown gen_ai.* attribute still marks a span as generative AI, but its +// value is never interpreted. +var genAIRecognizedAttributes = map[string]struct{}{ + genAIOperationName: {}, + genAIProviderName: {}, + genAIRequestModel: {}, + genAIResponseModel: {}, + genAIConversationID: {}, + genAIAgentName: {}, + genAIAgentID: {}, + genAIAgentDescription: {}, + genAIAgentVersion: {}, + genAIToolName: {}, + genAIToolCallID: {}, + genAIToolCallArguments: {}, + genAIToolCallResult: {}, + genAIInputMessages: {}, + genAIOutputMessages: {}, + genAIUsageInputTokens: {}, + genAIUsageOutputTokens: {}, + genAIUsageReasoningTokens: {}, + genAIUsageCacheReadTokens: {}, + genAIUsageCacheWriteTokens: {}, + genAIResponseFinishReasons: {}, + genAIResponseID: {}, + genAIDataSourceID: {}, + genAIOutputType: {}, + genAIUsageTotalTokens: {}, +} + +// genAITokenAttributes lists every token attribute the analyzer recognizes. The cache +// and reasoning counters are breakdowns of the input/output counters, so they are never +// added on top of them. +var genAITokenAttributes = []string{ + genAIUsageInputTokens, + genAIUsageOutputTokens, + genAIUsageReasoningTokens, + genAIUsageCacheReadTokens, + genAIUsageCacheWriteTokens, + genAIUsageTotalTokens, +} + +// genAIModelSignalAttributes mark a span as a model invocation even when the operation +// name is missing or carries a custom value. +var genAIModelSignalAttributes = []string{ + genAIRequestModel, + genAIResponseModel, + genAIResponseID, + genAIResponseFinishReasons, + genAIOutputType, + genAIDataSourceID, +} + +var ( + genAIModelOperations = map[string]struct{}{ + "chat": {}, + "generate_content": {}, + "text_completion": {}, + "embeddings": {}, + "fetch_response": {}, + } + genAIToolOperations = map[string]struct{}{ + "execute_tool": {}, + } + genAIAgentOperations = map[string]struct{}{ + "invoke_agent": {}, + "create_agent": {}, + "invoke_workflow": {}, + "plan": {}, + } +) + +// genAIClass is the mutually exclusive category a generative AI span falls into. +type genAIClass int + +const ( + genAIClassNone genAIClass = iota + genAIClassModel + genAIClassTool + genAIClassAgent +) + +// genAIOrderedSpan pairs a span with its resolved start time so the per-category +// detail lists can be emitted in chronological order. +type genAIOrderedSpan struct { + span *Span + startMicros int64 +} + +// HasGenAISpans reports whether any span carries generative AI attributes. +func HasGenAISpans(spans []Span) bool { + for i := range spans { + if HasGenAIAttributes(spans[i].Attributes) { + return true + } + } + return false +} + +// HasGenAIAttributes reports whether an attribute map contains generative AI attributes. +// The search path needs this check on raw provider tags, so it must not require a +// normalized span. +func HasGenAIAttributes(tags map[string]string) bool { + for key := range tags { + if isGenAIAttributeKey(key) { + return true + } + } + return false +} + +// BuildGenAISummary aggregates the generative AI footprint of the given spans. +// It never fails: spans without generative AI attributes yield Present=false. +func BuildGenAISummary(spans []Span) *GenAISummary { + summary := &GenAISummary{Present: false} + ordered := make([]genAIOrderedSpan, 0, len(spans)) + conversationIDs := map[string]struct{}{} + providers := map[string]struct{}{} + models := map[string]struct{}{} + for i := range spans { + span := &spans[i] + if !HasGenAIAttributes(span.Attributes) { + continue + } + summary.Present = true + ordered = append(ordered, genAIOrderedSpan{span: span, startMicros: spanStartMicros(span)}) + addString(conversationIDs, genAIAttribute(span.Attributes, genAIConversationID)) + addString(providers, genAIAttribute(span.Attributes, genAIProviderName)) + addString(models, genAIAttribute(span.Attributes, genAIRequestModel)) + addString(models, genAIAttribute(span.Attributes, genAIResponseModel)) + summary.InputTokens += genAIIntAttribute(span.Attributes, genAIUsageInputTokens) + summary.OutputTokens += genAIIntAttribute(span.Attributes, genAIUsageOutputTokens) + } + // The semantic conventions define the input and the output token counters + // only, there is no standard total token attribute. The total is therefore a + // derived value with a single definition: it always equals input plus output. + // A provider that reports the non standard gen_ai.usage.total_tokens only + // shows up as a reference value in the detail of a single call, it never + // takes part in the aggregate, so a consumer never has to choose between + // two definitions of the total. + summary.TotalTokens = summary.InputTokens + summary.OutputTokens + if !summary.Present { + return summary + } + sort.SliceStable(ordered, func(i, j int) bool { + if ordered[i].startMicros != ordered[j].startMicros { + return ordered[i].startMicros < ordered[j].startMicros + } + return ordered[i].span.SpanID < ordered[j].span.SpanID + }) + summary.ConversationIDs = sortedKeys(conversationIDs) + summary.Providers = sortedKeys(providers) + summary.Models = sortedKeys(models) + + for _, item := range ordered { + span := item.span + switch classifyGenAISpan(span) { + case genAIClassModel: + inputTokens := genAIIntAttribute(span.Attributes, genAIUsageInputTokens) + outputTokens := genAIIntAttribute(span.Attributes, genAIUsageOutputTokens) + totalTokens := genAISpanTotalTokens(span.Attributes) + summary.ModelCallCount++ + if len(summary.ModelCalls) < maxGenAICalls { + summary.ModelCalls = append(summary.ModelCalls, GenAIModelCall{ + SpanID: span.SpanID, + ServiceName: span.ServiceName, + OperationName: span.OperationName, + Operation: genAIAttribute(span.Attributes, genAIOperationName), + ProviderName: genAIAttribute(span.Attributes, genAIProviderName), + RequestModel: genAIAttribute(span.Attributes, genAIRequestModel), + ResponseModel: genAIAttribute(span.Attributes, genAIResponseModel), + InputTokens: inputTokens, + OutputTokens: outputTokens, + TotalTokens: totalTokens, + DurationMicros: span.DurationMicros, + Error: spanFailed(span), + }) + } else { + summary.Truncated = true + } + case genAIClassTool: + summary.ToolCallCount++ + if len(summary.ToolCalls) < maxGenAICalls { + summary.ToolCalls = append(summary.ToolCalls, GenAIToolCall{ + SpanID: span.SpanID, + ServiceName: span.ServiceName, + ToolName: genAIAttribute(span.Attributes, genAIToolName), + ToolCallID: genAIAttribute(span.Attributes, genAIToolCallID), + DurationMicros: span.DurationMicros, + Error: spanFailed(span), + }) + } else { + summary.Truncated = true + } + case genAIClassAgent: + summary.AgentCount++ + if len(summary.Agents) < maxGenAICalls { + summary.Agents = append(summary.Agents, GenAIAgent{ + SpanID: span.SpanID, + ServiceName: span.ServiceName, + AgentName: genAIAttribute(span.Attributes, genAIAgentName), + AgentID: genAIAttribute(span.Attributes, genAIAgentID), + DurationMicros: span.DurationMicros, + Error: spanFailed(span), + }) + } else { + summary.Truncated = true + } + } + } + return summary +} + +// classifyGenAISpan assigns exactly one category to a generative AI span. +// The precedence is model operation, then tool, then agent, and finally a model call +// inferred from model or token attributes. +func classifyGenAISpan(span *Span) genAIClass { + operation := strings.ToLower(strings.TrimSpace(genAIAttribute(span.Attributes, genAIOperationName))) + if _, ok := genAIModelOperations[operation]; ok { + return genAIClassModel + } + if _, ok := genAIToolOperations[operation]; ok { + return genAIClassTool + } + if _, ok := genAIAgentOperations[operation]; ok { + return genAIClassAgent + } + if _, ok := genAIAttributeOK(span.Attributes, genAIToolName); ok { + return genAIClassTool + } + if _, ok := genAIAttributeOK(span.Attributes, genAIAgentName); ok { + return genAIClassAgent + } + for _, name := range genAIModelSignalAttributes { + if _, ok := genAIAttributeOK(span.Attributes, name); ok { + return genAIClassModel + } + } + for _, name := range genAITokenAttributes { + if _, ok := genAIAttributeOK(span.Attributes, name); ok { + return genAIClassModel + } + } + return genAIClassNone +} + +// genAISpanTotalTokens returns the token total of a single span. A provider that +// reports the aggregate of the span wins, otherwise the total is derived from the +// input and the output counters of that span. The value is only meaningful per +// span: it is a reference value in the detail of a single call and never takes +// part in the aggregate of the trace, whose total is always input plus output. +func genAISpanTotalTokens(attributes map[string]string) int { + if override, ok := genAIAttributeOK(attributes, genAIUsageTotalTokens); ok { + if parsed, err := strconv.Atoi(strings.TrimSpace(override)); err == nil { + return parsed + } + } + return genAIIntAttribute(attributes, genAIUsageInputTokens) + genAIIntAttribute(attributes, genAIUsageOutputTokens) +} + +func isGenAIAttributeKey(key string) bool { + return strings.HasPrefix(strings.ToLower(strings.TrimSpace(key)), genAIPrefix) +} + +// genAIAttribute reads a recognized, non-content generative AI attribute. +func genAIAttribute(attributes map[string]string, name string) string { + value, _ := genAIAttributeOK(attributes, name) + return value +} + +// genAIAttributeOK reads a recognized, non-content generative AI attribute. +func genAIAttributeOK(attributes map[string]string, name string) (string, bool) { + if _, content := genAIContentAttributes[name]; content { + return "", false + } + if _, recognized := genAIRecognizedAttributes[name]; !recognized { + return "", false + } + return lookupAttributeOK(attributes, name) +} + +func genAIIntAttribute(attributes map[string]string, name string) int { + raw, ok := genAIAttributeOK(attributes, name) + if !ok { + return 0 + } + return parseIntegralAttribute(strings.TrimSpace(raw)) +} + +// parseIntegralAttribute reads an integer attribute value. The attribute map is +// filled from the provider payload, where a number arrives as a JSON number. +// The decoder turns it into a float64 and the formatter renders it with the +// 'g' format, so a token count of 1000000 becomes "1e+06". A plain integer +// parse drops such a value silently, so the integral scientific notation is +// accepted as well. +func parseIntegralAttribute(raw string) int { + if raw == "" { + return 0 + } + if parsed, err := strconv.Atoi(raw); err == nil { + return parsed + } + value, err := strconv.ParseFloat(raw, 64) + if err != nil { + return 0 + } + if value != math.Trunc(value) || value > math.MaxInt64 || value < math.MinInt64 { + return 0 + } + return int(value) +} + +func addString(target map[string]struct{}, value string) { + trimmed := strings.TrimSpace(value) + if trimmed == "" { + return + } + target[trimmed] = struct{}{} +} + +func sortedKeys(values map[string]struct{}) []string { + if len(values) == 0 { + return nil + } + result := make([]string, 0, len(values)) + for key := range values { + result = append(result, key) + } + sort.Strings(result) + return result +} + +// spanStartMicros converts a span start timestamp into unix microseconds. +// Unparsable timestamps degrade to zero so the analyzer never panics. +func spanStartMicros(span *Span) int64 { + parsed, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(span.StartTime)) + if err != nil { + return 0 + } + return parsed.UnixMicro() +} + +// spanFailed reports whether a span is in an error state. +func spanFailed(span *Span) bool { + return span.Error || strings.EqualFold(span.Status, "ERROR") +} diff --git a/pkg/mcp/tools/trace/genai_test.go b/pkg/mcp/tools/trace/genai_test.go new file mode 100644 index 000000000..fbea7e3c2 --- /dev/null +++ b/pkg/mcp/tools/trace/genai_test.go @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package trace + +import ( + "encoding/json" + "strings" + "testing" +) + +func genAISpan(spanID string, attributes map[string]string) Span { + return Span{ + SpanID: spanID, + ServiceName: "dubbo-admin-sre-agent", + OperationName: spanID, + StartTime: analysisBase, + Attributes: attributes, + } +} + +func TestHasGenAIAttributes(t *testing.T) { + if HasGenAIAttributes(map[string]string{"rpc.system": "apache_dubbo"}) { + t.Fatal("did not expect a non generative AI attribute to match") + } + if !HasGenAIAttributes(map[string]string{"gen_ai.operation.name": "chat"}) { + t.Fatal("expected a generative AI attribute to match") + } + if !HasGenAIAttributes(map[string]string{" GEN_AI.Operation.Name ": "chat"}) { + t.Fatal("expected the detection to tolerate casing and padding") + } +} + +func TestHasGenAISpans(t *testing.T) { + if HasGenAISpans([]Span{{SpanID: "01"}}) { + t.Fatal("did not expect a plain span to match") + } + if !HasGenAISpans([]Span{{SpanID: "01"}, {SpanID: "02", Attributes: map[string]string{"gen_ai.agent.name": "sre"}}}) { + t.Fatal("expected a generative AI span to match") + } +} + +func TestBuildGenAISummaryClassifiesModelCall(t *testing.T) { + summary := BuildGenAISummary([]Span{ + genAISpan("01", map[string]string{ + "gen_ai.operation.name": "chat", + "gen_ai.provider.name": "openai", + "gen_ai.request.model": "gpt-4o", + "gen_ai.response.model": "gpt-4o-2024-08-06", + "gen_ai.conversation.id": "conv-1", + "gen_ai.usage.input_tokens": "120", + "gen_ai.usage.output_tokens": "30", + }), + }) + if !summary.Present { + t.Fatal("expected the summary to be present") + } + if summary.ModelCallCount != 1 || summary.ToolCallCount != 0 || summary.AgentCount != 0 { + t.Fatalf("unexpected call counts: %#v", summary) + } + if len(summary.ModelCalls) != 1 { + t.Fatalf("unexpected model calls: %#v", summary.ModelCalls) + } + call := summary.ModelCalls[0] + if call.Operation != "chat" || call.ProviderName != "openai" || call.RequestModel != "gpt-4o" || call.ResponseModel != "gpt-4o-2024-08-06" { + t.Fatalf("unexpected model call: %#v", call) + } + if call.InputTokens != 120 || call.OutputTokens != 30 || call.TotalTokens != 150 { + t.Fatalf("unexpected token usage: %#v", call) + } + if summary.InputTokens != 120 || summary.OutputTokens != 30 || summary.TotalTokens != 150 { + t.Fatalf("unexpected token totals: %#v", summary) + } + if len(summary.Providers) != 1 || summary.Providers[0] != "openai" { + t.Fatalf("unexpected providers: %#v", summary.Providers) + } + if len(summary.Models) != 2 || summary.Models[0] != "gpt-4o" { + t.Fatalf("unexpected models: %#v", summary.Models) + } + if len(summary.ConversationIDs) != 1 || summary.ConversationIDs[0] != "conv-1" { + t.Fatalf("unexpected conversations: %#v", summary.ConversationIDs) + } +} + +func TestBuildGenAISummaryClassifiesToolAndAgent(t *testing.T) { + summary := BuildGenAISummary([]Span{ + genAISpan("01", map[string]string{ + "gen_ai.operation.name": "execute_tool", + "gen_ai.tool.name": "search_logs", + "gen_ai.tool.call.id": "call-1", + }), + genAISpan("02", map[string]string{ + "gen_ai.operation.name": "invoke_agent", + "gen_ai.agent.name": "dubbo-admin-sre-agent", + "gen_ai.agent.id": "agent-42", + }), + }) + if summary.ToolCallCount != 1 || summary.AgentCount != 1 || summary.ModelCallCount != 0 { + t.Fatalf("unexpected call counts: %#v", summary) + } + if len(summary.ToolCalls) != 1 || summary.ToolCalls[0].ToolName != "search_logs" || summary.ToolCalls[0].ToolCallID != "call-1" { + t.Fatalf("unexpected tool calls: %#v", summary.ToolCalls) + } + if len(summary.Agents) != 1 || summary.Agents[0].AgentName != "dubbo-admin-sre-agent" || summary.Agents[0].AgentID != "agent-42" { + t.Fatalf("unexpected agents: %#v", summary.Agents) + } +} + +func TestBuildGenAISummaryClassifiesByAttributesWithoutOperationName(t *testing.T) { + summary := BuildGenAISummary([]Span{ + genAISpan("01", map[string]string{"gen_ai.tool.name": "search_logs"}), + genAISpan("02", map[string]string{"gen_ai.agent.name": "sre"}), + genAISpan("03", map[string]string{"gen_ai.request.model": "gpt-4o"}), + genAISpan("04", map[string]string{"gen_ai.usage.input_tokens": "10"}), + }) + if summary.ToolCallCount != 1 || summary.AgentCount != 1 || summary.ModelCallCount != 2 { + t.Fatalf("unexpected call counts: %#v", summary) + } +} + +func TestBuildGenAISummaryKeepsProviderTokenTotalAsCallDetailOnly(t *testing.T) { + // The semantic conventions define the input and the output token counters + // only. The total of the summary is therefore always their sum, the value + // reported by the provider stays a reference in the detail of the call. + summary := BuildGenAISummary([]Span{ + genAISpan("01", map[string]string{ + "gen_ai.operation.name": "chat", + "gen_ai.usage.input_tokens": "10", + "gen_ai.usage.output_tokens": "5", + "gen_ai.usage.total_tokens": "99", + }), + }) + if summary.ModelCalls[0].TotalTokens != 99 { + t.Fatalf("expected the provider total to be kept in the call detail: %#v", summary.ModelCalls[0]) + } + if summary.TotalTokens != summary.InputTokens+summary.OutputTokens { + t.Fatalf("expected the total to equal the input plus the output tokens: %#v", summary) + } + if summary.TotalTokens != 15 { + t.Fatalf("expected the total to be derived from the input and the output tokens: %#v", summary) + } +} + +func TestBuildGenAISummaryWithoutGenAIAttributes(t *testing.T) { + summary := BuildGenAISummary([]Span{{SpanID: "01", ServiceName: "order-service"}}) + if summary.Present { + t.Fatal("did not expect the summary to be present") + } + if summary.TotalTokens != 0 || summary.ModelCallCount != 0 { + t.Fatalf("unexpected summary: %#v", summary) + } +} + +func TestBuildGenAISummaryDerivesAggregateTotalFromInputAndOutput(t *testing.T) { + // Only one span reports its own total. The aggregate must not be influenced + // by it, the total is always the sum of the input and the output tokens. + summary := BuildGenAISummary([]Span{ + genAISpan("01", map[string]string{ + "gen_ai.operation.name": "chat", + "gen_ai.usage.input_tokens": "100", + "gen_ai.usage.output_tokens": "50", + }), + genAISpan("02", map[string]string{ + "gen_ai.operation.name": "chat", + "gen_ai.usage.input_tokens": "10", + "gen_ai.usage.output_tokens": "5", + "gen_ai.usage.total_tokens": "999", + }), + }) + if summary.InputTokens != 110 || summary.OutputTokens != 55 { + t.Fatalf("unexpected token counters: %#v", summary) + } + // The reported total of the second span does not take part in the aggregate. + if summary.TotalTokens != 110+55 { + t.Fatalf("expected the total to equal the input plus the output tokens, got %d", summary.TotalTokens) + } + if len(summary.ModelCalls) != 2 { + t.Fatalf("unexpected model calls: %#v", summary.ModelCalls) + } + totals := map[string]int{} + for _, call := range summary.ModelCalls { + totals[call.SpanID] = call.TotalTokens + } + // The detail of a single call keeps the value reported by the provider. + if totals["01"] != 150 || totals["02"] != 999 { + t.Fatalf("unexpected per call totals: %#v", totals) + } +} + +func TestBuildGenAISummaryDoesNotReadContentAttributes(t *testing.T) { + summary := BuildGenAISummary([]Span{ + genAISpan("01", map[string]string{ + "gen_ai.operation.name": "chat", + "gen_ai.request.model": "gpt-4o", + "gen_ai.input.messages": `{"messages":[{"role":"user","content":"secret prompt"}]}`, + "gen_ai.output.messages": `{"messages":[{"role":"assistant","content":"secret answer"}]}`, + "gen_ai.tool.call.arguments": `{"query":"secret argument"}`, + "gen_ai.tool.call.result": `{"rows":["secret result"]}`, + }), + }) + if summary.ModelCallCount != 1 { + t.Fatalf("unexpected call counts: %#v", summary) + } + for _, call := range summary.ModelCalls { + if call.RequestModel != "gpt-4o" { + t.Fatalf("unexpected model call: %#v", call) + } + } + // The content attributes must not reach the tool output at all. + encoded, err := json.Marshal(summary) + if err != nil { + t.Fatal(err) + } + for _, secret := range []string{"secret prompt", "secret answer", "secret argument", "secret result"} { + if strings.Contains(string(encoded), secret) { + t.Fatalf("expected the content attribute to be omitted, got %s", encoded) + } + } +} + +func TestBuildGenAISummaryTruncatesDetailLists(t *testing.T) { + spans := make([]Span, 0, maxGenAICalls+5) + for i := 0; i < maxGenAICalls+5; i++ { + spans = append(spans, genAISpan(string(rune('a'+i%26))+string(rune('a'+i/26)), map[string]string{ + "gen_ai.operation.name": "chat", + "gen_ai.request.model": "gpt-4o", + })) + } + summary := BuildGenAISummary(spans) + if summary.ModelCallCount != maxGenAICalls+5 { + t.Fatalf("expected the real call count, got %d", summary.ModelCallCount) + } + if len(summary.ModelCalls) != maxGenAICalls { + t.Fatalf("expected the detail list to be capped, got %d", len(summary.ModelCalls)) + } + if !summary.Truncated { + t.Fatal("expected the summary to be marked as truncated") + } +} + +func TestBuildGenAISummarySortsDetailsByStartTime(t *testing.T) { + late := genAISpan("02", map[string]string{"gen_ai.operation.name": "chat"}) + late.StartTime = shiftTime(analysisBase, 5000) + early := genAISpan("01", map[string]string{"gen_ai.operation.name": "chat"}) + early.StartTime = shiftTime(analysisBase, 1000) + summary := BuildGenAISummary([]Span{late, early}) + if len(summary.ModelCalls) != 2 { + t.Fatalf("unexpected model calls: %#v", summary.ModelCalls) + } + if summary.ModelCalls[0].SpanID != "01" || summary.ModelCalls[1].SpanID != "02" { + t.Fatalf("unexpected model call order: %#v", summary.ModelCalls) + } +} + +func TestGenAIOperationClassification(t *testing.T) { + tests := []struct { + operation string + expected genAIClass + }{ + {operation: "chat", expected: genAIClassModel}, + {operation: "generate_content", expected: genAIClassModel}, + {operation: "text_completion", expected: genAIClassModel}, + {operation: "embeddings", expected: genAIClassModel}, + {operation: "fetch_response", expected: genAIClassModel}, + {operation: "execute_tool", expected: genAIClassTool}, + {operation: "invoke_agent", expected: genAIClassAgent}, + {operation: "create_agent", expected: genAIClassAgent}, + {operation: "invoke_workflow", expected: genAIClassAgent}, + {operation: "plan", expected: genAIClassAgent}, + } + for _, test := range tests { + t.Run(test.operation, func(t *testing.T) { + span := genAISpan("01", map[string]string{"gen_ai.operation.name": test.operation}) + if actual := classifyGenAISpan(&span); actual != test.expected { + t.Fatalf("expected %v, got %v", test.expected, actual) + } + }) + } +} + +func TestGenAIAnalysisEmbedsSummaryInTraceAnalysis(t *testing.T) { + trace := &Trace{ + TraceID: testTraceID, + Spans: []Span{ + spanAt(0, "01", "", "dubbo-admin-sre-agent", "chat gpt-4o", 1000), + spanAt(100, "02", "01", "dubbo-admin-sre-agent", "execute_tool search_logs", 500), + }, + } + trace.Spans[0].Attributes = map[string]string{"gen_ai.operation.name": "chat", "gen_ai.usage.input_tokens": "5"} + trace.Spans[1].Attributes = map[string]string{"gen_ai.operation.name": "execute_tool", "gen_ai.tool.name": "search_logs"} + analysis := BuildTraceAnalysis(trace) + if analysis.GenAI == nil { + t.Fatal("expected a generative AI summary") + } + if !analysis.GenAI.Present || analysis.GenAI.ModelCallCount != 1 || analysis.GenAI.ToolCallCount != 1 { + t.Fatalf("unexpected generative AI summary: %#v", analysis.GenAI) + } +} + +func TestGenAIIntAttributeAcceptsIntegralScientificNotation(t *testing.T) { + // The attribute map is filled from a JSON payload, so a number arrives as a + // float64 and the formatter renders it with the 'g' format: 1000000 becomes + // "1e+06". A plain integer parse drops such a value silently, which would + // report a million token model call as a call without any token. + attributes := map[string]string{ + "gen_ai.usage.input_tokens": "1e+06", + "gen_ai.usage.output_tokens": "1500000", + "gen_ai.usage.total_tokens": "12.5", + } + if got := genAIIntAttribute(attributes, "gen_ai.usage.input_tokens"); got != 1000000 { + t.Fatalf("input tokens = %d, want %d", got, 1000000) + } + if got := genAIIntAttribute(attributes, "gen_ai.usage.output_tokens"); got != 1500000 { + t.Fatalf("output tokens = %d, want %d", got, 1500000) + } + // A fractional value is not a token count and must not be rounded. + if got := genAIIntAttribute(attributes, "gen_ai.usage.total_tokens"); got != 0 { + t.Fatalf("total tokens = %d, want %d", got, 0) + } +} diff --git a/pkg/mcp/tools/trace/jaeger.go b/pkg/mcp/tools/trace/jaeger.go index 354c5ebf2..23898059e 100644 --- a/pkg/mcp/tools/trace/jaeger.go +++ b/pkg/mcp/tools/trace/jaeger.go @@ -20,6 +20,7 @@ package trace import ( "context" "encoding/json" + "errors" "fmt" "io" "net/http" @@ -116,6 +117,20 @@ func newJaegerProvider(cfg observabilitycfg.TraceProviderConfig) (*jaegerProvide if err != nil { return nil, fmt.Errorf("invalid jaeger endpoint: %w", err) } + if baseURL.Scheme != "http" && baseURL.Scheme != "https" { + return nil, fmt.Errorf("invalid jaeger endpoint: scheme must be http or https") + } + if baseURL.Host == "" { + return nil, fmt.Errorf("invalid jaeger endpoint: host is missing") + } + // A query string on the endpoint would be carried into every request URL and + // into the transport error of a failed lookup, which reaches the tool result + // and the model context. Credentials belong in bearerToken, so the query is + // rejected instead of being forwarded. The error message does not repeat the + // endpoint, because the endpoint is exactly what may hold the secret. + if baseURL.RawQuery != "" || baseURL.Fragment != "" { + return nil, fmt.Errorf("invalid jaeger endpoint: query parameters are not supported, use bearerToken for credentials") + } return &jaegerProvider{ config: cfg, baseURL: baseURL, @@ -145,7 +160,12 @@ func (p *jaegerProvider) GetTraceByID(ctx context.Context, traceID string) (*Tra resp, err := p.client.Do(req) if err != nil { p.logFailure(traceID, startedAt, 0, "transport", err) - return nil, fmt.Errorf("jaeger trace query failed: %w", err) + // The transport error embeds the request URL, which repeats the configured + // endpoint and carries the retrieval conditions. It is sanitized before it + // reaches the tool result and the model context. The trace identifier is + // kept, because the model chose it and has to be able to report which + // lookup failed. + return nil, fmt.Errorf("jaeger trace query failed for trace %s: %w", traceID, sanitizedTransportError(err)) } defer resp.Body.Close() if resp.StatusCode == http.StatusNotFound { @@ -184,6 +204,237 @@ func (p *jaegerProvider) GetTraceByID(ctx context.Context, traceID string) (*Tra return result, nil } +// SearchTraces discovers candidate traces through the Jaeger v1 query API. +// The summaries are computed from the raw spans so that no information is lost to +// the normalization applied by the single trace lookup. +func (p *jaegerProvider) SearchTraces(ctx context.Context, query TraceSearchQuery) (*TraceSearchResult, error) { + if strings.TrimSpace(query.Service) == "" { + return nil, fmt.Errorf("service is required for trace search") + } + values := url.Values{} + values.Set("service", query.Service) + if query.Operation != "" { + values.Set("operation", query.Operation) + } + if query.TraceID != "" { + values.Set("traceID", query.TraceID) + } + if len(query.Tags) > 0 { + encoded, err := json.Marshal(query.Tags) + if err != nil { + return nil, fmt.Errorf("failed to encode trace search tags: %w", err) + } + values.Set("tags", string(encoded)) + } + if !query.StartTime.IsZero() { + values.Set("start", strconv.FormatInt(query.StartTime.UnixMicro(), 10)) + } + if !query.EndTime.IsZero() { + values.Set("end", strconv.FormatInt(query.EndTime.UnixMicro(), 10)) + } + if query.MinDuration > 0 { + // Jaeger parses minDuration with time.ParseDuration, so the value has to + // carry a unit. A bare microsecond count is rejected with a 400 + // ("missing unit in duration"), so the duration is sent in its string + // form ("250ms") instead. + values.Set("minDuration", query.MinDuration.String()) + } + if query.Limit > 0 { + // One trace more than the caller asked for is requested. A full result + // then proves that more traces match than were returned, which is what + // the truncated flag has to tell the agent. + values.Set("limit", strconv.Itoa(query.Limit+1)) + } + + endpoint := *p.baseURL + endpoint.Path = strings.TrimRight(endpoint.Path, "/") + "/api/traces" + endpoint.RawQuery = values.Encode() + startedAt := time.Now() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint.String(), nil) + if err != nil { + return nil, fmt.Errorf("failed to create jaeger search request: %w", err) + } + req.Header.Set("Accept", "application/json") + if p.config.BearerToken != "" { + req.Header.Set("Authorization", "Bearer "+p.config.BearerToken) + } + if p.config.Tenant != "" { + req.Header.Set(tenantHeader, p.config.Tenant) + } + + resp, err := p.client.Do(req) + if err != nil { + p.logSearchFailure(query, startedAt, 0, "transport", err) + // The transport error embeds the request URL, which repeats the configured + // endpoint and carries the retrieval conditions. It is sanitized before it + // reaches the tool result and the model context. The service anchor is + // kept, because the model chose it and has to be able to report which + // lookup failed. + return nil, fmt.Errorf("jaeger trace search failed for service %s: %w", query.Service, sanitizedTransportError(err)) + } + defer resp.Body.Close() + if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices { + body, _ := io.ReadAll(io.LimitReader(resp.Body, maxTraceErrorBodyBytes)) + p.logSearchFailure(query, startedAt, resp.StatusCode, "upstream_status", nil) + return nil, fmt.Errorf("jaeger trace search failed with status %d: %s", resp.StatusCode, strings.TrimSpace(string(body))) + } + body, err := readTraceBody(resp.Body) + if err != nil { + p.logSearchFailure(query, startedAt, resp.StatusCode, "response_limit", err) + return nil, err + } + var upstream jaegerResponse + if err := json.Unmarshal(body, &upstream); err != nil { + p.logSearchFailure(query, startedAt, resp.StatusCode, "decode", err) + return nil, fmt.Errorf("failed to decode jaeger search response: %w", err) + } + if len(upstream.Errors) > 0 { + p.logSearchFailure(query, startedAt, resp.StatusCode, "upstream_error", nil) + return nil, fmt.Errorf("jaeger trace search failed: %s", upstream.Errors[0].Message) + } + result := summarizeJaegerTraces(upstream.Data, query.Limit) + logger.Sugar().Infow("trace search completed", "provider", p.config.Name, "engine", p.config.Type, "service", query.Service, "elapsed_ms", time.Since(startedAt).Milliseconds(), "traces", len(result.Traces), "total", result.Total, "truncated", result.Truncated) + return result, nil +} + +// jaegerTraceSummary keeps the numeric start time next to the summary so the result +// can be ordered on the absolute timeline instead of on the formatted timestamp. +type jaegerTraceSummary struct { + summary TraceSummary + startMicros int64 +} + +// summarizeJaegerTraces converts the raw traces into bounded per-trace summaries. +func summarizeJaegerTraces(traces []jaegerTrace, limit int) *TraceSearchResult { + result := &TraceSearchResult{Traces: []TraceSummary{}, SourceEngine: "jaeger"} + summaries := make([]jaegerTraceSummary, 0, len(traces)) + for _, trace := range traces { + summary, startMicros := summarizeJaegerTrace(trace) + summaries = append(summaries, jaegerTraceSummary{summary: summary, startMicros: startMicros}) + } + sort.SliceStable(summaries, func(i, j int) bool { + if summaries[i].startMicros != summaries[j].startMicros { + return summaries[i].startMicros < summaries[j].startMicros + } + return summaries[i].summary.TraceID < summaries[j].summary.TraceID + }) + // The provider was asked for one trace more than the caller wants, so a + // result that reached that limit proves that more traces match than were + // returned. Without the extra trace a full result would look complete. + total := len(traces) + if limit > 0 && len(summaries) > limit { + result.Truncated = true + summaries = summaries[:limit] + total = limit + } + if len(summaries) > maxSearchTraceSummaries { + summaries = summaries[:maxSearchTraceSummaries] + result.Truncated = true + } + for _, item := range summaries { + result.Traces = append(result.Traces, item.summary) + } + result.Total = total + return result +} + +// summarizeJaegerTrace derives the summary of one trace from all of its raw spans. +func summarizeJaegerTrace(input jaegerTrace) (TraceSummary, int64) { + summary := TraceSummary{TraceID: input.TraceID} + services := map[string]struct{}{} + var minStart, maxEnd int64 + hasStart := false + var rootSpan jaegerSpan + hasRoot := false + genai := false + // The span ids of the trace decide which spans are roots: a span whose parent + // is not part of the trace is an orphan and roots the trace on its own. + spanIDs := make(map[string]struct{}, len(input.Spans)) + for _, rawSpan := range input.Spans { + if rawSpan.SpanID != "" { + spanIDs[rawSpan.SpanID] = struct{}{} + } + } + for _, rawSpan := range input.Spans { + process := input.Processes[rawSpan.ProcessID] + if rawSpan.Process != nil { + process = *rawSpan.Process + } + if process.ServiceName != "" { + services[process.ServiceName] = struct{}{} + } + if errorSpan, _ := spanStatus(rawSpan.Tags); errorSpan { + summary.ErrorSpanCount++ + } + if hasGenAIJaegerTags(rawSpan.Tags) { + genai = true + } + if !hasStart || rawSpan.StartTime < minStart { + minStart = rawSpan.StartTime + hasStart = true + } + if end := rawSpan.StartTime + rawSpan.Duration; end > maxEnd { + maxEnd = end + } + if isJaegerRootSpan(rawSpan, spanIDs) && (!hasRoot || betterJaegerRoot(rawSpan, rootSpan)) { + rootSpan = rawSpan + hasRoot = true + summary.RootService = process.ServiceName + summary.RootOperation = rawSpan.OperationName + } + } + summary.SpanCount = len(input.Spans) + summary.ServiceCount = len(services) + summary.HasError = summary.ErrorSpanCount > 0 + if hasStart { + summary.StartTime = jaegerMicrosTime(minStart) + summary.DurationMicros = maxEnd - minStart + } + if genai { + summary.TraceKind = "genai" + } + return summary, minStart +} + +// isJaegerRootSpan reports whether the span roots the trace. A span is a root when +// it has no parent reference at all, or when the referenced parent is not part of +// the trace. The latter happens for orphans whose parent was dropped by the +// sampling or by the span limit. The analyzer uses the same rule, so the search +// summary and the diagnosis agree on the root of a trace. +func isJaegerRootSpan(rawSpan jaegerSpan, spanIDs map[string]struct{}) bool { + parentID := parentSpanID(rawSpan.References) + if parentID == "" { + return true + } + _, present := spanIDs[parentID] + return !present +} + +// betterJaegerRoot prefers the root span that finishes last, because that span +// determines the end of the request. +func betterJaegerRoot(candidate, current jaegerSpan) bool { + candidateEnd := candidate.StartTime + candidate.Duration + currentEnd := current.StartTime + current.Duration + if candidateEnd != currentEnd { + return candidateEnd > currentEnd + } + if candidate.Duration != current.Duration { + return candidate.Duration > current.Duration + } + return candidate.SpanID < current.SpanID +} + +// hasGenAIJaegerTags detects generative AI attributes on raw provider tags without +// allocating a normalized attribute map. +func hasGenAIJaegerTags(tags []jaegerKeyValue) bool { + for _, tag := range tags { + if isGenAIAttributeKey(tag.Key) { + return true + } + } + return false +} + func normalizeJaegerTrace(input jaegerTrace) *Trace { result := &Trace{TraceID: input.TraceID, SourceEngine: "jaeger", SpanCount: len(input.Spans), Spans: []Span{}} serviceSet := map[string]struct{}{} @@ -394,13 +645,74 @@ func truncateUTF8(value string, limit int) (string, bool) { return value, true } +// logFailure records a failed trace query. The error is sanitized, because the +// design document section 10 only allows the provider name, the service name, the +// elapsed time, the error type and the upstream status code in a failure log: the +// query payload must never be logged. func (p *jaegerProvider) logFailure(traceID string, startedAt time.Time, status int, errorType string, err error) { fields := []any{"provider", p.config.Name, "engine", p.config.Type, "trace_id", maskedTraceID(traceID), "elapsed_ms", time.Since(startedAt).Milliseconds(), "error_type", errorType} if status != 0 { fields = append(fields, "upstream_status", status) } - if err != nil { - fields = append(fields, "error", err) + if sanitized := sanitizeTransportError(err); sanitized != nil { + fields = append(fields, "error", sanitized) } logger.Sugar().Warnw("trace query failed", fields...) } + +// logSearchFailure records a failed search. The query payload and the credentials +// are deliberately left out so the log never leaks the lookup filters. The service +// name is kept because it is the only anchor of the failed lookup, but the error +// is sanitized because it would otherwise repeat the whole request URL. +func (p *jaegerProvider) logSearchFailure(query TraceSearchQuery, startedAt time.Time, status int, errorType string, err error) { + fields := []any{"provider", p.config.Name, "engine", p.config.Type, "service", query.Service, "elapsed_ms", time.Since(startedAt).Milliseconds(), "error_type", errorType} + if query.TraceID != "" { + fields = append(fields, "trace_id", maskedTraceID(query.TraceID)) + } + if status != 0 { + fields = append(fields, "upstream_status", status) + } + if sanitized := sanitizeTransportError(err); sanitized != nil { + fields = append(fields, "error", sanitized) + } + logger.Sugar().Warnw("trace search failed", fields...) +} + +// TODO: the tool result still returns the upstream error text of a failed +// response unchanged, and that text can carry sensitive data of the returned +// spans, such as user identifiers or business data. The request URL is no +// longer part of it, because the transport error is sanitized. A production +// deployment still has to decide per deployment scenario whether the span +// content of the tool result is filtered or redacted. That mechanism is not +// implemented yet, this round only records the decision. + +// sanitizeTransportError removes the request URL from a transport failure. +// http.Client.Do reports transport problems as *url.Error, whose message embeds +// the full request URL including the query string. The query carries the caller's +// filters and may contain business identifiers or secrets, so only the inner +// error is kept: it still explains the failure ("dial tcp ...: connection refused") +// without repeating the payload. The function guards the failure logs and the +// tool result. +func sanitizeTransportError(err error) error { + if err == nil { + return nil + } + var urlErr *url.Error + if errors.As(err, &urlErr) { + if urlErr.Err == nil { + return nil + } + return urlErr.Err + } + return err +} + +// sanitizedTransportError returns the sanitized form of a transport failure and +// never returns nil. The tool result needs a message in every case, while the +// failure logs only add the field when there is something to report. +func sanitizedTransportError(err error) error { + if sanitized := sanitizeTransportError(err); sanitized != nil { + return sanitized + } + return errors.New("unknown transport error") +} diff --git a/pkg/mcp/tools/trace/jaeger_logging_test.go b/pkg/mcp/tools/trace/jaeger_logging_test.go new file mode 100644 index 000000000..3eb670b24 --- /dev/null +++ b/pkg/mcp/tools/trace/jaeger_logging_test.go @@ -0,0 +1,247 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package trace + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "net/url" + "os" + "strings" + "testing" + "time" +) + +// The logger is a process wide singleton that writes to a file next to the +// working directory. The tests therefore read the lines that their own call +// appended instead of replacing the logger. + +// traceLogFile is the default output path of the shared logger. +const traceLogFile = "logs/dubbo-admin.log" + +// traceLogOffset returns the current size of the log file so that a test can +// read only the lines that its own call appends. +func traceLogOffset(t *testing.T) int64 { + t.Helper() + info, err := os.Stat(traceLogFile) + if err != nil { + if os.IsNotExist(err) { + return 0 + } + t.Fatal(err) + } + return info.Size() +} + +// traceLogLines returns the log lines that were appended after the offset. +func traceLogLines(t *testing.T, offset int64) []string { + t.Helper() + data, err := os.ReadFile(traceLogFile) + if err != nil { + if os.IsNotExist(err) { + return nil + } + t.Fatal(err) + } + if int64(len(data)) < offset { + offset = 0 + } + var lines []string + for _, line := range strings.Split(string(data[offset:]), "\n") { + if strings.TrimSpace(line) != "" { + lines = append(lines, line) + } + } + return lines +} + +// traceLogSince waits for the log lines that were appended after the offset. +func traceLogSince(t *testing.T, offset int64) []string { + t.Helper() + deadline := time.Now().Add(2 * time.Second) + for { + lines := traceLogLines(t, offset) + if len(lines) > 0 || time.Now().After(deadline) { + return lines + } + time.Sleep(20 * time.Millisecond) + } +} + +// traceLogErrorFields returns the error field of every appended log line. +func traceLogErrorFields(t *testing.T, lines []string) []string { + t.Helper() + var values []string + for _, line := range lines { + fields := map[string]any{} + if err := json.Unmarshal([]byte(line), &fields); err != nil { + continue + } + if value, ok := fields["error"]; ok { + values = append(values, fmt.Sprint(value)) + } + } + return values +} + +func TestSanitizeTransportError(t *testing.T) { + if sanitizeTransportError(nil) != nil { + t.Fatal("expected no error for a nil input") + } + inner := errors.New("dial tcp 127.0.0.1:1: connect: connection refused") + urlErr := &url.Error{Op: "Get", URL: "http://127.0.0.1:1/api/traces?service=secret-service", Err: inner} + sanitized := sanitizeTransportError(urlErr) + if sanitized != inner { + t.Fatalf("expected the inner error, got %v", sanitized) + } + if strings.Contains(sanitized.Error(), "secret-service") { + t.Fatalf("expected the request url to be removed, got %q", sanitized.Error()) + } + plain := errors.New("boom") + if sanitizeTransportError(plain) != plain { + t.Fatal("expected a plain error to be kept as it is") + } + // A wrapped transport error must be unwrapped as well. + wrapped := fmt.Errorf("jaeger trace query failed: %w", urlErr) + if sanitizeTransportError(wrapped) != inner { + t.Fatalf("expected the inner error of a wrapped failure, got %v", sanitizeTransportError(wrapped)) + } +} + +func TestJaegerProviderSearchFailureDoesNotLeakQueryPayload(t *testing.T) { + provider := newSearchProvider(t, "http://127.0.0.1:1") + offset := traceLogOffset(t) + if _, err := provider.SearchTraces(context.Background(), TraceSearchQuery{ + Service: "secret-service", + Operation: "secret-operation", + Tags: map[string]string{"secret.key": "secret-value"}, + }); err == nil { + t.Fatal("expected a transport failure") + } + lines := traceLogSince(t, offset) + if len(lines) == 0 { + t.Fatal("expected the failure to be logged") + } + logged := strings.Join(lines, "\n") + // The query payload must never reach the log. The service name itself is + // logged on purpose because it is the only anchor of the failed lookup, but + // it must not appear as part of the request URL. + for _, forbidden := range []string{ + "http://127.0.0.1:1/api/traces", + "service=secret-service", + "operation=secret-operation", + "tags=", + "secret-operation", + "secret-value", + "%7B", + } { + if strings.Contains(logged, forbidden) { + t.Fatalf("expected the log to omit %q, got %s", forbidden, logged) + } + } + // The recorded error must not repeat the request URL either. + recordedErrors := traceLogErrorFields(t, lines) + if len(recordedErrors) == 0 { + t.Fatal("expected the failure to record the error") + } + for _, value := range recordedErrors { + for _, forbidden := range []string{"/api/traces", "secret-service", "secret-operation", "secret-value"} { + if strings.Contains(value, forbidden) { + t.Fatalf("expected the error to omit %q, got %q", forbidden, value) + } + } + } +} + +func TestJaegerProviderTraceFailureDoesNotLeakTraceID(t *testing.T) { + provider := newSearchProvider(t, "http://127.0.0.1:1") + offset := traceLogOffset(t) + if _, err := provider.GetTraceByID(context.Background(), testTraceID); err == nil { + t.Fatal("expected a transport failure") + } + lines := traceLogSince(t, offset) + if len(lines) == 0 { + t.Fatal("expected the failure to be logged") + } + logged := strings.Join(lines, "\n") + // The trace id must be masked, the raw identifier must not be logged. + if strings.Contains(logged, testTraceID) { + t.Fatalf("expected the trace id to be masked, got %s", logged) + } + if !strings.Contains(logged, maskedTraceID(testTraceID)) { + t.Fatalf("expected the masked trace id in the log, got %s", logged) + } + for _, value := range traceLogErrorFields(t, lines) { + for _, forbidden := range []string{testTraceID, "/api/traces"} { + if strings.Contains(value, forbidden) { + t.Fatalf("expected the error to omit %q, got %q", forbidden, value) + } + } + } +} + +// The tool result reaches the model context, so it keeps the transport detail of +// the upstream failure: the model has to be able to answer which lookup failed +// and why. The request URL must not appear in it, neither in the tool result nor +// in the failure log: it repeats the configured endpoint and carries the +// retrieval conditions, and an endpoint may be configured with a credential. +func TestJaegerProviderToolResultKeepsTransportDetailButNotTheURL(t *testing.T) { + provider := newSearchProvider(t, "http://127.0.0.1:1") + offset := traceLogOffset(t) + _, err := provider.GetTraceByID(context.Background(), testTraceID) + if err == nil { + t.Fatal("expected a transport failure") + } + // The trace identifier may reappear in the tool result: the model built it + // itself and it has to be able to report the failed lookup back to the user. + if !strings.Contains(err.Error(), testTraceID) { + t.Fatalf("expected the tool result to keep the trace identifier: %s", err.Error()) + } + // The request URL must not: it repeats the endpoint path and the query + // string. The "dial tcp host:port" form of the inner error stays, it names + // the host that refused the connection without repeating the payload. + if strings.Contains(err.Error(), "/api/traces/") { + t.Fatalf("expected the tool result to omit the request URL: %s", err.Error()) + } + if !strings.Contains(err.Error(), "connection refused") { + t.Fatalf("expected the tool result to keep the diagnostic detail: %s", err.Error()) + } + + // The log must not contain the request URL and must not repeat the raw trace + // identifier. + lines := traceLogSince(t, offset) + if len(lines) == 0 { + t.Fatal("expected the failure to be logged") + } + logged := strings.Join(lines, "\n") + if strings.Contains(logged, testTraceID) { + t.Fatalf("expected the log to mask the trace identifier, got %s", logged) + } + if strings.Contains(logged, "/api/traces") { + t.Fatalf("expected the log to omit the request URL, got %s", logged) + } + for _, value := range traceLogErrorFields(t, lines) { + for _, forbidden := range []string{testTraceID, "/api/traces"} { + if strings.Contains(value, forbidden) { + t.Fatalf("expected the error to omit %q, got %q", forbidden, value) + } + } + } +} diff --git a/pkg/mcp/tools/trace/jaeger_test.go b/pkg/mcp/tools/trace/jaeger_test.go index 8132b611b..5668bada2 100644 --- a/pkg/mcp/tools/trace/jaeger_test.go +++ b/pkg/mcp/tools/trace/jaeger_test.go @@ -21,6 +21,7 @@ import ( "context" "net/http" "net/http/httptest" + "strings" "testing" observabilitycfg "github.com/apache/dubbo-admin/pkg/config/observability" @@ -97,3 +98,32 @@ func TestNormalizeJaegerTraceTruncatesSpans(t *testing.T) { t.Fatalf("expected trace truncation, got %#v", result) } } + +func TestNewJaegerProviderRejectsEndpointWithQuery(t *testing.T) { + // A credential in the endpoint query would be carried into every request URL + // and into the transport error of a failed lookup, which reaches the tool + // result and the model context. Credentials belong in bearerToken. + const endpoint = "https://jaeger.example?api_key=super-secret" + _, err := newJaegerProvider(observabilitycfg.TraceProviderConfig{ + Name: "jaeger-main", Type: observabilitycfg.TraceProviderJaeger, Endpoint: endpoint, + }) + if err == nil { + t.Fatal("expected an endpoint with a query string to be rejected") + } + // The rejection message must not repeat the endpoint, because the endpoint + // is exactly what holds the secret. + if strings.Contains(err.Error(), "super-secret") { + t.Fatalf("expected the rejection to omit the endpoint: %s", err.Error()) + } +} + +func TestNewJaegerProviderRejectsInvalidEndpoint(t *testing.T) { + for _, endpoint := range []string{"jaeger.example:16686", "ftp://jaeger.example", "https://", ""} { + _, err := newJaegerProvider(observabilitycfg.TraceProviderConfig{ + Name: "jaeger-main", Type: observabilitycfg.TraceProviderJaeger, Endpoint: endpoint, + }) + if err == nil { + t.Fatalf("expected the endpoint %q to be rejected", endpoint) + } + } +} diff --git a/pkg/mcp/tools/trace/model.go b/pkg/mcp/tools/trace/model.go index 78d07ce60..170a52e28 100644 --- a/pkg/mcp/tools/trace/model.go +++ b/pkg/mcp/tools/trace/model.go @@ -17,6 +17,8 @@ package trace +import "time" + // Trace is a backend-independent, bounded representation of a distributed trace. type Trace struct { TraceID string `json:"traceId"` @@ -53,3 +55,161 @@ type Event struct { Timestamp string `json:"timestamp"` Attributes map[string]string `json:"attributes,omitempty"` } + +// TraceSearchQuery bounds a provider-side trace lookup. +type TraceSearchQuery struct { + Service string + Operation string + TraceID string + Tags map[string]string + StartTime time.Time + EndTime time.Time + Limit int + MinDuration time.Duration +} + +// TraceSummary is a bounded per-trace overview produced by trace search. +type TraceSummary struct { + TraceID string `json:"traceId"` + RootService string `json:"rootService,omitempty"` + RootOperation string `json:"rootOperation,omitempty"` + StartTime string `json:"startTime"` + DurationMicros int64 `json:"durationMicros"` + SpanCount int `json:"spanCount"` + ErrorSpanCount int `json:"errorSpanCount"` + ServiceCount int `json:"serviceCount"` + HasError bool `json:"hasError"` + TraceKind string `json:"traceKind,omitempty"` // "genai" when the trace carries gen_ai.* attributes +} + +// TraceSearchResult aggregates the outcome of a trace search. +type TraceSearchResult struct { + Traces []TraceSummary `json:"traces"` + Total int `json:"total"` + Truncated bool `json:"truncated"` + SourceEngine string `json:"sourceEngine"` +} + +// CriticalPathEntry is one hop of the latency-determining root-to-leaf chain. +type CriticalPathEntry struct { + SpanID string `json:"spanId"` + ServiceName string `json:"serviceName"` + OperationName string `json:"operationName"` + StartTime string `json:"startTime"` + DurationMicros int64 `json:"durationMicros"` + SelfMicros int64 `json:"selfMicros"` + WaitMicros int64 `json:"waitMicros"` +} + +// ServiceLatency aggregates the spans contributed by one service. +type ServiceLatency struct { + ServiceName string `json:"serviceName"` + SpanCount int `json:"spanCount"` + ErrorSpanCount int `json:"errorSpanCount"` + TotalMicros int64 `json:"totalMicros"` + MaxMicros int64 `json:"maxMicros"` + AvgMicros int64 `json:"avgMicros"` + SharePercent float64 `json:"sharePercent"` +} + +// SlowSpan is a span ranked by the time it consumed on its own. +type SlowSpan struct { + SpanID string `json:"spanId"` + ServiceName string `json:"serviceName"` + OperationName string `json:"operationName"` + DurationMicros int64 `json:"durationMicros"` + SharePercent float64 `json:"sharePercent"` + OnCriticalPath bool `json:"onCriticalPath"` +} + +// ErrorSpan describes a failed span together with its root-cause verdict. +type ErrorSpan struct { + SpanID string `json:"spanId"` + ServiceName string `json:"serviceName"` + OperationName string `json:"operationName"` + ErrorType string `json:"errorType,omitempty"` + ErrorMessage string `json:"errorMessage,omitempty"` + Status string `json:"status"` + RootCause bool `json:"rootCause"` +} + +// DependencyEdge is a cross-service call observed inside a single trace. +type DependencyEdge struct { + CallerService string `json:"callerService"` + CalleeService string `json:"calleeService"` + SpanCount int `json:"spanCount"` + TotalMicros int64 `json:"totalMicros"` + ErrorCount int `json:"errorCount"` +} + +// GenAIModelCall describes one generative model invocation observed in a trace. +type GenAIModelCall struct { + SpanID string `json:"spanId"` + ServiceName string `json:"serviceName"` + OperationName string `json:"operationName"` + Operation string `json:"operation,omitempty"` + ProviderName string `json:"providerName,omitempty"` + RequestModel string `json:"requestModel,omitempty"` + ResponseModel string `json:"responseModel,omitempty"` + InputTokens int `json:"inputTokens,omitempty"` + OutputTokens int `json:"outputTokens,omitempty"` + TotalTokens int `json:"totalTokens,omitempty"` + DurationMicros int64 `json:"durationMicros"` + Error bool `json:"error"` +} + +// GenAIToolCall describes one tool invocation observed in a trace. +type GenAIToolCall struct { + SpanID string `json:"spanId"` + ServiceName string `json:"serviceName"` + ToolName string `json:"toolName,omitempty"` + ToolCallID string `json:"toolCallId,omitempty"` + DurationMicros int64 `json:"durationMicros"` + Error bool `json:"error"` +} + +// GenAIAgent describes one agent-scoped span observed in a trace. +type GenAIAgent struct { + SpanID string `json:"spanId"` + ServiceName string `json:"serviceName"` + AgentName string `json:"agentName,omitempty"` + AgentID string `json:"agentId,omitempty"` + DurationMicros int64 `json:"durationMicros"` + Error bool `json:"error"` +} + +// GenAISummary aggregates the generative AI footprint of a trace. +type GenAISummary struct { + Present bool `json:"present"` + ConversationIDs []string `json:"conversationIds,omitempty"` + Providers []string `json:"providers,omitempty"` + Models []string `json:"models,omitempty"` + InputTokens int `json:"inputTokens"` + OutputTokens int `json:"outputTokens"` + TotalTokens int `json:"totalTokens"` + ModelCallCount int `json:"modelCallCount"` + ToolCallCount int `json:"toolCallCount"` + AgentCount int `json:"agentCount"` + ModelCalls []GenAIModelCall `json:"modelCalls,omitempty"` + ToolCalls []GenAIToolCall `json:"toolCalls,omitempty"` + Agents []GenAIAgent `json:"agents,omitempty"` + Truncated bool `json:"truncated"` +} + +// TraceAnalysis is the structured diagnosis of a single trace. +type TraceAnalysis struct { + TraceID string `json:"traceId"` + Summary string `json:"summary"` + StartTime string `json:"startTime"` + DurationMicros int64 `json:"durationMicros"` + SpanCount int `json:"spanCount"` + ServiceCount int `json:"serviceCount"` + ErrorSpanCount int `json:"errorSpanCount"` + CriticalPath []CriticalPathEntry `json:"criticalPath,omitempty"` + ServiceBreakdown []ServiceLatency `json:"serviceBreakdown,omitempty"` + SlowSpans []SlowSpan `json:"slowSpans,omitempty"` + ErrorSpans []ErrorSpan `json:"errorSpans,omitempty"` + Dependencies []DependencyEdge `json:"dependencies,omitempty"` + GenAI *GenAISummary `json:"genai,omitempty"` + Truncated bool `json:"truncated"` +} diff --git a/pkg/mcp/tools/trace/provider.go b/pkg/mcp/tools/trace/provider.go index 863256af2..decf86c3a 100644 --- a/pkg/mcp/tools/trace/provider.go +++ b/pkg/mcp/tools/trace/provider.go @@ -27,6 +27,9 @@ import ( // Provider hides backend wire formats behind a stable trace lookup contract. type Provider interface { GetTraceByID(ctx context.Context, traceID string) (*Trace, error) + // SearchTraces discovers candidate traces without knowing a trace identifier up + // front. Implementations must keep the result bounded. + SearchTraces(ctx context.Context, query TraceSearchQuery) (*TraceSearchResult, error) } func newProvider(cfg observabilitycfg.TraceProviderConfig) (Provider, error) { diff --git a/pkg/mcp/tools/trace/search.go b/pkg/mcp/tools/trace/search.go new file mode 100644 index 000000000..cca6b565a --- /dev/null +++ b/pkg/mcp/tools/trace/search.go @@ -0,0 +1,172 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package trace + +import ( + "fmt" + "strings" + "time" + + consolectx "github.com/apache/dubbo-admin/pkg/console/context" + "github.com/apache/dubbo-admin/pkg/mcp/common" +) + +const ( + // defaultSearchLimit is the number of traces requested from the provider when the + // caller does not ask for a specific amount. + defaultSearchLimit = 20 + // maxSearchLimit bounds the limit that is forwarded to the provider. + maxSearchLimit = 100 + // maxSearchTraceSummaries bounds how many per-trace summaries are kept locally. + // The provider may return more traces than the agent can consume. + maxSearchTraceSummaries = 100 +) + +// SearchTraces discovers candidate traces for a service so the agent can start an +// investigation without asking the user for a trace identifier. +func SearchTraces(ctx consolectx.Context, args map[string]any) (*common.ToolResult, error) { + query, err := buildTraceSearchQuery(args) + if err != nil { + return common.ErrorResult(err), nil + } + providerConfig, err := providerConfigFromContext(ctx, common.NewArgsHelper(args).GetString("provider", "")) + if err != nil { + return common.ErrorResult(err), nil + } + provider, err := newProvider(providerConfig) + if err != nil { + return common.ErrorResult(err), nil + } + result, err := provider.SearchTraces(traceRequestContext(ctx), query) + if err != nil { + return common.ErrorResult(err), nil + } + return common.JsonResult(result) +} + +// SearchProperties returns the MCP input schema properties for trace search. +func SearchProperties() map[string]common.PropertyDef { + return map[string]common.PropertyDef{ + "service": {Type: "string", Description: "Service name used as the trace search anchor"}, + "operation": {Type: "string", Description: "Optional root operation name filter"}, + "traceId": {Type: "string", Description: "Optional Jaeger 64-bit or 128-bit hexadecimal TraceID anchor"}, + "tags": {Type: "string", Description: "Optional tag filter formatted as key=value pairs separated by commas"}, + "startTime": {Type: "string", Description: "Optional RFC3339 lower bound of the search window"}, + "endTime": {Type: "string", Description: "Optional RFC3339 upper bound of the search window"}, + "minDurationMs": {Type: "integer", Description: "Optional minimum trace duration in milliseconds"}, + "limit": {Type: "integer", Description: "Maximum number of traces to request", Default: defaultSearchLimit}, + "provider": {Type: "string", Description: "Optional trace provider name; defaults to defaultProvider"}, + } +} + +// buildTraceSearchQuery validates the tool arguments and converts them into the +// provider side query. +func buildTraceSearchQuery(args map[string]any) (TraceSearchQuery, error) { + helper := common.NewArgsHelper(args) + service := strings.TrimSpace(helper.GetString("service", "")) + if service == "" { + return TraceSearchQuery{}, fmt.Errorf("service is required for trace search") + } + query := TraceSearchQuery{Service: service} + query.Operation = strings.TrimSpace(helper.GetString("operation", "")) + + if traceID := strings.TrimSpace(helper.GetString("traceId", "")); traceID != "" { + if !traceIDPattern.MatchString(traceID) { + return TraceSearchQuery{}, fmt.Errorf("traceId must be a 16 or 32 character hexadecimal value") + } + query.TraceID = traceID + } + + tags, err := parseSearchTags(helper.GetString("tags", "")) + if err != nil { + return TraceSearchQuery{}, err + } + query.Tags = tags + + startTime, err := parseSearchTime(helper.GetString("startTime", ""), "startTime") + if err != nil { + return TraceSearchQuery{}, err + } + query.StartTime = startTime + + endTime, err := parseSearchTime(helper.GetString("endTime", ""), "endTime") + if err != nil { + return TraceSearchQuery{}, err + } + query.EndTime = endTime + if !query.StartTime.IsZero() && !query.EndTime.IsZero() && query.EndTime.Before(query.StartTime) { + return TraceSearchQuery{}, fmt.Errorf("endTime must not be earlier than startTime") + } + + minDurationMs := helper.GetInt("minDurationMs", 0) + if minDurationMs < 0 { + return TraceSearchQuery{}, fmt.Errorf("minDurationMs must not be negative") + } + query.MinDuration = time.Duration(minDurationMs) * time.Millisecond + + limit := helper.GetInt("limit", defaultSearchLimit) + if limit <= 0 { + limit = defaultSearchLimit + } + if limit > maxSearchLimit { + limit = maxSearchLimit + } + query.Limit = limit + return query, nil +} + +// parseSearchTags converts the "k=v,k2=v2" filter syntax into a tag map. +// A repeated key keeps the last value, which matches the object semantics of the +// provider side tag filter. +func parseSearchTags(raw string) (map[string]string, error) { + trimmed := strings.TrimSpace(raw) + if trimmed == "" { + return nil, nil + } + result := map[string]string{} + for _, pair := range strings.Split(trimmed, ",") { + pair = strings.TrimSpace(pair) + if pair == "" { + continue + } + key, value, found := strings.Cut(pair, "=") + key = strings.TrimSpace(key) + if !found || key == "" { + return nil, fmt.Errorf("tags must be formatted as key=value pairs separated by commas") + } + result[key] = strings.TrimSpace(value) + } + if len(result) == 0 { + return nil, nil + } + return result, nil +} + +// parseSearchTime converts an RFC3339 timestamp into a time value. An empty input +// yields the zero time, which the provider treats as "unbounded". +func parseSearchTime(raw string, name string) (time.Time, error) { + trimmed := strings.TrimSpace(raw) + if trimmed == "" { + return time.Time{}, nil + } + parsed, err := time.Parse(time.RFC3339, trimmed) + if err != nil { + return time.Time{}, fmt.Errorf("%s must be an RFC3339 timestamp: %w", name, err) + } + return parsed, nil +} diff --git a/pkg/mcp/tools/trace/search_test.go b/pkg/mcp/tools/trace/search_test.go new file mode 100644 index 000000000..9b7406659 --- /dev/null +++ b/pkg/mcp/tools/trace/search_test.go @@ -0,0 +1,500 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package trace + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "testing" + "time" + + observabilitycfg "github.com/apache/dubbo-admin/pkg/config/observability" +) + +const testSearchTraceID = "aabbccddeeff00112233445566778899" + +func newSearchProvider(t *testing.T, endpoint string) *jaegerProvider { + t.Helper() + provider, err := newJaegerProvider(observabilitycfg.TraceProviderConfig{ + Name: "jaeger-main", Type: observabilitycfg.TraceProviderJaeger, Endpoint: endpoint, BearerToken: "secret", Tenant: "tenant-a", + }) + if err != nil { + t.Fatal(err) + } + return provider +} + +func TestJaegerProviderSearchTracesBuildsRequest(t *testing.T) { + var captured *http.Request + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + captured = r + _, _ = w.Write([]byte(`{"data":[{"traceID":"` + testSearchTraceID + `","spans":[{"traceID":"` + testSearchTraceID + `","spanID":"01","operationName":"root","references":[],"startTime":2000000,"duration":3000,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}]}`)) + })) + defer server.Close() + + provider := newSearchProvider(t, server.URL) + start := time.Unix(1700000000, 0).UTC() + end := time.Unix(1700003600, 0).UTC() + result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{ + Service: "shop-order", + Operation: "checkout", + TraceID: testSearchTraceID, + Tags: map[string]string{"rpc.system": "apache_dubbo"}, + StartTime: start, + EndTime: end, + Limit: 17, + MinDuration: 250 * time.Millisecond, + }) + if err != nil { + t.Fatal(err) + } + if captured == nil { + t.Fatal("expected the provider to issue a request") + } + if captured.URL.Path != "/api/traces" { + t.Fatalf("unexpected path %s", captured.URL.Path) + } + if captured.Header.Get("Authorization") != "Bearer secret" || captured.Header.Get(tenantHeader) != "tenant-a" { + t.Fatalf("missing auth headers: %#v", captured.Header) + } + query := captured.URL.Query() + if query.Get("service") != "shop-order" || query.Get("operation") != "checkout" { + t.Fatalf("unexpected service filter: %v", query) + } + if query.Get("traceID") != testSearchTraceID { + t.Fatalf("unexpected trace id filter: %v", query) + } + if query.Get("tags") != `{"rpc.system":"apache_dubbo"}` { + t.Fatalf("unexpected tags filter: %v", query) + } + if query.Get("start") != "1700000000000000" || query.Get("end") != "1700003600000000" { + t.Fatalf("unexpected time window: %v", query) + } + // Jaeger parses minDuration with time.ParseDuration, so the wire format is a + // duration string and not a bare microsecond count. + minDuration := query.Get("minDuration") + if minDuration != "250ms" { + t.Fatalf("unexpected min duration: %v", query) + } + // The stub does not validate the parameter like Jaeger does, so the value is + // parsed here to keep a unit-less microsecond count from slipping through. + parsed, err := time.ParseDuration(minDuration) + if err != nil { + t.Fatalf("min duration %q is not a duration Jaeger can parse: %v", minDuration, err) + } + if parsed != 250*time.Millisecond { + t.Fatalf("unexpected parsed min duration: %v", parsed) + } + // One trace more than the caller asked for is requested, so a full result + // proves that more traces match than were returned. + if query.Get("limit") != "18" { + t.Fatalf("unexpected limit: %v", query) + } + if result.Total != 1 || len(result.Traces) != 1 || result.SourceEngine != "jaeger" { + t.Fatalf("unexpected search result: %#v", result) + } + summary := result.Traces[0] + if summary.TraceID != testSearchTraceID || summary.RootService != "shop-order" || summary.RootOperation != "root" { + t.Fatalf("unexpected summary: %#v", summary) + } + if summary.SpanCount != 1 || summary.ServiceCount != 1 || summary.ErrorSpanCount != 0 || summary.HasError { + t.Fatalf("unexpected summary counters: %#v", summary) + } + if summary.DurationMicros != 3000 { + t.Fatalf("unexpected summary duration: %#v", summary) + } +} + +func TestJaegerProviderSearchTracesOmitsEmptyParameters(t *testing.T) { + var captured *http.Request + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + captured = r + _, _ = w.Write([]byte(`{"data":[]}`)) + })) + defer server.Close() + + provider := newSearchProvider(t, server.URL) + result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"}) + if err != nil { + t.Fatal(err) + } + query := captured.URL.Query() + for _, key := range []string{"operation", "traceID", "tags", "start", "end", "minDuration", "limit"} { + if query.Has(key) { + t.Fatalf("expected parameter %q to be omitted, got %v", key, query) + } + } + if query.Get("service") != "shop-order" { + t.Fatalf("unexpected service filter: %v", query) + } + if result.Total != 0 || len(result.Traces) != 0 || result.Truncated { + t.Fatalf("expected an empty but successful result, got %#v", result) + } +} + +func TestJaegerProviderSearchTracesRequiresService(t *testing.T) { + provider := newSearchProvider(t, "http://127.0.0.1:1") + _, err := provider.SearchTraces(context.Background(), TraceSearchQuery{}) + if err == nil { + t.Fatal("expected a missing service error") + } + if !strings.Contains(err.Error(), "service") { + t.Fatalf("expected the error to name the missing service, got %v", err) + } +} + +func TestJaegerProviderSearchTracesSummarizesOrphanRoot(t *testing.T) { + // The parent of the span is not part of the trace, so the span is an orphan + // and roots the trace on its own. The summary and the analyzer have to agree + // on that root. + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"data":[{"traceID":"` + testSearchTraceID + `","spans":[{"traceID":"` + testSearchTraceID + `","spanID":"02","operationName":"orphan","references":[{"refType":"CHILD_OF","traceID":"` + testSearchTraceID + `","spanID":"missing"}],"startTime":2000000,"duration":3000,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}]}`)) + })) + defer server.Close() + + provider := newSearchProvider(t, server.URL) + result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"}) + if err != nil { + t.Fatal(err) + } + if len(result.Traces) != 1 { + t.Fatalf("unexpected number of traces: %#v", result.Traces) + } + summary := result.Traces[0] + if summary.RootService != "shop-order" || summary.RootOperation != "orphan" { + t.Fatalf("expected the orphan to be reported as the root: %#v", summary) + } +} + +func TestJaegerProviderSearchTracesDetectsGenAIKind(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"data":[{"traceID":"` + testSearchTraceID + `","spans":[{"traceID":"` + testSearchTraceID + `","spanID":"01","operationName":"chat","references":[],"startTime":2000000,"duration":3000,"tags":[{"key":"gen_ai.operation.name","type":"string","value":"chat"}],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"dubbo-admin-sre-agent","tags":[]}}}]}`)) + })) + defer server.Close() + + provider := newSearchProvider(t, server.URL) + result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "dubbo-admin-sre-agent"}) + if err != nil { + t.Fatal(err) + } + if len(result.Traces) != 1 || result.Traces[0].TraceKind != "genai" { + t.Fatalf("expected a genai trace kind, got %#v", result.Traces) + } +} + +func TestJaegerProviderSearchTracesSortsByStartTime(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + // The traces are returned out of order and use timestamps whose RFC3339 + // rendering would sort differently than the numeric microsecond value. + _, _ = w.Write([]byte(`{"data":[ + {"traceID":"cccccccccccccccccccccccccccccccc","spans":[{"spanID":"01","operationName":"late","references":[],"startTime":9000000,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}, + {"traceID":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","spans":[{"spanID":"01","operationName":"early","references":[],"startTime":1000000,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}, + {"traceID":"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","spans":[{"spanID":"01","operationName":"middle","references":[],"startTime":5000000,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}} + ]}`)) + })) + defer server.Close() + + provider := newSearchProvider(t, server.URL) + result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"}) + if err != nil { + t.Fatal(err) + } + expected := []string{"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "cccccccccccccccccccccccccccccccc"} + if len(result.Traces) != len(expected) { + t.Fatalf("unexpected number of traces: %#v", result.Traces) + } + for i, traceID := range expected { + if result.Traces[i].TraceID != traceID { + t.Fatalf("unexpected trace order: %v", result.Traces) + } + } +} + +func TestJaegerProviderSearchTracesTruncatesSummaries(t *testing.T) { + spans := make([]string, 0, maxSearchTraceSummaries+5) + for i := 0; i < maxSearchTraceSummaries+5; i++ { + spans = append(spans, `{"traceID":"trace`+strconv.Itoa(i)+`","spans":[{"spanID":"01","operationName":"op","references":[],"startTime":`+strconv.Itoa(1000000+i)+`,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}`) + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"data":[` + strings.Join(spans, ",") + `]}`)) + })) + defer server.Close() + + provider := newSearchProvider(t, server.URL) + result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"}) + if err != nil { + t.Fatal(err) + } + if !result.Truncated { + t.Fatal("expected the result to be marked as truncated") + } + if len(result.Traces) != maxSearchTraceSummaries { + t.Fatalf("unexpected number of summaries: %d", len(result.Traces)) + } + if result.Total != maxSearchTraceSummaries+5 { + t.Fatalf("expected the backend total to be preserved, got %d", result.Total) + } +} + +func TestJaegerProviderSearchTracesReportsUpstreamFailure(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + _, _ = w.Write([]byte("upstream exploded")) + })) + defer server.Close() + + provider := newSearchProvider(t, server.URL) + result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"}) + if err == nil { + t.Fatal("expected an upstream error") + } + if result != nil { + t.Fatalf("expected no result on failure, got %#v", result) + } + if !strings.Contains(err.Error(), "500") { + t.Fatalf("expected the status code in the error, got %v", err) + } +} + +func TestJaegerProviderSearchTracesReportsEnvelopeErrors(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"data":[],"errors":[{"code":500,"msg":"query rejected"}]}`)) + })) + defer server.Close() + + provider := newSearchProvider(t, server.URL) + if _, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order"}); err == nil { + t.Fatal("expected an envelope error") + } +} + +func TestBuildTraceSearchQuery(t *testing.T) { + query, err := buildTraceSearchQuery(map[string]any{ + "service": "shop-order", + "operation": "checkout", + "traceId": testSearchTraceID, + "tags": "rpc.system=apache_dubbo, http.status_code=500", + "startTime": "2026-10-24T10:00:00Z", + "endTime": "2026-10-24T11:00:00Z", + "minDurationMs": 250, + "limit": 33, + }) + if err != nil { + t.Fatal(err) + } + if query.Service != "shop-order" || query.Operation != "checkout" || query.TraceID != testSearchTraceID { + t.Fatalf("unexpected query: %#v", query) + } + if len(query.Tags) != 2 || query.Tags["rpc.system"] != "apache_dubbo" || query.Tags["http.status_code"] != "500" { + t.Fatalf("unexpected tags: %#v", query.Tags) + } + if query.StartTime.UTC().Format(time.RFC3339) != "2026-10-24T10:00:00Z" { + t.Fatalf("unexpected start time: %v", query.StartTime) + } + if query.EndTime.UTC().Format(time.RFC3339) != "2026-10-24T11:00:00Z" { + t.Fatalf("unexpected end time: %v", query.EndTime) + } + if query.MinDuration != 250*time.Millisecond { + t.Fatalf("unexpected min duration: %v", query.MinDuration) + } + if query.Limit != 33 { + t.Fatalf("unexpected limit: %d", query.Limit) + } +} + +func TestBuildTraceSearchQueryDefaultsAndLimits(t *testing.T) { + query, err := buildTraceSearchQuery(map[string]any{"service": "shop-order"}) + if err != nil { + t.Fatal(err) + } + if query.Limit != defaultSearchLimit { + t.Fatalf("unexpected default limit: %d", query.Limit) + } + if !query.StartTime.IsZero() || !query.EndTime.IsZero() || query.MinDuration != 0 || query.Tags != nil { + t.Fatalf("expected empty optional filters, got %#v", query) + } + + query, err = buildTraceSearchQuery(map[string]any{"service": "shop-order", "limit": float64(1000)}) + if err != nil { + t.Fatal(err) + } + if query.Limit != maxSearchLimit { + t.Fatalf("expected the limit to be capped, got %d", query.Limit) + } + + query, err = buildTraceSearchQuery(map[string]any{"service": "shop-order", "limit": float64(-5)}) + if err != nil { + t.Fatal(err) + } + if query.Limit != defaultSearchLimit { + t.Fatalf("expected a negative limit to fall back, got %d", query.Limit) + } +} + +func TestBuildTraceSearchQueryRejectsInvalidArguments(t *testing.T) { + tests := []struct { + name string + args map[string]any + }{ + {name: "missing service", args: map[string]any{}}, + {name: "blank service", args: map[string]any{"service": " "}}, + {name: "invalid trace id", args: map[string]any{"service": "shop-order", "traceId": "not-hex"}}, + {name: "malformed tags", args: map[string]any{"service": "shop-order", "tags": "rpc.system"}}, + {name: "tags without key", args: map[string]any{"service": "shop-order", "tags": "=value"}}, + {name: "invalid start time", args: map[string]any{"service": "shop-order", "startTime": "2026-10-24"}}, + {name: "invalid end time", args: map[string]any{"service": "shop-order", "endTime": "yesterday"}}, + {name: "reversed window", args: map[string]any{"service": "shop-order", "startTime": "2026-10-24T11:00:00Z", "endTime": "2026-10-24T10:00:00Z"}}, + {name: "negative duration", args: map[string]any{"service": "shop-order", "minDurationMs": float64(-1)}}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + if _, err := buildTraceSearchQuery(test.args); err == nil { + t.Fatal("expected a validation error") + } + }) + } +} + +func TestParseSearchTags(t *testing.T) { + tags, err := parseSearchTags("") + if err != nil || tags != nil { + t.Fatalf("expected no tags, got %#v (%v)", tags, err) + } + tags, err = parseSearchTags(" a = 1 , b=2 ") + if err != nil { + t.Fatal(err) + } + if tags["a"] != "1" || tags["b"] != "2" { + t.Fatalf("unexpected tags: %#v", tags) + } + tags, err = parseSearchTags("a=1,a=2") + if err != nil { + t.Fatal(err) + } + if tags["a"] != "2" { + t.Fatalf("expected the last value to win, got %#v", tags) + } + tags, err = parseSearchTags("a=b=c") + if err != nil { + t.Fatal(err) + } + if tags["a"] != "b=c" { + t.Fatalf("unexpected tags: %#v", tags) + } + if _, err := parseSearchTags("broken"); err == nil { + t.Fatal("expected a malformed tag error") + } +} + +func TestSearchPropertiesExposeRequiredParameters(t *testing.T) { + properties := SearchProperties() + for _, name := range []string{"service", "operation", "traceId", "tags", "startTime", "endTime", "minDurationMs", "limit", "provider"} { + if _, ok := properties[name]; !ok { + t.Fatalf("search properties are missing %q", name) + } + } + if properties["minDurationMs"].Type != "integer" { + t.Fatalf("unexpected minDurationMs type: %s", properties["minDurationMs"].Type) + } +} + +func TestSearchTracesToolResultIsSerializable(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"data":[{"traceID":"` + testSearchTraceID + `","spans":[{"traceID":"` + testSearchTraceID + `","spanID":"01","operationName":"chat","references":[],"startTime":2000000,"duration":3000,"tags":[{"key":"gen_ai.operation.name","type":"string","value":"chat"}],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}]}`)) + })) + defer server.Close() + + // The handler must produce a payload that survives a round trip. + result, err := SearchTraces(newTraceToolTestContext(server.URL), map[string]any{"service": "shop-order"}) + if err != nil { + t.Fatal(err) + } + if result.IsError { + t.Fatalf("unexpected error result: %s", result.Content[0].Text) + } + var payload TraceSearchResult + if err := json.Unmarshal([]byte(result.Content[0].Text), &payload); err != nil { + t.Fatal(err) + } + if payload.Total != 1 || len(payload.Traces) != 1 { + t.Fatalf("unexpected payload: %#v", payload) + } + if payload.Traces[0].TraceID != testSearchTraceID { + t.Fatalf("unexpected summary: %#v", payload.Traces[0]) + } + if !strings.Contains(result.Content[0].Text, `"traceId":"`+testSearchTraceID+`"`) { + t.Fatalf("unexpected payload: %s", result.Content[0].Text) + } + if !strings.Contains(result.Content[0].Text, `"traceKind":"genai"`) { + t.Fatalf("expected the genai kind in the payload: %s", result.Content[0].Text) + } + // The tool result envelope must be serializable as well. + if _, err := json.Marshal(result); err != nil { + t.Fatal(err) + } +} + +func TestJaegerProviderSearchTracesReportsTruncationAtTheBackendLimit(t *testing.T) { + // The provider is asked for one trace more than the caller wants, so a result + // that reached the limit proves that more traces match than were returned. + // Without the extra trace a full result would look complete. + available := 30 + traces := make([]string, 0, available) + for i := 0; i < available; i++ { + traces = append(traces, `{"traceID":"trace`+strconv.Itoa(i)+`","spans":[{"spanID":"01","operationName":"op","references":[],"startTime":`+strconv.Itoa(1000000+i)+`,"duration":10,"tags":[],"logs":[],"processID":"p1"}],"processes":{"p1":{"serviceName":"shop-order","tags":[]}}}`) + } + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + limit, _ := strconv.Atoi(r.URL.Query().Get("limit")) + if limit > len(traces) { + limit = len(traces) + } + _, _ = w.Write([]byte(`{"data":[` + strings.Join(traces[:limit], ",") + `]}`)) + })) + defer server.Close() + provider := newSearchProvider(t, server.URL) + + // The backend has fewer traces than the caller asked for, so the result is + // complete even though the request asked for one trace more. + result, err := provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order", Limit: 50}) + if err != nil { + t.Fatal(err) + } + if result.Truncated { + t.Fatal("did not expect the result to be truncated") + } + if len(result.Traces) != available || result.Total != available { + t.Fatalf("unexpected result: %d traces, total %d", len(result.Traces), result.Total) + } + + // The backend reached the requested limit, so more traces match than were + // returned and the agent has to be told about it. + result, err = provider.SearchTraces(context.Background(), TraceSearchQuery{Service: "shop-order", Limit: 20}) + if err != nil { + t.Fatal(err) + } + if !result.Truncated { + t.Fatal("expected the result to be marked as truncated") + } + if len(result.Traces) != 20 || result.Total != 20 { + t.Fatalf("unexpected result: %d traces, total %d", len(result.Traces), result.Total) + } +} diff --git a/pkg/mcp/tools/trace/tools.go b/pkg/mcp/tools/trace/tools.go index d59318759..7659e5f32 100644 --- a/pkg/mcp/tools/trace/tools.go +++ b/pkg/mcp/tools/trace/tools.go @@ -18,7 +18,6 @@ package trace import ( - "context" "fmt" "regexp" @@ -44,11 +43,7 @@ func GetTraceByID(ctx consolectx.Context, args map[string]any) (*common.ToolResu if err != nil { return common.ErrorResult(err), nil } - requestCtx := context.Background() - if ctx.AppContext() != nil { - requestCtx = ctx.AppContext() - } - result, err := provider.GetTraceByID(requestCtx, traceID) + result, err := provider.GetTraceByID(traceRequestContext(ctx), traceID) if err != nil { return common.ErrorResult(err), nil } diff --git a/pkg/mcp/tools/trace/tools_test.go b/pkg/mcp/tools/trace/tools_test.go new file mode 100644 index 000000000..0707c649e --- /dev/null +++ b/pkg/mcp/tools/trace/tools_test.go @@ -0,0 +1,231 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package trace + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/apache/dubbo-admin/pkg/config/app" + "github.com/apache/dubbo-admin/pkg/config/observability" + consolectx "github.com/apache/dubbo-admin/pkg/console/context" + "github.com/apache/dubbo-admin/pkg/console/counter" + "github.com/apache/dubbo-admin/pkg/core/lock" + "github.com/apache/dubbo-admin/pkg/core/manager" + "github.com/apache/dubbo-admin/pkg/core/versioning" +) + +type traceToolTestContext struct { + config app.AdminConfig +} + +func (c *traceToolTestContext) RuleVersioning() *versioning.Service { return nil } + +func (c *traceToolTestContext) ResourceManager() manager.ResourceManager { return nil } + +func (c *traceToolTestContext) CounterManager() counter.CounterManager { return nil } + +func (c *traceToolTestContext) Config() app.AdminConfig { return c.config } + +func (c *traceToolTestContext) AppContext() context.Context { return context.Background() } + +func (c *traceToolTestContext) LockManager() lock.Lock { return nil } + +func newTraceToolTestContext(endpoint string) consolectx.Context { + return &traceToolTestContext{ + config: app.AdminConfig{ + Observability: &observability.Config{ + Tracing: &observability.TracingConfig{ + DefaultProvider: "jaeger-main", + Providers: []observability.TraceProviderConfig{{ + Name: "jaeger-main", + Type: observability.TraceProviderJaeger, + Endpoint: endpoint, + }}, + }, + }, + }, + } +} + +const traceToolTestPayload = `{"data":[{"traceID":"` + testSearchTraceID + `","spans":[ + {"traceID":"` + testSearchTraceID + `","spanID":"01","operationName":"checkout","references":[],"startTime":2000000,"duration":100000,"tags":[],"logs":[],"processID":"p1"}, + {"traceID":"` + testSearchTraceID + `","spanID":"02","operationName":"charge","references":[{"refType":"CHILD_OF","traceID":"` + testSearchTraceID + `","spanID":"01"}],"startTime":2010000,"duration":50000,"tags":[{"key":"error","type":"bool","value":true},{"key":"error.type","type":"string","value":"timeout"}],"logs":[],"processID":"p2"} +],"processes":{"p1":{"serviceName":"order-service","tags":[]},"p2":{"serviceName":"payment-service","tags":[]}}}]}` + +func TestSearchTracesToolReturnsSummaries(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(traceToolTestPayload)) + })) + defer server.Close() + + result, err := SearchTraces(newTraceToolTestContext(server.URL), map[string]any{"service": "order-service"}) + if err != nil { + t.Fatal(err) + } + if result.IsError { + t.Fatalf("unexpected error result: %s", result.Content[0].Text) + } + var payload TraceSearchResult + if err := json.Unmarshal([]byte(result.Content[0].Text), &payload); err != nil { + t.Fatal(err) + } + if payload.Total != 1 || len(payload.Traces) != 1 { + t.Fatalf("unexpected payload: %#v", payload) + } + if payload.Traces[0].TraceID != testSearchTraceID || !payload.Traces[0].HasError { + t.Fatalf("unexpected summary: %#v", payload.Traces[0]) + } + if payload.Traces[0].ErrorSpanCount != 1 || payload.Traces[0].SpanCount != 2 { + t.Fatalf("unexpected summary counters: %#v", payload.Traces[0]) + } +} + +func TestSearchTracesToolRejectsMissingService(t *testing.T) { + result, err := SearchTraces(newTraceToolTestContext("http://127.0.0.1:1"), map[string]any{}) + if err != nil { + t.Fatal(err) + } + if !result.IsError { + t.Fatal("expected an error result") + } + if !strings.Contains(result.Content[0].Text, "service is required") { + t.Fatalf("unexpected error message: %s", result.Content[0].Text) + } +} + +func TestSearchTracesToolRequiresConfiguredProvider(t *testing.T) { + result, err := SearchTraces(&traceToolTestContext{}, map[string]any{"service": "order-service"}) + if err != nil { + t.Fatal(err) + } + if !result.IsError { + t.Fatal("expected an error result") + } + if !strings.Contains(result.Content[0].Text, "trace provider is not configured") { + t.Fatalf("unexpected error message: %s", result.Content[0].Text) + } +} + +func TestSearchTracesToolReportsUnknownProvider(t *testing.T) { + result, err := SearchTraces(newTraceToolTestContext("http://127.0.0.1:1"), map[string]any{ + "service": "order-service", + "provider": "missing", + }) + if err != nil { + t.Fatal(err) + } + if !result.IsError { + t.Fatal("expected an error result") + } + if !strings.Contains(result.Content[0].Text, "missing") { + t.Fatalf("unexpected error message: %s", result.Content[0].Text) + } +} + +func TestSearchTracesToolReturnsEmptyResultWithoutError(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(`{"data":[]}`)) + })) + defer server.Close() + + result, err := SearchTraces(newTraceToolTestContext(server.URL), map[string]any{"service": "unknown-service"}) + if err != nil { + t.Fatal(err) + } + if result.IsError { + t.Fatalf("unexpected error result: %s", result.Content[0].Text) + } + var payload TraceSearchResult + if err := json.Unmarshal([]byte(result.Content[0].Text), &payload); err != nil { + t.Fatal(err) + } + if payload.Total != 0 || len(payload.Traces) != 0 { + t.Fatalf("unexpected payload: %#v", payload) + } +} + +func TestAnalyzeTraceToolReturnsDiagnosis(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = w.Write([]byte(traceToolTestPayload)) + })) + defer server.Close() + + result, err := AnalyzeTrace(newTraceToolTestContext(server.URL), map[string]any{"traceId": testSearchTraceID}) + if err != nil { + t.Fatal(err) + } + if result.IsError { + t.Fatalf("unexpected error result: %s", result.Content[0].Text) + } + var payload TraceAnalysis + if err := json.Unmarshal([]byte(result.Content[0].Text), &payload); err != nil { + t.Fatal(err) + } + if payload.TraceID != testSearchTraceID || payload.SpanCount != 2 { + t.Fatalf("unexpected payload: %#v", payload) + } + if len(payload.CriticalPath) != 2 || payload.CriticalPath[1].ServiceName != "payment-service" { + t.Fatalf("unexpected critical path: %#v", payload.CriticalPath) + } + if payload.ErrorSpanCount != 1 || len(payload.ErrorSpans) != 1 { + t.Fatalf("unexpected error spans: %#v", payload.ErrorSpans) + } + if !payload.ErrorSpans[0].RootCause || payload.ErrorSpans[0].ErrorType != "timeout" { + t.Fatalf("unexpected error span: %#v", payload.ErrorSpans[0]) + } + if len(payload.Dependencies) != 1 || payload.Dependencies[0].CallerService != "order-service" { + t.Fatalf("unexpected dependencies: %#v", payload.Dependencies) + } + if payload.Summary == "" { + t.Fatal("expected a summary") + } +} + +func TestAnalyzeTraceToolRejectsInvalidTraceID(t *testing.T) { + result, err := AnalyzeTrace(newTraceToolTestContext("http://127.0.0.1:1"), map[string]any{"traceId": "not-a-trace-id"}) + if err != nil { + t.Fatal(err) + } + if !result.IsError { + t.Fatal("expected an error result") + } + if !strings.Contains(result.Content[0].Text, "traceId must be a 16 or 32 character hexadecimal value") { + t.Fatalf("unexpected error message: %s", result.Content[0].Text) + } +} + +func TestAnalyzeTraceToolReportsUpstreamFailure(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusInternalServerError) + _, _ = w.Write([]byte("upstream exploded")) + })) + defer server.Close() + + result, err := AnalyzeTrace(newTraceToolTestContext(server.URL), map[string]any{"traceId": testSearchTraceID}) + if err != nil { + t.Fatal(err) + } + if !result.IsError { + t.Fatal("expected an error result") + } +}