[Cherry-Pick][Benchmark] Add inner benchmark metrics component (#7881)#7831
Conversation
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览当前 PR 的 Required CI 尚未全部通过：Required 任务 10 个，已通过 9 个，失败 1 个。阻塞项为
2 任务状态汇总日志列说明：失败任务直接使用日志链接；运行中任务使用 Job 链接。 2.1 Required任务 : 9/10 通过
2.2 可选任务 — 23/27 通过
3 失败详情（仅 required）Approval — 需要人工审批（置信度: 高）该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。
|
|
Thanks for your contribution! |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release/2.6 #7831 +/- ##
==============================================
Coverage ? 72.26%
==============================================
Files ? 382
Lines ? 54419
Branches ? 8513
==============================================
Hits ? 39328
Misses ? 12309
Partials ? 2782
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Deleter-D
force-pushed
the
2.6_inner_benchmark
branch
from
2b635b8 to
f821677
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Deleter-D
force-pushed
the
2.6_inner_benchmark
branch
from
951188f to
8bfb67e
Compare
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-20 19:39:57
📋 Review 摘要
PR 概述：新增进程内 Benchmark Metrics Logger，通过后台线程采集每请求延迟指标，支持滑动/翻滚窗口聚合统计并写入 JSONL 文件
变更范围：fastdeploy/config.py、fastdeploy/metrics/、fastdeploy/output/token_processor.py、fastdeploy/engine/、docs/
影响面 Tag：[FDConfig] [Engine] [Benchmark] [DataProcessor] [Docs]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🔴 Bug | fastdeploy/config.py:2461 |
assert 用于运行时配置校验，python -O 下会被静默跳过 |
| 🟡 建议 | fastdeploy/metrics/benchmark_metrics_logger.py:116 |
datetime.now() 无时区，分布式/跨时区环境时间戳语义模糊 |
| 📝 PR 规范 | — | 标题缺少官方 Tag，目标分支为 release/2.6 须遵循 Cherry-Pick 格式；描述各段落为空 |
📝 PR 规范检查
PR 标题 Add inner benchmark metrics component 缺少官方 Tag；目标分支为 release/2.6（非 develop），按规范须采用 Cherry-Pick 格式并附原 PR 号；描述中 Motivation / Modifications / Usage or Command / Accuracy Tests 各段均为空，Checklist 未勾选。
标题建议（可直接复制）：
[Cherry-Pick][Benchmark] Add inner benchmark metrics component (#<develop_PR_号>)
PR 描述建议（可直接复制，必须复刻 checklist §D2 模板的完整结构）：
## Motivation
为 FastDeploy 添加进程内性能监控模块（Benchmark Metrics Logger），在推理进程内部实时采集每请求延迟数据，以滑动窗口或翻滚窗口模式计算聚合统计指标（TTFT、TPOT、E2EL、吞吐量等），写入 JSONL 文件供实时监控与事后分析，无需依赖外部 benchmark 工具。
## Modifications
- `fastdeploy/config.py`：新增 `BenchmarkMetricsConfig` 配置类；在 `FDConfig` 中添加 `benchmark_metrics_config` 字段，并在 `FDConfig.check()` 中补充参数合法性校验
- `fastdeploy/engine/args_utils.py`：新增 `--benchmark-metrics-config` CLI 参数及 `create_benchmark_metrics_config()` 方法
- `fastdeploy/engine/common_engine.py`：引擎初始化时按配置实例化 `BenchmarkMetricsLogger` 并注入 `token_processor`
- `fastdeploy/metrics/benchmark_metrics_logger.py`：新增 `BenchmarkMetricsLogger`，后台 daemon 线程 + deque 窗口，支持滑动/翻滚两种模式
- `fastdeploy/output/token_processor.py`：在 `_record_metrics` 中收集 ITL 样本，在 `_record_completion_metrics` 中组装 `CompletedRequestRecord` 并回调 logger
- `docs/benchmark.md` / `docs/zh/benchmark.md`：新增进程内监控模块中英文文档
## Usage or Command
python -m fastdeploy.entrypoints.openai.api_server \
--model baidu/ERNIE-4.5-0.3B-Base-Paddle \
--benchmark-metrics-config '{"enable": true, "window_size": 64, "window_mode": "sliding"}'
查看最新统计快照：tail -1 log/benchmark_metrics.jsonl | python -m json.tool
## Accuracy Tests
N/A（本 PR 为性能监控功能，不涉及模型前向推理逻辑变更，不影响模型输出精度）
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [x] Add unit tests. Please write the reason in this PR if no unit tests.
- [x] Provide accuracy results.
- [ ] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.总体评价
功能实现完整、逻辑清晰、单测覆盖充分。需修复 FDConfig.check() 中 assert 校验在 python -O 模式下失效的风险，并完善 PR 标题格式与描述内容。
|
|
||
| if self.benchmark_metrics_config is not None: | ||
| cfg = self.benchmark_metrics_config | ||
| assert isinstance( |
There was a problem hiding this comment.
🔴 Bug assert 被用于运行时配置校验，在 python -O（优化模式）下所有 assert 语句会被静默忽略，导致非法配置值绕过校验进入推理流程。
建议将此处所有 assert 替换为显式 raise ValueError：
if not isinstance(cfg.enable, bool):
raise ValueError(f"BenchmarkMetricsConfig: 'enable' must be a bool, got {type(cfg.enable).__name__}")
if not (isinstance(cfg.window_size, int) and cfg.window_size >= 0):
raise ValueError(f"BenchmarkMetricsConfig: 'window_size' must be a non-negative integer, got {cfg.window_size!r}")
if cfg.window_mode not in ("sliding", "tumbling"):
raise ValueError(f"BenchmarkMetricsConfig: 'window_mode' must be 'sliding' or 'tumbling', got {cfg.window_mode!r}")
# ... 其余校验同理| records = list(self._window) | ||
| n = len(records) | ||
| if n == 0: | ||
| return {"timestamp": datetime.now().isoformat(), "completed": 0} |
There was a problem hiding this comment.
🟡 建议 使用 datetime.now() 输出本地时间，在多节点/跨时区分布式环境中时间戳含义不明确，建议改用 UTC：
from datetime import timezone
# 将两处 datetime.now().isoformat() 替换为：
datetime.now(timezone.utc).isoformat()两处均需修改：n==0 提前返回处（第 116 行）和 result 字典构造处（第 164 行）。
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览当前 Required 任务 10 个，通过 9 个，失败 1 个；失败项为
2 任务状态汇总日志列说明：失败任务直接使用预生成日志链接；运行中任务使用 Job 链接。 2.1 Required任务 : 9/10 通过
2.2 可选任务 — 23/27 通过
3 失败详情（仅 required）Approval — 需要审批（置信度: 高）该 Job 需要人工 Approval，完成审批后 CI 才会继续执行。
|
Deleter-D
changed the title
Jiang-Jia-Jun
merged commit e7815be
into
PaddlePaddle:release/2.6
5 participants
Motivation
为 FastDeploy 添加进程内性能监控模块（Benchmark Metrics Logger），在推理进程内部实时采集每请求延迟数据，以滑动窗口或翻滚窗口模式计算聚合统计指标（TTFT、TPOT、E2EL、吞吐量等），写入 JSONL 文件供实时监控与事后分析，无需依赖外部 benchmark 工具。
Modifications
fastdeploy/config.py：新增BenchmarkMetricsConfig配置类；在FDConfig中添加benchmark_metrics_config字段，并在FDConfig.check()中补充参数合法性校验fastdeploy/engine/args_utils.py：新增--benchmark-metrics-configCLI 参数及create_benchmark_metrics_config()方法fastdeploy/engine/common_engine.py：引擎初始化时按配置实例化BenchmarkMetricsLogger并注入token_processorfastdeploy/metrics/benchmark_metrics_logger.py：新增BenchmarkMetricsLogger，后台 daemon 线程 + deque 窗口，支持滑动/翻滚两种模式fastdeploy/output/token_processor.py：在_record_metrics中收集 ITL 样本，在_record_completion_metrics中组装CompletedRequestRecord并回调 loggerdocs/benchmark.md/docs/zh/benchmark.md：新增进程内监控模块中英文文档Usage or Command
python -m fastdeploy.entrypoints.openai.api_server
--model baidu/ERNIE-4.5-0.3B-Base-Paddle
--benchmark-metrics-config '{"enable": true, "window_size": 64, "window_mode": "sliding"}'
查看最新统计快照：tail -1 log/benchmark_metrics.jsonl | python -m json.tool
Accuracy Tests
N/A（本 PR 为性能监控功能，不涉及模型前向推理逻辑变更，不影响模型输出精度）
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.