Skip to content

feat(dashboard): add cache and checkpoint observability - #445

Merged
liunyl merged 1 commit into
mainfrom
codex/cache-checkpoint-observability
Aug 30, 2026
Merged

feat(dashboard): add cache and checkpoint observability#445
liunyl merged 1 commit into
mainfrom
codex/cache-checkpoint-observability

Conversation

@liunyl

@liunyl liunyl commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Context

Companion dashboard delivery for tx_service #561. EloqKV operators need node-level visibility into resident/dirty key populations and checkpoint scheduling, durable progress, and failures.

Changes

Updates both EloqKV overview variants with a collapsed six-panel Cache and Checkpoint Observability row:

  • resident and dirty key populations by node;
  • dirty-key ratio as a ratio of summed populations;
  • checkpoint attempt interval average and p95;
  • checkpoint durable-advance interval average and p95;
  • cumulative checkpoint failures by node and reason;
  • the existing consecutive-failure alert signal as a node-level panel.

The existing is_continuous_checkpoint_failures > 0 alert rule is preserved.

Verification

jq empty <both dashboard files>
  passed
unique panel-ID check
  passed for both files
new-row panel count check
  passed: exactly six panels in each file
PromQL inspection
  passed: node_id aggregation, histogram le grouping, guarded denominators
live Grafana import + three-node Prometheus scrape
  passed

Manual validation confirmed dirty counts rise after writes and return to zero after checkpoint while resident counts remain, and checkpoint attempt/advance counters continue increasing on a synchronized standby.

Dependency and rollout

Requires the metrics added by tx_service #561. Deploy the tx_service change before relying on these panels; older servers will show no data for the new series. Rollback is a revert of this PR.

@liunyl
liunyl merged commit 1f8b4b3 into main Aug 30, 2026
2 of 3 checks passed
@liunyl
liunyl deleted the codex/cache-checkpoint-observability branch August 30, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants