perf(tdigest): optimize partial aggregate compute path

[tisonkun]

Summary

• validate complete native T-Digest payloads once, then decode fixed-width centroid and buffered-value records directly from borrowed slices
• linearly merge fully compressed, sorted centroid buffers instead of invoking a general stable sort; inputs are checked for ordering and all other shapes retain the existing merge path
• preserve the public API, wire format, stable ordering for equal means, serialized output bytes, and reference-compatible deserialization

The benchmarks model a native final-aggregate workload: deserialize 64 partial states, merge them sequentially, and query one quantile. Both changes apply without caller changes. The partial update/serialize path remains unchanged; a batch-update prototype improved that full lifecycle by only about 2% at 64 rows/state and regressed the 8-row case slightly, so it was intentionally left out.

Performance

Divan A/B runs used at least two seconds and 300 samples for the merge workloads. Times are medians from the same machine. The compute rows isolate the merge change against the branch with native direct decoding already applied; the deserialize rows compare this PR with main.

workload	before -> after	change
compute: 64 native states, 8 rows/state	75.20 us -> 30.04 us	60.1% faster
compute: 64 native states, 64 rows/state	108.6 us -> 63.46 us	41.6% faster
compute: 64 overlapping native states, 64 rows/state	122.0 us -> 73.09 us	40.1% faster
deserialize: 512 native states, 8 centroids/state	21.83 us -> 19.78 us	9.4% faster
deserialize: 512 native states, 64 centroids/state	121.3 us -> 106.6 us	12.1% faster

For the 64-row compute workload, allocations fall from 117 / 300.1 KB to 65 / 66.56 KB. These local microbenchmarks are directional rather than a performance contract.

Compatibility

• the public API and serialized format are unchanged
• the fast merge path preserves the old stable ordering for equal centroid means; a regression snapshot confirms byte-for-byte identical merged output
• unsorted, buffered, or otherwise uncompressed merge inputs continue through the original sort-and-compress path
• C++, Java, Go, and reference-compatible deserialization retain their existing implementation and behavior
• native value, weight, overflow, and payload-length validation is unchanged

Validation

• cargo x check
• cargo x test
• cargo x lint
• cargo x bench

[Copilot]

Pull request overview

Optimizes the T-Digest “partial aggregate” hot path by (1) decoding validated native payloads directly from borrowed slices and (2) adding a linear-time merge for already-sorted, fully-compressed centroid buffers—while keeping the public API and serialized bytes stable.

Changes:

• Add a fast merge path that linearly merges sorted centroid buffers (preserving stable ordering on equal means).
• Optimize deserialization by validating the full payload once and decoding fixed-width centroid/buffered records from slices (avoiding per-field cursor I/O).
• Add/extend regression coverage and benchmarks for serialized partial-state workloads; update changelog entry.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
tests-integration/tests/serde_tests/tdigest.rs	Adds a byte-stability regression case covering merge after deserialize/serialize.
datasketches/src/tdigest/sketch.rs	Implements sorted-centroid linear merge path and slice-based native payload decoding.
CHANGELOG.md	Expands the Unreleased performance note to mention the new merge and decode optimizations.
benchmarks/tdigest/merge.rs	Adds serialized-partials merge benchmarks to reflect the optimized workload.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[ariesdevil]

LGTM~

[tisonkun]

Thanks for your review @ariesdevil!

I'm working on a refactor now. I noticed that we may have an opportunity to refactor the compression + merge at once.