StructReader::partition_expr doesn't hold lock if the expression is being newly partitioned#8892
Conversation
…eing newly partitioned Signed-off-by: Robert Kruszewski <github@robertk.io>
|
Not sure if this is better, we can redundantly recompute partitioning at the beginning but lookups no longer take a write lock |
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
Benchmarks: Vortex queries 📖Verdict: No clear signal (medium confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.862x ✅, 2↑ 0↓)
datafusion / parquet (0.951x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.998x ➖, 0↑ 0↓)
duckdb / parquet (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Signed-off-by: Robert Kruszewski <github@robertk.io>
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.031x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.031x ➖, 1↑ 4↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.818x ✅, 21↑ 0↓)
datafusion / vortex-compact (0.890x ✅, 11↑ 0↓)
datafusion / parquet (0.857x ✅, 15↑ 0↓)
datafusion / arrow (0.856x ✅, 16↑ 1↓)
duckdb / vortex-file-compressed (0.991x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.997x ➖, 0↑ 0↓)
duckdb / parquet (0.962x ➖, 3↑ 0↓)
duckdb / duckdb (0.987x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.959x ➖, 1↑ 0↓)
datafusion / vortex-compact (0.966x ➖, 1↑ 0↓)
datafusion / parquet (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.009x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.002x ➖, 0↑ 0↓)
duckdb / parquet (1.019x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.939x ➖, 10↑ 0↓)
datafusion / vortex-compact (0.961x ➖, 2↑ 0↓)
datafusion / parquet (0.952x ➖, 8↑ 1↓)
duckdb / vortex-file-compressed (0.948x ➖, 12↑ 0↓)
duckdb / vortex-compact (0.966x ➖, 3↑ 0↓)
duckdb / parquet (0.969x ➖, 2↑ 1↓)
duckdb / duckdb (0.950x ➖, 7↑ 0↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (0.970x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.974x ➖, 0↑ 0↓)
duckdb / parquet (0.976x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.021x ➖, 0↑ 1↓)
datafusion / vortex-compact (0.943x ➖, 1↑ 0↓)
datafusion / parquet (1.031x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.979x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.957x ➖, 0↑ 0↓)
duckdb / parquet (1.001x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.970x ➖, 2↑ 0↓)
datafusion / parquet (1.022x ➖, 0↑ 2↓)
duckdb / vortex-file-compressed (1.071x ➖, 0↑ 2↓)
duckdb / parquet (1.014x ➖, 0↑ 0↓)
duckdb / duckdb (1.013x ➖, 0↑ 0↓)
File Size Changes (201 files changed, +0.0% overall, 109↑ 92↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.945x ➖, 4↑ 0↓)
datafusion / vortex-compact (0.944x ➖, 4↑ 0↓)
datafusion / parquet (0.972x ➖, 2↑ 0↓)
datafusion / arrow (0.958x ➖, 5↑ 0↓)
duckdb / vortex-file-compressed (0.971x ➖, 2↑ 0↓)
duckdb / vortex-compact (1.002x ➖, 0↑ 0↓)
duckdb / parquet (0.989x ➖, 0↑ 0↓)
duckdb / duckdb (0.953x ➖, 5↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.929x ➖, 7↑ 0↓)
datafusion / parquet (0.946x ➖, 6↑ 1↓)
duckdb / vortex-file-compressed (0.959x ➖, 4↑ 1↓)
duckdb / parquet (0.969x ➖, 1↑ 0↓)
duckdb / duckdb (0.963x ➖, 1↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.009x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.005x ➖, 0↑ 0↓)
datafusion / parquet (1.052x ➖, 1↑ 3↓)
duckdb / vortex-file-compressed (0.957x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.973x ➖, 0↑ 0↓)
duckdb / parquet (0.983x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Vortex (geomean): 1.070x ➖ How to read Verdict and Engines
unknown / unknown (1.046x ➖, 0↑ 8↓)
|
Benchmarks: Appian on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.927x ➖, 2↑ 0↓)
datafusion / parquet (0.925x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (0.930x ➖, 1↑ 0↓)
duckdb / parquet (0.933x ➖, 1↑ 0↓)
duckdb / duckdb (0.936x ➖, 1↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=10 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.026x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.042x ➖, 0↑ 0↓)
datafusion / parquet (1.003x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (1.023x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.033x ➖, 0↑ 0↓)
duckdb / parquet (1.032x ➖, 0↑ 0↓)
|
Benchmarks: Compression 📖Vortex (geomean): 1.011x ➖ How to read Verdict and Engines
unknown / unknown (1.048x ➖, 1↑ 30↓)
|
We also simplify ExactExpr hash behaviour to match the equality semantics and
avoid deep tree hashing