Conversation
The ZVL256B GEMV-N kernel for unit y stride walked one column of A at a time. For each column it loaded y, loaded the column, issued one vfmacc, stored y back, and then repeated the whole sequence for the next column. Every column therefore paid a y load and a y store round-trip, and because VL was recomputed from the remaining row count on each pass, the vsetvli and the pointer bumps also sat inside the hot loop. Process GEMV_N_BLOCK (4) consecutive columns per pass over y. A single y slice stays resident in vector registers while all four columns are accumulated into it, so the y round-trip is paid once per block instead of once per column, and the main loop's VL is pinned to the hardware maximum, which lets the compiler hoist the vsetvli and strength-reduce the pointer arithmetic out of the loop. The columns are still consumed in ascending j and the four contributions are folded into the accumulator in that same order, so each y[i] sees the same sequence of addends as the original column-at-a-time loop; the intermediate y vector merely stays in a register instead of being stored and reloaded. The row remainder of each block keeps a variable VL, and the columns left over after the last full block are finished by the original two-column and single-column loops, unchanged. The result is bit-identical. A differential run of the baseline and patched kernels against each other, and of both against a double-precision reference, over 21375 single-precision and 21375 double-precision cases covering m = 1..129, n = 1..33, several leading dimensions and x increments, and alpha values including 0, with guard pages around every buffer, shows no mismatch and no out-of-bounds access. The same sweep for a non-unit y stride, which the patch leaves untouched, is likewise bit-identical. The library built from the patched kernel passes 125/125 utests and 1473/1473 extension tests. Measured on one X100 core at 2.2 GHz, warm-up once and the median of three runs per size: sgemv 256 -1.15%, 512 +30.01%, 1024 +13.57%; dgemv 256 +31.04%, 512 +22.14%, 1024 +2.40%. Co-authored-by: Yuansheng <yuansheng@isrc.iscas.ac.cn> Co-authored-by: Ning Tian <tianning24@iscas.ac.cn> Signed-off-by: jiakai xu <xujiakai2025@iscas.ac.cn>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Process four consecutive columns per pass over
yin the ZVL256B GEMV-N kernelfor unit
ystride.The kernel previously walked one column of
Aat a time. For each column itloaded
y, loaded the column, issued onevfmacc, storedyback, and thenrepeated the whole sequence for the next column. Every column therefore paid a
yload and aystore round-trip, and because VL was recomputed from theremaining row count on each pass, the
vsetvliand the pointer bumps also satinside the hot loop.
GEMV_N_BLOCK(4) consecutive columns are now accumulated into a singleyslice that stays resident in vector registers, so the
yround-trip is paidonce per block instead of once per column, and the main loop's VL is pinned to
the hardware maximum, which lets the compiler hoist the
vsetvliandstrength-reduce the pointer arithmetic out of the loop.
The columns are still consumed in ascending
jand the four contributions arefolded into the accumulator in that same order, so each
y[i]sees the samesequence of addends as the original column-at-a-time loop; the intermediate
yvector merely stays in a register instead of being stored and reloaded. The row
remainder of each block keeps a variable VL, and the columns left over after the
last full block are finished by the two-column and single-column loops. The
result is bit-identical.
The non-unit
ystride path is untouched.This backs both SGEMV_N and DGEMV_N.
Performance
benchmark/sgemv.gotoandbenchmark/dgemv.goto, one X100 core at 2.2 GHz,single thread, warm-up once and the median of three runs per size:
Testing
make testsreturns 0: 125/125 utests, 1473/1473 extension tests, no newCBLAS failure.
single-precision, 21375 double-precision and 21375 non-unit-
y-stride cases,covering
m = 1..129,n = 1..33, several leading dimensions andxincrements, and alpha values including 0, produce output bit-identical to the
unpatched kernel, with no out-of-bounds access. Both builds were also checked
against a double-precision reference.
inc_y != 1branch was diffed against the baseline and is byte-identical.sgemv_entry point on both thebaseline and patched libraries produces identical results.