Conversation
The ZVL256B SGEMM main pass advanced one k-step per iteration: each
iteration loaded eight B scalars and one 16-lane A vector, issued eight
accumulating FMAs, then bumped the B and A pointers and took the loop
backedge. On the tall-and-narrow shapes this kernel is used on, that
per-k loop control is a measurable fraction of the iteration.
Fold two consecutive k-steps into one iteration. Both A vectors are
loaded before the FMA stream (A double-buffering), so the A-load latency
is exposed earlier, and the k counter, one of the two B/A pointer bumps
and the backedge are halved. Each accumulator still consumes the two
k-steps of a pair in ascending order, so the per-column accumulation
order is unchanged.
The peeled first k-step (the initial vfmul, ahead of the loop) is left
alone, so the restructured main pass reads
for (k = K; k > 2; k -= 2) { ... two k-steps ... }
for (; --k; ) { ... one k-step ... }
which consumes exactly K steps for every K: the unrolled loop runs
floor((K-1)/2) times and the trailing loop takes the remainder. The
unrolled loop is skipped entirely when K <= 2, in which case the trailing
loop alone (or, for K = 1, the peeled step alone) finishes the pass.
Because the accumulation order itself is preserved rather than merely
re-summed, the emitted results are bit-identical to the previous kernel.
Measured on one X100 core at 2.2 GHz, cblas_sgemm, warm-up once and the
median of three runs per size: 256x256 +5.13%, 512x512 +5.56%,
1024x1024 +3.43%.
In a default RISCV64_ZVL256B build (USE_TRMM=1) this source is compiled once,
for SGEMM; STRMM is built from its own strmm_kernel_16x8_zvl256b.c. The
Makefile's USE_TRMM=0 branch instead reuses this source for the four STRMM
variants, so the patched file was also compiled in those four -DTRMMKERNEL
configurations: all build with a warning set unchanged from the unpatched
tree. Verified further by a differential run against a double-precision
reference: 360 packed-kernel invocations and 20400 cblas_sgemm cases,
covering the M/N/K tail and edge paths, all four transpose combinations and
several alpha/beta values, produce output bit-identical to the unpatched
kernel.
Co-authored-by: Yuansheng <yuansheng@isrc.iscas.ac.cn>
Co-authored-by: Ning Tian <tianning24@iscas.ac.cn>
Signed-off-by: jiakai xu <xujiakai2025@iscas.ac.cn>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Unroll the K-loop of the ZVL256B SGEMM micro-kernel by two, with the two A
vectors loaded before the FMA stream (A double-buffering).
The main pass previously advanced one k-step per iteration: eight B scalar
loads, one 16-lane A load, eight accumulating FMAs, then the B/A pointer
bumps and the loop backedge. Folding two k-steps into one iteration halves
the k counter, one of the two pointer bumps and the backedge, and issues both
A loads up front so the A-load latency is exposed earlier.
The first k-step is already peeled ahead of the loop (the
vfmulcalls thatinitialize the accumulators), so the restructured main pass is
which consumes exactly K steps for every K. Each accumulator still consumes
the k-steps in ascending order, so the per-column accumulation order is
unchanged and the emitted results are bit-identical to the previous kernel.
Performance
benchmark/cblas_sgemm.goto, one X100 core at 2.2 GHz, single thread,warm-up once and the median of three runs per size:
Testing
make testsreturns 0: 125/125 utests, 1473/1473 extension tests, no newCBLAS failure.
USE_TRMM=1) the source is compiledonce, for SGEMM (STRMM is built from its own
strmm_kernel_16x8_zvl256b.c).The Makefile's
USE_TRMM=0branch instead reuses this source for the fourSTRMM variants, so the patched file was compiled in those four
-DTRMMKERNELconfigurations too; all build with a warning set unchangedfrom the unpatched tree.
packed-kernel invocations plus 20400
cblas_sgemmcases, covering the M/N/Ktail and edge paths, all four transpose combinations and several alpha/beta
values, produce output bit-identical to the unpatched kernel.