Repository navigation

support ds4 - #1355

Open

WANDY666 wants to merge 216 commits into

mainfrom

Commits on Jun 3, 2026

one pass
WANDY666
committed
7c7bd61

Commits on Jun 5, 2026

Optimization
WANDY666
committed
d790ad2
add prompt cache
WANDY666
committed
a161244
support cudagraph
WANDY666
committed
61eed87

Commits on Jun 8, 2026

refact tokenizer
WANDY666
committed
19866d0
add statement
WANDY666
committed
29c6082
format
WANDY666
committed
ffafdbf

Commits on Jun 11, 2026

pass gsm8k but need review
WANDY666
committed
e8009cb
fix
WANDY666
committed
b3b8123
fix rope
WANDY666
committed
6002866
dsv4: enable decode cudagraph; fix warmup-baked FlashMLASchedMeta

authored and
wanzihao
committed
6bc34ad
dsv4: enable prefill cudagraph; zero pad-row attention output
wanzihao
committed
e78e0d4
fix profile
WANDY666
committed
c09dc6a

Commits on Jun 12, 2026

Commits on Jun 14, 2026

compress infer
WANDY666
committed
3a5dcdc
add c128 to mem_manager
WANDY666
committed
d76450f

Commits on Jun 15, 2026

refact
WANDY666
committed
07d2308
opt
WANDY666
committed
d4dcd8a
opt
WANDY666
committed
62c16d5
delete launch.sh
WANDY666
committed
69824d0
fix
WANDY666
committed
df70ecb

Commits on Jun 16, 2026

restore
WANDY666
committed
1ad981d
support parser
WANDY666
committed
7b17bb5
fix
WANDY666
committed
6837abd
Merge branch 'main' of https://github.com/ModelTC/LightLLM into support_ds4
WANDY666
committed
e1376fe

Commits on Jun 18, 2026

add c4 paged indexes
WANDY666
committed
02a24ce
fix chunk_size and page_size
WANDY666
committed
52a1528
add sglang third_party
WANDY666
committed
0dbc90b
fix tpsp
WANDY666
committed
e8c49d1

Commits on Jun 21, 2026

fix profile
WANDY666
committed
88309b5

Commits on Jun 22, 2026

fix swa insufficient
WANDY666
committed
cf433fb
fix
WANDY666
committed
40f5810
rename
WANDY666
committed
f527ca2
tune config
WANDY666
committed
255e90d
prepare opt
WANDY666
committed
d88dc71
delete
WANDY666
committed
a56c79b

Commits on Jun 23, 2026

Commits on Jun 24, 2026

fix parser
WANDY666
committed
51f5c84
fix multi-invoke
WANDY666
committed
2f12a07
speed up prepare
WANDY666
committed
da3fec2
fix arguments
WANDY666
committed
82cb6d6
tune H100
WANDY666
committed
bc22591

Commits on Jun 25, 2026

add encoding_dsv4
WANDY666
committed
77baa05

Commits on Jun 26, 2026

fix c4 error
WANDY666
committed
2225e1a
fuse wq_a+wkv & indexer wkv+wgate GEMMs; fp8 wo_a at tp8 (1 group/rank)
WANDY666
committed
112247b

Commits on Jun 28, 2026

reduce alloc fragment
WANDY666
committed
0d52fde
set _C4_PREFILL_LOGITS_BUDGET_BYTES reduce max memory usage
WANDY666
committed
c711fe9

Commits on Jul 1, 2026

fix(deepseek-v4): align fp8 serving numerics with reference
WANDY666
committed
fced38b

Commits on Jul 2, 2026

fix stirde bug (if-inverse 73)
WANDY666
committed
17a0d54
default topk from huggingface's 50 to -1, if_inverse 70.6 -> 73
WANDY666
committed
c19b853

Commits on Jul 3, 2026

Optimize AI code
WANDY666
committed
bc6ad97
Optimize AI code
WANDY666
committed
99520f9

Commits on Jul 6, 2026

convert list to text
WANDY666
committed
57bf22d
support DS4 cache budgets in recover_paused_reqs
WANDY666
committed
a61d9bc
add error info
WANDY666
committed
7dd651a

Commits on Jul 7, 2026

support mtp
WANDY666
committed
09402d2
Merge branch 'ds4_prod' of https://github.com/ModelTC/LightLLM into support_ds4
WANDY666
committed
a0b258c
fix
WANDY666
committed
1b7e95c

Commits on Jul 8, 2026

v4 don't need ragged_mem_buffers to save mem
WANDY666
committed
730766c
fix error alloc
WANDY666
committed
9f4b534
fix
WANDY666
committed
a3676bb
free infer_state.mtp_draft_input_hiddens
WANDY666
committed
f65915d
fix
WANDY666
committed
6bab228
support dpep
WANDY666
committed
6f03e72

Commits on Jul 9, 2026

draft need only swa
WANDY666
committed
c97c086
speed up
WANDY666
committed
444ca14

Commits on Jul 13, 2026

support overlap
WANDY666
committed
a0acc06
refact
WANDY666
committed
b8c3a63
mega_moe support clamp_limit
WANDY666
committed
1a2f29d
Move DSV4 SWA/c4/c128 slot preparation into the model-specific path.
WANDY666
committed
2408462
autotune
WANDY666
committed
388dd29
delete third_party
WANDY666
committed
6bfd98f

Commits on Jul 14, 2026

optimize flashmla
WANDY666
committed
b39625c
fix 9.11 < 9.9
WANDY666
committed
51194ee
refact
WANDY666
committed
5f17bd4
reuse stable EP buffers to reduce memory fragmentation
WANDY666
committed
dbe0f5e
add test/benchmark/static_inference/static_benchmark.py
WANDY666
committed
4ac53e8

Commits on Jul 15, 2026

static benchmark support multi nodes
WANDY666
committed
0a48e27

Commits on Jul 17, 2026

Commits on Jul 20, 2026

feat: support /v1/responses

sufubao
authored and
WANDY666
committed
18b83bc
fix: align Responses API behavior

sufubao
authored and
WANDY666
committed
0f49762
fix: correct Responses API edge cases

sufubao
authored and
WANDY666
committed
0460344
Merge branch 'codex/pr-1384-merge' into support_ds4
WANDY666
committed
a7e2ef8
use peak SWA demand for paused request recovery
WANDY666
committed
77c8247

Commits on Jul 21, 2026

DeepSeek DSML parsing
WANDY666
committed
5925ca6

Commits on Jul 22, 2026

Deleted some useless
WANDY666
committed
d67f54b
delete metadata
WANDY666
committed
ec11275

Commits on Jul 23, 2026

Optimize DSV4 KV cache write and fix FP8 scaling
WANDY666
committed
99cf2a2
refact metadata
WANDY666
committed
a45e07d
ep delete shared reduced
WANDY666
committed
82c2bfe

Commits on Aug 3, 2026

support cpucache
WANDY666
committed
e726a69

Commits on Aug 4, 2026

dsv4 support micro batch overlap
WANDY666
committed
f81bef3

Commits on Aug 5, 2026

support PD Disaggregation
WANDY666
committed
534c504
support enable_dp_prompt_cache_fetch
WANDY666
committed
9fee60d
fix(deepep): derive NVSHMEM QP depth from decode capacity
WANDY666
committed
97ae2d1

Commits on Aug 6, 2026

Merge branch 'main' of https://github.com/ModelTC/LightLLM into support_ds4
WANDY666
committed
4747840

Commits on Aug 7, 2026

handle consecutive DeepSeek DSML tool-call blocks
WANDY666
committed
4552d58
fix: support UE8M0 FP8 quantization on SM100 for TP
WANDY666
committed
f64e82e

Commits on Aug 10, 2026

Commits on Aug 11, 2026

fix dsv4 dpep2
WANDY666
committed
b46551c
disable c4 aux stream during prefill microbatch overlap
WANDY666
committed
2cbf662
fix pdl cuda bug
WANDY666
committed
eae1968

Commits on Aug 12, 2026

pack need owner gpu
WANDY666
committed
f9e93b9
pd try fix
WANDY666
committed
7958b80

Commits on Aug 13, 2026

async execute self.tokens
WANDY666
committed
a0ea9b2

Commits on Aug 17, 2026

warmup tilelang
WANDY666
committed
fc8b7ec
default thinking high
WANDY666
committed
082c459

Commits on Aug 18, 2026

Commits on Aug 19, 2026

avoid decode-only allocations on prefill nodes
WANDY666
committed
7e24cee
fix: align per-DP request limits with DeepEP capacity
WANDY666
committed
c211f48
Merge https://github.com/ModelTC/LightLLM into support_ds4
WANDY666
committed
57737da

Commits on Aug 20, 2026

fix ttft
WANDY666
committed
bc347a9
feat: imbalance statistics
niushengxiao
committed
a3160f2
add dp cache aware
WANDY666
committed
fbe49a2

Commits on Aug 21, 2026

- check full KV, SWA, C4, and C128 capacity before allocation
WANDY666
committed
085b05c
fix(pd): preserve aborts received before request registration
WANDY666
committed
7c65cd0
Merge branch 'main' of https://github.com/ModelTC/LightLLM into support_ds4
WANDY666
committed
70134e6
fix(dsv4): only protect the last SWA page of active radix matches
WANDY666
committed
34a5a6d
support PreTrainedTokenizerFast
WANDY666
committed
aca684c
fix: fix DeepEP workspace error
niushengxiao
committed
7445e42

Commits on Aug 23, 2026

feat: support disk cache
niushengxiao
committed
4ba4307

Commits on Aug 24, 2026

fix(dsv4): preserve radix-owned SWA prefix when skip prefill
WANDY666
committed
bab7bc6
revert 7445e and just use max(num_rdma_bytes, workspace_size * len(self.groups))
WANDY666
committed
216ec51
set qp_depth minimum 128
WANDY666
committed
69c260f

Commits on Aug 25, 2026

perf(server): reduce PD streaming and parser overhead
WANDY666
committed
baadd7e
perf(openai): reduce chat streaming serialization overhead
WANDY666
committed
095c043
feat(metrics): add PD pipeline latency metrics
WANDY666
committed
5e491fb
allocate model request state per DP
WANDY666
committed
f2e8f14
fix(pd): prevent PD master hang on prefill timeout race
WANDY666
committed
7fad48a
os.getenv(DSV4_SWA_FULL_TOKENS_RATIO)
WANDY666
committed
1e09ad1
fix(pd): recover NIXL transfer task failures (#1474)

sufubao
authored and
WANDY666
committed
7dd1ba0
fix: detach request shared-memory handles on release (#1482)

sufubao
authored and
WANDY666
committed
25c0e57
fix(pd): propagate node generation failures (#1494)
WANDY666
committed
2b2e8e7
fix(dp): serialize DP collectives after overlap forward
WANDY666
committed
d5b04bf

Commits on Aug 26, 2026

Merge branch 'main' of https://github.com/ModelTC/LightLLM into support_ds4
WANDY666
committed
2676145

Commits on Aug 27, 2026

feat: support DeepSeek-V4 DSpark
shihaobai
committed
83b8dfa
feat: add DeepSeek-V4 MTP BF16 conversion support
shihaobai
committed
16ca39b
fix: pad DSpark scratch pages for CUDA graphs
shihaobai
committed
5dfa35d
feat: support DSpark in DP backend
shihaobai
committed
f0e36f3
fix: keep dspark empty DP ranks in sync
shihaobai
committed
4afca8c
delte ==
WANDY666
committed
53bba62
fix: use mtp step for DeepSeek-V4 DSpark width
shihaobai
committed
8c9942a
fix ZMQ crash when cache workers finish concurrently
WANDY666
committed
811230e
perf(dp): move scheduler control to Gloo for decode overlap
WANDY666
committed
94fdfde
perf(dsv4): add five-stream decode overlap
WANDY666
committed
b70bc7a
perf(dsv4): eliminate CUDA list-index sync in decode slot prep
WANDY666
committed
4f60ca5

Commits on Aug 28, 2026

Commits on Sep 1, 2026

use underlying HF tokenizer for xgramma
WANDY666
committed
b8073f8

Commits on Sep 2, 2026

support DeepSeek-V4-Flash-Vision
WANDY666
committed
e365535

Commits on Sep 3, 2026

support fp8xfp8 mega moe
WANDY666
committed
c8f53be

Commits on Sep 4, 2026

fix: account for DSpark SWA scratch pages
niushengxiao
committed
f4f4d68
fix: avoid EAGLE SWA page overcount in DSpark admission
WANDY666
committed
a7f9c36
clean disk cache on shutdown
WANDY666
committed
7229e56
Merge branch 'main' of https://github.com/ModelTC/LightLLM into support_ds4
WANDY666
committed
fff7900

Commits on Sep 7, 2026

synchronize PDL top-k and fail fast on model thread errors
WANDY666
committed
ca4f075
fix(dsv4): invalidate C128 mapping before releasing slots
WANDY666
committed
9fb708f
fix(pd): handle deep prompt-cache keys iteratively
WANDY666
committed
7bc0955
perf(dsv4): avoid redundant FlashMLA prefill output copy
WANDY666
committed
02aede0
fix(pd): measure decode KV timeout from request progress
WANDY666
committed
8ed3da6
fix(api): return 400 for oversized guided JSON schemas
WANDY666
committed
aa9b2a3
fix(pd): wait for KV transfer modules to become ready
WANDY666
committed
ef0df43
improve intra-node DP load balancing for PD serving
WANDY666
committed
7d5a94a

Commits on Sep 8, 2026

Commits on Sep 9, 2026

feat: add EPLB
niushengxiao
committed
7722b82
feat: support EPLB for DeepSeek v4
niushengxiao
committed
33cd0c9
perf(deepseek_v4): enable fused SDPA for vision attention
WANDY666
committed
2ce90e7
perf(vision): use shared ViT attention backends for DeepSeek V4
WANDY666
committed
924c62e

Commits on Sep 10, 2026

fix(pd): reserve transfer buffer during memory profiling (#1543)

shihaobai
authored and
WANDY666
committed
32b5797

Commits on Sep 11, 2026

fix(dsv4): preallocate FlashMLA and C4 prefill workspaces
WANDY666
committed
02ad09e
add DISK_CACHE_NAME
WANDY666
committed
62907ac

Commits on Sep 14, 2026

Structured output is not supported
WANDY666
committed
f14dd40
PD nodes unavailable: 503
WANDY666
committed
2a7e6bd
fix(pd): harden control sends and reconnect cleanup
WANDY666
committed
33da96f

Commits on Sep 15, 2026

fix(pd): clean up requests correctly after node disconnect
WANDY666
committed
9a3f99d
unit test
WANDY666
committed
54c5730
add convert_deepseek_v4_scales_to_ue8m0_fp32
WANDY666
committed
2f37b95
align B300
WANDY666
committed
f5f3ed2

Commits on Sep 16, 2026

fix has_nvlink
WANDY666
committed
7cc1027
add triton ep backend
WANDY666
committed
c899b31
fix abort
WANDY666
committed
9bb68eb

Commits on Sep 17, 2026

fix(api): ignore structured output constraints when xgrammar is disabled
WANDY666
committed
8ee409a

Commits on Sep 18, 2026

fix(stream): emit keepalive deltas for long DSML tool arguments
WANDY666
committed
2bab226
add LIGHTLLM_DSV4_THINKING_EFFORT
WANDY666
committed
2483650

Commits on Sep 20, 2026

support tp and tppd
WANDY666
committed
21763ad
perf(dspark): merge context WKV projections across draft stages
WANDY666
committed
8ce69cc
fix(dsv4): fall back to FlashMLA when workspace op is unavailable
WANDY666
committed
8415357
Merge main 317e70b6 into support_ds4
WANDY666
committed
cf74723
Unify DeepSeek-V4 compressed history on 256-token pages
WANDY666
committed
16176f8
Integrate DeepSeek-V4 private continuation with hybrid checkpoints
WANDY666
committed
38847af
Use V4 SWA slots for DSpark draft KV writes
WANDY666
committed
889abf5
Initialize DSpark commit SWA metadata in decode
WANDY666
committed
7ab3a66
Allow read-only hybrid radix prefix probes
WANDY666
committed
71d2ae9

Commits on Sep 21, 2026

Reuse GPU staging for V4 DP checkpoint transfers
WANDY666
committed
00fe650
Use shared NCCL group for V4 checkpoint P2P
WANDY666
committed
6ca7779
Transfer V4 DP checkpoints through CPU process group
WANDY666
committed
1fda8b7

Commits on Sep 22, 2026

Commits on Sep 24, 2026

revert: trim non-model support changes
WANDY666
committed
a4a4c05