fix: correct LoRA initialization and forward pass under tensor parallelism#150

Merged

kilinchange merged 7 commits into

Jul 23, 2026

Commits on Jul 22, 2026

refactor: fuse base+LoRA matmuls before collective to fix loss divergence
chen2021673
committed
fix: broadcast lora_A init from TP rank 0 to ensure consistent replicated weights
chen2021673
committed
feat: add rank-aware Broadcast/Scatter to ProcessGroup
chen2021673
committed
fix(lora): initialize LoRA A with TP collectives
chen2021673
committed
feat: export portable full tensors when saving LoRA weights under TP
chen2021673
committed
fix(lora): handle DDP module prefix for adapter weights
chen2021673
committed
refactor: reuse tensor-parallel shard gather logic
chen2021673
committed