Skip to content

[Feature] Support GLM-5.2 source-layer DSA indexer training - #2022

Open
sjyouuuuug wants to merge 2 commits into
InternLM:mainfrom
sjyouuuuug:feat/glm52-indexer-training
Open

[Feature] Support GLM-5.2 source-layer DSA indexer training#2022
sjyouuuuug wants to merge 2 commits into
InternLM:mainfrom
sjyouuuuug:feat/glm52-indexer-training

Conversation

@sjyouuuuug

Copy link
Copy Markdown

Motivation

Enable training of GLM-5.2 source-layer DSA indexers with attention-derived KL supervision.

Changes

  • Add a cuDNN DSA indexer KL-loss operator and custom backward integration.
  • Separate differentiable indexer feature projection from no-grad Top-k selection.
  • Construct sparse attention teacher and indexer student distributions.
  • Aggregate source-layer indexer losses at the model boundary.
  • Keep physical MTP indexers frozen.
  • Add a GLM-5.2 training configuration and launch script.
  • Add tests for KL construction, gradient boundaries, score scaling, query masking, source/shared-layer behavior, and MTP freezing.

Validation

  • Indexer loss CPU tests: 10 passed.
  • Source-indexer attention tests: 10 passed.
  • Multi-GPU tiny training completed all configured training steps.
  • Indexer loss decreased during the 300-step training run.

Current limitations

  • IndexShare reuses source-layer Top-k indices, but shared-layer teacher supervision is not included.
  • Indexer training currently requires:
    • SP_SIZE=1
    • INTRA_LAYER_MICRO_BATCH=1
    • the cudnn_dsa backend

Results

5cad86b0b8795b336b4f04cfd95114c image

@windreamer
windreamer requested a review from jayhenry August 14, 2026 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant