Skip to content

Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096

Draft
ssss141414 wants to merge 2 commits into
mainfrom
shzhen/add-layoutlm-invoices
Draft

Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096
ssss141414 wants to merge 2 commits into
mainfrom
shzhen/add-layoutlm-invoices

Conversation

@ssss141414

Copy link
Copy Markdown
Contributor

What

Adds Windows ML support for impira/layoutlm-invoices (LayoutLMForQuestionAnswering, task document-question-answering) — the first LayoutLM v1 document-QA recipe in the catalog.

Why it needs code (not just a recipe)

Optimum's TasksManager registers layoutlm only for feature-extraction, fill-mask, text-classification, and token-classification. There is no document-question-answering ONNX config for LayoutLM v1, so winml config -t document-question-answering --model-type layoutlm fails with "layoutlm doesn't support task document-question-answering for the onnx backend."

LayoutLMForQuestionAnswering is an extractive QA head (returns start_logits / end_logits), and layoutlm v1 is reachable only through AutoModelForDocumentQuestionAnswering.

Changes

  1. src/winml/modelkit/models/hf/layoutlm.py (new) — registers a document-question-answering OnnxConfig for layoutlm:
    • Reuses LayoutLMOnnxConfig inputs (input_ids, bbox, attention_mask, token_type_ids).
    • Overrides outputs to {start_logits, end_logits}. Optimum's default document-question-answering output is a single logits tensor, which does not match this extractive head.
    • Applies a RoBERTa-style max_position_embeddings adjustment (this checkpoint uses a RoBERTa vocab: max_position_embeddings=514, pad_token_id=1 → usable 512).
  2. src/winml/modelkit/models/hf/__init__.py — imports the new config to trigger registration (matches the existing # triggers registration pattern).
  3. examples/recipes/impira_layoutlm-invoices/document-question-answering_fp16_config.json (new) — fp16 recipe with hand-tuned, embedding-safe input specs (see below).
  4. examples/recipes/README.md — total 75 → 76, added the row.

Recipe deltas vs. autoconf (why the recipe is required)

Autoconf's raw I/O resolution produces a build that fails; the recipe pins:

  • sequence_length 514 → 512 — raw max_position_embeddings=514 OOBs the position embeddings during export tracing ("index out of range in self").
  • token_type_ids value_range [0,2] → [0,1] — this checkpoint has type_vocab_size=1 (single-row token-type embedding), so token_type_ids must be 0. Value 1 triggers an ONNXRuntime Gather bounds error at /layoutlm/embeddings/token_type_embeddings/Gather.

Validation (CPU only)

  • winml build ... -p fp16 --ep cpu: exit 0 in 97.5s (Export 40.4s / Optimize 45.0s / FP16 11.0s).
  • Genuine fp16: 206 FLOAT16 / 0 FLOAT32 initializers; model.onnx.data 242.6 MB (half of the 485 MB fp32 export).
  • I/O: inputs input_ids, bbox, attention_mask, token_type_ids; outputs start_logits, end_logits [1, 512].
  • winml perf --device cpu --ep cpu --input-data <npz>: P50 280.62 ms, 3.56 samples/sec.
    • Note: winml perf with default random inputs fails because the generator emits token_type_ids=1 (OOB for type_vocab_size=1); a valid .npz (token_type_ids=0, bbox=0) is required.

Scope / not covered

  • Validated only on CPU (MLAS). NPU/GPU EPs (QNN, OpenVINO, VitisAI, DML, NvTensorRTRTX, MIGraphX) are not validated here.
  • No accuracy/eval-dataset run; this is a build + inference-smoke validation.
  • LayoutLMv2 / v3 already have Optimum QA configs and are unaffected.

Follow-ups (not in this PR)

  • winml value_range resolution should clamp token_type_ids to [0, type_vocab_size].
  • winml perf random-input generation should respect embedding-table bounds (or expose per-input range overrides) so degenerate type_vocab_size=1 models don't require a hand-built .npz.

…invoices)

Register a document-question-answering OnnxConfig for layoutlm (Optimum has none for v1) that emits extractive-QA outputs (start_logits/end_logits) and applies the RoBERTa-style position-offset adjustment used by this checkpoint. Add an fp16 recipe with corrected sequence length (512) and embedding-safe input value ranges (token_type_ids pinned to 0 for type_vocab_size=1). Validated fp16 build + CPU inference (P50 280.6ms).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant