Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096
Draft
ssss141414 wants to merge 2 commits into
Draft
Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096ssss141414 wants to merge 2 commits into
ssss141414 wants to merge 2 commits into
Conversation
…invoices) Register a document-question-answering OnnxConfig for layoutlm (Optimum has none for v1) that emits extractive-QA outputs (start_logits/end_logits) and applies the RoBERTa-style position-offset adjustment used by this checkpoint. Add an fp16 recipe with corrected sequence length (512) and embedding-safe input value ranges (token_type_ids pinned to 0 for type_vocab_size=1). Validated fp16 build + CPU inference (P50 280.6ms).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Windows ML support for
impira/layoutlm-invoices(LayoutLMForQuestionAnswering, taskdocument-question-answering) — the first LayoutLM v1 document-QA recipe in the catalog.Why it needs code (not just a recipe)
Optimum's
TasksManagerregisterslayoutlmonly forfeature-extraction,fill-mask,text-classification, andtoken-classification. There is nodocument-question-answeringONNX config for LayoutLM v1, sowinml config -t document-question-answering --model-type layoutlmfails with "layoutlm doesn't support task document-question-answering for the onnx backend."LayoutLMForQuestionAnsweringis an extractive QA head (returnsstart_logits/end_logits), and layoutlm v1 is reachable only throughAutoModelForDocumentQuestionAnswering.Changes
src/winml/modelkit/models/hf/layoutlm.py(new) — registers adocument-question-answeringOnnxConfig forlayoutlm:LayoutLMOnnxConfiginputs (input_ids,bbox,attention_mask,token_type_ids).outputsto{start_logits, end_logits}. Optimum's defaultdocument-question-answeringoutput is a singlelogitstensor, which does not match this extractive head.max_position_embeddingsadjustment (this checkpoint uses a RoBERTa vocab:max_position_embeddings=514,pad_token_id=1→ usable512).src/winml/modelkit/models/hf/__init__.py— imports the new config to trigger registration (matches the existing# triggers registrationpattern).examples/recipes/impira_layoutlm-invoices/document-question-answering_fp16_config.json(new) — fp16 recipe with hand-tuned, embedding-safe input specs (see below).examples/recipes/README.md— total75 → 76, added the row.Recipe deltas vs. autoconf (why the recipe is required)
Autoconf's raw I/O resolution produces a build that fails; the recipe pins:
sequence_length514 → 512 — rawmax_position_embeddings=514OOBs the position embeddings during export tracing ("index out of range in self").token_type_idsvalue_range[0,2] → [0,1]— this checkpoint hastype_vocab_size=1(single-row token-type embedding), sotoken_type_idsmust be0. Value1triggers an ONNXRuntime Gather bounds error at/layoutlm/embeddings/token_type_embeddings/Gather.Validation (CPU only)
winml build ... -p fp16 --ep cpu: exit 0 in 97.5s (Export 40.4s / Optimize 45.0s / FP16 11.0s).model.onnx.data242.6 MB (half of the 485 MB fp32 export).input_ids,bbox,attention_mask,token_type_ids; outputsstart_logits,end_logits[1, 512].winml perf --device cpu --ep cpu --input-data <npz>: P50 280.62 ms, 3.56 samples/sec.winml perfwith default random inputs fails because the generator emitstoken_type_ids=1(OOB fortype_vocab_size=1); a valid.npz(token_type_ids=0,bbox=0) is required.Scope / not covered
Follow-ups (not in this PR)
winmlvalue_range resolution should clamptoken_type_idsto[0, type_vocab_size].winml perfrandom-input generation should respect embedding-table bounds (or expose per-input range overrides) so degeneratetype_vocab_size=1models don't require a hand-built.npz.