fix: prefer embedded PDF text over OCR for hi_res table tokens

[claytonlin1110]

Summary

• Fixes table-token generation in hi_res pipeline to prioritize embedded text (extracted_regions) for infer_table_structure=True.
• Preserves OCR as fallback for scanned/image-only documents.
• Adds regression test to ensure embedded text is used when both sources are available.

Closes #4092

Root cause

• supplement_element_with_table_extraction() passed extracted_regions through the pipeline but get_table_tokens() ignored it and always generated tokens from OCR output.
• This made digital PDFs susceptible to OCR hallucinations/substitutions in table cells.

What changed

• Updated get_table_tokens() signature and logic to accept:
  • extracted_regions
  • table_bbox
  • padding
• Added _get_table_tokens_from_extracted_regions() to:
  • select text regions inside table bbox
  • convert coordinates into cropped-table space
  • keep stable reading order
• Updated caller in supplement_element_with_table_extraction() to pass required context.
• Added test_get_table_tokens_prefers_extracted_regions_over_ocr().

---

Note

Medium Risk
Changes table token generation in the hi_res PDF table-extraction path, which can affect table-structure inference outputs; fallback logic helps, but coverage heuristics could shift behavior on edge-case PDFs.

Overview
Updates hi_res PDF table extraction to generate tatr table OCR tokens from embedded PDF text (extracted_regions) when it provides comparable coverage, falling back to OCR tokens when embedded text is missing or sparse.

This threads table context (table_bbox + crop padding) into get_table_tokens(), adds selection/coordinate-mapping helpers and a coverage heuristic, and adds regression tests for both the preferred-embedded and OCR-fallback paths. Also bumps __version__ to 0.22.28 and adds a 0.22.28 changelog entry.

^{Reviewed by Cursor Bugbot for commit 321bfaa. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Non-contiguous span_num when tokens are skipped

Medium Severity

In _get_table_tokens_from_extracted_regions, span_num is assigned from enumerate(sorted_indices), but when a token is skipped by the continue on the degenerate-bbox check, the counter still increments, producing gaps in span_num values (e.g., [0, 2, 4] instead of [0, 1, 2]). The OCR path always produces contiguous span numbers. Downstream consumers (table transformer) receive tokens whose span_num field is inconsistent with the contract established by the OCR path, which could affect cell-to-token assignment or reading-order logic.

 

^{Reviewed by Cursor Bugbot for commit 1fda8ae. Configure here.}

[claytonlin1110]

@SudSampath Would you please review?

[claytonlin1110]

@cragwolfe Would you please review?

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 321bfaa. Configure here.}

[cursor]

Embedded tokens lose preference

Medium Severity

The _prefer_extracted_table_tokens function compares token counts across different granularities (line-level for embedded text vs. word-level for OCR). This can cause complete embedded table text to be incorrectly rejected, leading to a fallback to OCR and reintroducing OCR-related substitutions.

 

^{Reviewed by Cursor Bugbot for commit 321bfaa. Configure here.}

[cursor]

Low-fidelity text can replace OCR

Medium Severity

_get_table_tokens_from_extracted_regions() accepts any non-empty PDFMiner text and ignores extraction quality. Invisible or low-fidelity OCR layers in scanned PDFs can be preferred over fresh OCR, so the intended OCR fallback for scanned documents is bypassed.

 

^{Reviewed by Cursor Bugbot for commit 321bfaa. Configure here.}