fix(epub): resolve percent-encoded manifest hrefs to ZIP entries - #2413
Open
Sushant Lokhande (sushantlokhande14) wants to merge 1 commit into
Open
Conversation
Manifest hrefs are URI references, so a chapter stored as `chapter 1.xhtml` appears in content.opf as `chapter%201.xhtml`. ZIP entry names are not URI-encoded, so joining the href to the OPF's directory verbatim produced `OEBPS/chapter%201.xhtml`, which is not a member of the archive. The spine loop skips anything missing from `z.namelist()`, so the chapter was dropped with no error: any EPUB whose filenames contain a space or a non-ASCII character lost that content silently. Decode the href and normalise the joined path before matching. The raw href is kept as a fallback candidate, so an archive that stores a literally-encoded name still resolves exactly as it did before. Normalising also lets an href reach outside the OPF's own directory, which the old string concatenation could not express. Adds tests for a space, a non-ASCII name, a literally-encoded entry, and a parent-relative href. The first, second and fourth fail without this change; the third passes either way and guards against regressing today's behaviour.
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Manifest hrefs in
content.opfare URI references, so a chapter stored in thearchive as
chapter 1.xhtmlis declared as:ZIP entry names are not URI-encoded.
EpubConverter.convertjoined the hrefto the OPF's directory verbatim:
which looks for
OEBPS/chapter%201.xhtml. That is not a member of the archive,and the spine loop skips anything missing from
z.namelist():So the chapter is dropped with no error and no warning. Any EPUB whose
filenames contain a space or a non-ASCII character loses that content, and the
caller has no way to tell.
Reproduction
A minimal EPUB with two chapters, one named
chapter 1.xhtml(hrefchapter%201.xhtml) and one namedplain.xhtml:The first chapter is gone. The archive members are
['OEBPS/chapter 1.xhtml', 'OEBPS/plain.xhtml'], while the converter looked forOEBPS/chapter%201.xhtml.Fix
Decode the href and normalise the joined path before matching it against the
archive. The raw href is retained as a fallback candidate, so an archive that
stores a literally-encoded name (
OEBPS/chapter%201.xhtmlas an actual entry)resolves exactly as it does today. Normalising also lets an href reach outside
the OPF's own directory, e.g.
../shared/chapter.xhtml, which the previousstring concatenation could not express.
The resolution is a small helper rather than inline logic so the candidate
order and the fallback are explicit.
Tests
New
packages/markitdown/tests/test_epub_converter.pycovers four cases:chapter%201.xhtmlOEBPS/chapter 1.xhtmlcap%C3%ADtulo.xhtmlOEBPS/capítulo.xhtmlchapter%201.xhtmlOEBPS/chapter%201.xhtml../shared/chapter.xhtmlshared/chapter.xhtmlThe third passes with and without the change on purpose: it is the guard that
this fix does not regress archives that work today.
Verification
pytest packages/markitdown/tests/test_epub_converter.py— 4 passed_epub_converter.pyand keeping the tests: 3 failed, 1 passed,matching the table above
pytest packages/markitdown/tests/, excluding the ContentUnderstanding and network-backed vectors) — 302 passed, 6 failed
identically on a clean checkout with this change stashed. They are Windows
console encoding on the CJK vector (
| ?? | ?? | ?? |),test_file_uris/test_convert_case_insensitive_uri_schemes, andtest_speech_transcription(no ffmpeg locally).
black --checkclean with the pinned23.7.0from.pre-commit-config.yamlScope: this is a fidelity fix to an existing converter, per What to Contribute
in the README. No new dependencies —
posixpathandurllib.parseare stdlib.Environment: Windows 11, Python 3.12.5.
AI assistance
AI assistance (Claude Code) was used to locate this defect and draft the patch
and tests. I reviewed every changed line, ran everything above locally, and can
defend the change end to end.
🤖 Generated with Claude Code