Skip to content

fix(epub): resolve percent-encoded manifest hrefs to ZIP entries - #2413

Open
Sushant Lokhande (sushantlokhande14) wants to merge 1 commit into
microsoft:mainfrom
sushantlokhande14:fix/epub-percent-encoded-href
Open

fix(epub): resolve percent-encoded manifest hrefs to ZIP entries#2413
Sushant Lokhande (sushantlokhande14) wants to merge 1 commit into
microsoft:mainfrom
sushantlokhande14:fix/epub-percent-encoded-href

Conversation

@sushantlokhande14

Copy link
Copy Markdown

Problem

Manifest hrefs in content.opf are URI references, so a chapter stored in the
archive as chapter 1.xhtml is declared as:

<item id="c1" href="chapter%201.xhtml" media-type="application/xhtml+xml"/>

ZIP entry names are not URI-encoded. EpubConverter.convert joined the href
to the OPF's directory verbatim:

spine = [
    f"{base_path}/{manifest[item_id]}" if base_path else manifest[item_id]
    ...
]

which looks for OEBPS/chapter%201.xhtml. That is not a member of the archive,
and the spine loop skips anything missing from z.namelist():

for file in spine:
    if file in z.namelist():   # silently false

So the chapter is dropped with no error and no warning. Any EPUB whose
filenames contain a space or a non-ASCII character loses that content, and the
caller has no way to tell.

Reproduction

A minimal EPUB with two chapters, one named chapter 1.xhtml (href
chapter%201.xhtml) and one named plain.xhtml:

$ markitdown spaced.epub
**Title:** Spaced Chapters
**Authors:** Test Author
**Language:** en

# Second Chapter

UNIQUE_PLAIN_BODY

The first chapter is gone. The archive members are
['OEBPS/chapter 1.xhtml', 'OEBPS/plain.xhtml'], while the converter looked for
OEBPS/chapter%201.xhtml.

Fix

Decode the href and normalise the joined path before matching it against the
archive. The raw href is retained as a fallback candidate, so an archive that
stores a literally-encoded name (OEBPS/chapter%201.xhtml as an actual entry)
resolves exactly as it does today. Normalising also lets an href reach outside
the OPF's own directory, e.g. ../shared/chapter.xhtml, which the previous
string concatenation could not express.

The resolution is a small helper rather than inline logic so the candidate
order and the fallback are explicit.

Tests

New packages/markitdown/tests/test_epub_converter.py covers four cases:

case href archive entry fails before?
space chapter%201.xhtml OEBPS/chapter 1.xhtml yes
non-ASCII cap%C3%ADtulo.xhtml OEBPS/capítulo.xhtml yes
literally encoded chapter%201.xhtml OEBPS/chapter%201.xhtml no, by design
parent-relative ../shared/chapter.xhtml shared/chapter.xhtml yes

The third passes with and without the change on purpose: it is the guard that
this fix does not regress archives that work today.

Verification

  • pytest packages/markitdown/tests/test_epub_converter.py — 4 passed
  • Reverting only _epub_converter.py and keeping the tests: 3 failed, 1 passed,
    matching the table above
  • Full suite (pytest packages/markitdown/tests/, excluding the Content
    Understanding and network-backed vectors) — 302 passed, 6 failed
  • Those 6 failures are pre-existing and unrelated: I confirmed they fail
    identically on a clean checkout with this change stashed. They are Windows
    console encoding on the CJK vector (| ?? | ?? | ?? |), test_file_uris /
    test_convert_case_insensitive_uri_schemes, and test_speech_transcription
    (no ffmpeg locally).
  • black --check clean with the pinned 23.7.0 from .pre-commit-config.yaml

Scope: this is a fidelity fix to an existing converter, per What to Contribute
in the README. No new dependencies — posixpath and urllib.parse are stdlib.

Environment: Windows 11, Python 3.12.5.

AI assistance

AI assistance (Claude Code) was used to locate this defect and draft the patch
and tests. I reviewed every changed line, ran everything above locally, and can
defend the change end to end.

🤖 Generated with Claude Code

Manifest hrefs are URI references, so a chapter stored as `chapter 1.xhtml`
appears in content.opf as `chapter%201.xhtml`. ZIP entry names are not
URI-encoded, so joining the href to the OPF's directory verbatim produced
`OEBPS/chapter%201.xhtml`, which is not a member of the archive. The spine loop
skips anything missing from `z.namelist()`, so the chapter was dropped with no
error: any EPUB whose filenames contain a space or a non-ASCII character lost
that content silently.

Decode the href and normalise the joined path before matching. The raw href is
kept as a fallback candidate, so an archive that stores a literally-encoded
name still resolves exactly as it did before. Normalising also lets an href
reach outside the OPF's own directory, which the old string concatenation could
not express.

Adds tests for a space, a non-ASCII name, a literally-encoded entry, and a
parent-relative href. The first, second and fourth fail without this change;
the third passes either way and guards against regressing today's behaviour.
@sushantlokhande14

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant