Readability-Extractor

This is a tiny JS wrapper library around Mozilla's article-text extraction tool https://github.com/mozilla/readability.

It's designed to be used as an ArchiveBox extractor.

Install

npm install -g 'git+https://github.com/pirate/readability-extractor'

# which is equivalent to this:
curl https://raw.githubusercontent.com/pirate/readability-extractor/master/readability-extractor > /usr/local/bin/readability-extractor
chmod +x /usr/local/bin/readability-extractor

Usage

# readability-extractor <input HTML path> <original url?> <suggested encoding?> > <output JSON path>
readability-extractor some_article.html 'https://exmaple.com/original/url/some/article.html' 'UTF-8' > some_article.json

{
    "title": "Title autodetected from article html",
    "byline": "Autodetected author...",
    "excerpt": "Autodetected short description",
    "dir": "ltr",
    "length": 1337,
    "lang": null,
    "charset": "UTF-8",
    "content": "<div id=\"readability-page-1\" class=\"page\">abc some article body text...</div>",
    "textContent": "abc some article body text..."
}

Large inline data: URLs over 1 MiB are replaced before HTML parsing to bound parser memory. Ordinary smaller inline images and remote images are preserved. The existing safeguard for documents over 14,000,000 input bytes also removes their inline base64 images. Use the original saved page when full embedded resources are required; article extraction retains the text and omits these oversized resources.

ArchiveBox Integration

# You don't have to run these commands usually.
# Readability is on by default and ArchiveBox will find any 
# installed version in your $PATH automatically

# However, if you explicitly want to turn readability on
# and/or specify a manual path to the binary, you can do this:
archivebox config --set SAVE_READABILITY=True
archivebox config --set READABILITY_BINARY="$(which readability-extractor)"

# test archiving oneshot using only singlefile+readability
archivebox add --extract=singlefile,readability 'https://exmaple.com'

Development and releases

Run npm ci and npm test. The regression tests invoke the real CLI against saved HTML.

.github/workflows/release.yml tests pushes and pull requests. A v<VERSION> tag publishes the matching package.json version to npm using GitHub Actions OIDC. Before the first automated release, the npm package maintainer must configure a trusted publisher in the package settings: GitHub organization ArchiveBox, repository readability-extractor, workflow filename release.yml, no environment, with permission to publish. No npm token or GitHub Actions secret is required.

To release, update the version with npm version patch --no-git-tag-version, commit and push the version files, wait for the main-branch checks to pass, then create and push the matching tag. Verify the version with npm view readability-extractor version before announcing publication. ArchiveBox's Readability plugin installs this npm package through its pnpm provider; a GitHub source push alone does not update that installed binary.