Skip to content

check --repair: share one object header parser, stop re-validating unchanged packs #10318

Description

@mr-raj12

Follow-ups from the review of #10094, tracked under #8476. All three are about pack object headers and
they touch the same code, so they are filed together, but they can be done independently.

1. Object header parsing exists in four copies

The check "does this buffer start with a valid object header" (OBJ_MAGIC, a version in
SUPPORTED_OBJ_VERSIONS, sizes that fit) is written out four times:

  • RepoObj.extract_crypted_data, repoobj.py:83
  • RepoObj.parse_meta, repoobj.py:157
  • RepoObj.parse, repoobj.py:186
  • PackReader._parse_header, repository.py:432

They do not agree. A wrong magic is "invalid object magic" in repoobj.py and "no object header" in
_parse_header. Only _parse_header bounds the object by the pack size and by MAX_DATA_SIZE.
parse and extract_crypted_data require the buffer to be exactly one object, parse_meta accepts
trailing bytes. A fifth copy is one new caller away.

Two more places take a single header field without checking the header at all: object_validator
takes data_size (repoobj.py:295) and Repository.get_many takes meta_size to size its read
(repository.py:1735).

Proposal: one RepoObj.parse_header(buf) classmethod that returns either the ObjHeader or a name
for the problem it found. The RepoObj methods raise IntegrityError from that name and keep their
own size rule, _parse_header adds only what is specific to a pack (does the object fit into this
pack, is it within MAX_DATA_SIZE).

Refactoring only. The one visible change is that the error strings stop disagreeing.

2. finish() validates every pack a second time

Under --repair with chunks_modified set, ArchiveChecker.finish() (archive.py:2787) rebuilds the
chunks index from every pack with the same validator check() already ran, although only the packs the
repair rewrote have changed. Per object that costs a metadata slot read of up to 1 KiB plus one
decryption, where a header-only walk reads the 49 header bytes and nothing else.

#10379 narrows this walk to the packs the repair wrote.

3. A full check --repair rebuilds the index from all packs twice before that

With a corrupt chunks index and no pack errors, Repository.check() rebuilds the index from every pack
and stores it (repository.py:1589, slow_rebuild=True, write_immediately=True).
ArchiveChecker.__init__ then drops that index and rebuilds it from every pack again
(archive.py:2306, slow_rebuild=repair). Both walks validate every object, so the packs are read and
decrypted twice, and a third time in finish() if the repair changed any pack.

Proposal: in a full check, let the archives phase use the index the repository check just built and
stored, instead of building its own.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions