Follow-ups from the review of #10094, tracked under #8476. All three are about pack object headers and
they touch the same code, so they are filed together, but they can be done independently.
1. Object header parsing exists in four copies
The check "does this buffer start with a valid object header" (OBJ_MAGIC, a version in
SUPPORTED_OBJ_VERSIONS, sizes that fit) is written out four times:
RepoObj.extract_crypted_data, repoobj.py:83
RepoObj.parse_meta, repoobj.py:157
RepoObj.parse, repoobj.py:186
PackReader._parse_header, repository.py:432
They do not agree. A wrong magic is "invalid object magic" in repoobj.py and "no object header" in
_parse_header. Only _parse_header bounds the object by the pack size and by MAX_DATA_SIZE.
parse and extract_crypted_data require the buffer to be exactly one object, parse_meta accepts
trailing bytes. A fifth copy is one new caller away.
Two more places take a single header field without checking the header at all: object_validator
takes data_size (repoobj.py:295) and Repository.get_many takes meta_size to size its read
(repository.py:1735).
Proposal: one RepoObj.parse_header(buf) classmethod that returns either the ObjHeader or a name
for the problem it found. The RepoObj methods raise IntegrityError from that name and keep their
own size rule, _parse_header adds only what is specific to a pack (does the object fit into this
pack, is it within MAX_DATA_SIZE).
Refactoring only. The one visible change is that the error strings stop disagreeing.
2. finish() validates every pack a second time
Under --repair with chunks_modified set, ArchiveChecker.finish() (archive.py:2787) rebuilds the
chunks index from every pack with the same validator check() already ran, although only the packs the
repair rewrote have changed. Per object that costs a metadata slot read of up to 1 KiB plus one
decryption, where a header-only walk reads the 49 header bytes and nothing else.
#10379 narrows this walk to the packs the repair wrote.
3. A full check --repair rebuilds the index from all packs twice before that
With a corrupt chunks index and no pack errors, Repository.check() rebuilds the index from every pack
and stores it (repository.py:1589, slow_rebuild=True, write_immediately=True).
ArchiveChecker.__init__ then drops that index and rebuilds it from every pack again
(archive.py:2306, slow_rebuild=repair). Both walks validate every object, so the packs are read and
decrypted twice, and a third time in finish() if the repair changed any pack.
Proposal: in a full check, let the archives phase use the index the repository check just built and
stored, instead of building its own.
Follow-ups from the review of #10094, tracked under #8476. All three are about pack object headers and
they touch the same code, so they are filed together, but they can be done independently.
1. Object header parsing exists in four copies
The check "does this buffer start with a valid object header" (
OBJ_MAGIC, a version inSUPPORTED_OBJ_VERSIONS, sizes that fit) is written out four times:RepoObj.extract_crypted_data,repoobj.py:83RepoObj.parse_meta,repoobj.py:157RepoObj.parse,repoobj.py:186PackReader._parse_header,repository.py:432They do not agree. A wrong magic is "invalid object magic" in
repoobj.pyand "no object header" in_parse_header. Only_parse_headerbounds the object by the pack size and byMAX_DATA_SIZE.parseandextract_crypted_datarequire the buffer to be exactly one object,parse_metaacceptstrailing bytes. A fifth copy is one new caller away.
Two more places take a single header field without checking the header at all:
object_validatortakes
data_size(repoobj.py:295) andRepository.get_manytakesmeta_sizeto size its read(
repository.py:1735).Proposal: one
RepoObj.parse_header(buf)classmethod that returns either theObjHeaderor a namefor the problem it found. The
RepoObjmethods raiseIntegrityErrorfrom that name and keep theirown size rule,
_parse_headeradds only what is specific to a pack (does the object fit into thispack, is it within
MAX_DATA_SIZE).Refactoring only. The one visible change is that the error strings stop disagreeing.
2.
finish()validates every pack a second timeUnder
--repairwithchunks_modifiedset,ArchiveChecker.finish()(archive.py:2787) rebuilds thechunks index from every pack with the same validator
check()already ran, although only the packs therepair rewrote have changed. Per object that costs a metadata slot read of up to 1 KiB plus one
decryption, where a header-only walk reads the 49 header bytes and nothing else.
#10379 narrows this walk to the packs the repair wrote.
3. A full
check --repairrebuilds the index from all packs twice before thatWith a corrupt chunks index and no pack errors,
Repository.check()rebuilds the index from every packand stores it (
repository.py:1589,slow_rebuild=True, write_immediately=True).ArchiveChecker.__init__then drops that index and rebuilds it from every pack again(
archive.py:2306,slow_rebuild=repair). Both walks validate every object, so the packs are read anddecrypted twice, and a third time in
finish()if the repair changed any pack.Proposal: in a full check, let the archives phase use the index the repository check just built and
stored, instead of building its own.