Merge gitdb and smmap into the GitPython repository#2179
Merged
Conversation
Change force_bytes to use UTF-8 encoding by default
Tell PyPI long_description is Markdown
…pped-python-2-support Remove setup.cfg to have a pure python wheel
[skip CI]
Co-authored-by: Sebastian Thiel <sebastian.thiel@icloud.com>
…-smmap Publish updated package metadata by bumping smmap to 5.0.3
Bumps [gitdb/ext/smmap](https://github.com/gitpython-developers/smmap) from `5ec977a` to `e4ad410`. - [Release notes](https://github.com/gitpython-developers/smmap/releases) - [Commits](gitpython-developers/smmap@5ec977a...e4ad410) --- updated-dependencies: - dependency-name: gitdb/ext/smmap dependency-version: e4ad410 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
…es/gitdb/ext/smmap-e4ad410 Bump gitdb/ext/smmap from `5ec977a` to `e4ad410`
Python 3.14 emits a DeprecationWarning for codecs.open(), which gitdb
hits inside ReferenceDB._update_dbs_from_ref_file:
DeprecationWarning: codecs.open() is deprecated. Use open() instead.
The built-in open() has supported the encoding kwarg since Python 3.0
and the call site already passes encoding="utf-8", so the replacement
is byte-for-byte equivalent on every supported Python version.
Dropped the now-unused codecs import.
Verified the change with the existing test_ref.py suite.
Closes #128
fix: replace deprecated codecs.open with built-in open (#128)
Closes #120 DecompressMemMapReader.read(N) could return b'' mid-stream when zlib consumed input without producing output on a single decompress call (small N, header / dictionary frames in flight). The original `if dcompdat and ...` guard at the recursion site skipped the "refill to size" recursion in that case, so a caller using the standard idiom while chunk := stream.read(4096): yield chunk terminated at the first empty chunk -- before _br == _s. The guard exists for compressed_bytes_read(), which manipulates _br=0 and then drains the inner zip past its EOF. Recursing there would loop forever because the inner zip is already done. The fix uses zlib's own `eof` attribute (available on standard zlib.Decompress objects since Python 3.6) to distinguish: - dcompdat empty AND zip not at EOF -> still digesting, recurse - dcompdat empty AND zip at EOF -> compressed_bytes_read scrub or genuine EOF; do not recurse. `getattr(_zip, 'eof', False)` keeps the conservative behavior when running against a custom zlib object that does not expose the attribute. Adds a regression test that reads with chunk_size in {1, 4, 16, 64} from a 13 KB highly-compressible stream. With the old guard, the chunk_size <= 16 cases stopped at byte 0; the new test asserts they read all 13000 bytes. The full existing test suite (24 tests) still passes, including test_decompress_reader_special_case and test_pack which exercise the compressed_bytes_read scrub path that the original guard existed to protect.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Fix DecompressMemMapReader.read returning b'' before EOF (#120)
Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…tions/actions/checkout-7 Bump actions/checkout from 6 to 7
Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v6...v7) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…ctions/actions/checkout-7 Bump actions/checkout from 6 to 7
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6 to 7. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v6...v7) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…ctions/actions/setup-python-7 Bump actions/setup-python from 6 to 7
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Byron
force-pushed
the
subtree-merge-deps
branch
2 times, most recently
from
July 25, 2026 07:56
f377fc7 to
7e817cd
Compare
Member
Author
|
@EliahKagan While I reviewed it hunk by hunk, the change also touches dependabot, and it's unclear if it will still pick up its work for the now 2 additional top-level packages. Let's forward-fix this if needed. Also, I have never tried to release and would hope GitPython is unaffected, but I will find out. Also GitDB and SMMap releases are uncertain now, but if there is an issue I think it's easy to fix as well, maybe even with unified script machinery. |
GitPython previously depended on gitdb and smmap through nested git submodules. That made a complete checkout depend on extra repository state and forced related changes to be coordinated across three repositories. Keep both projects as independently buildable distributions under their own top-level directories, while updating GitPython packaging, CI, documentation, and test fixtures to consume the in-tree sources. Remove the obsolete gitlinks and imported repository automation so a normal clone contains all code needed to build and test the three packages. Co-authored-by: Sebastian Thiel <sebastian.thiel@icloud.com>
Byron
force-pushed
the
subtree-merge-deps
branch
from
July 25, 2026 08:23
7e817cd to
8a08afa
Compare
Byron
marked this pull request as ready for review
July 25, 2026 08:47
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request vendors the gitdb and smmap dependency projects into the GitPython repository (monorepo layout), and updates GitPython’s tests, packaging, and CI to use the in-tree projects instead of submodules.
Changes:
- Removes the gitdb submodule and adds in-tree
gitdb/andsmmap/projects (code, tests, docs, packaging). - Updates GitPython tests/fixtures to reconstruct dependency repos from the merged history for deterministic local cloning/submodule tests.
- Adjusts packaging/typing/tooling and CI workflows to install and validate
smmapandgitdbfrom local paths.
Reviewed changes
Copilot reviewed 107 out of 118 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/test_submodule.py | Updates submodule tests to use reconstructed local dependency repo URLs. |
| test/test_repo.py | Reworks submodule-related repo tests to use a RW fixture repo. |
| test/test_installation.py | Installs smmap/gitdb and then GitPython in venv for installation validation. |
| test/test_fun.py | Adds coverage for tree_entries_from_data accepting bytearray. |
| test/test_fixture_health.py | Replaces submodule fixture checks with “no gitlinks” and trust checks. |
| test/test_docs.py | Makes tutorial submodule snippet run against a RW fixture repo and local URLs. |
| test/test_clone.py | Uses no_hardlinks=True to avoid Windows cleanup issues when cloning locally. |
| test/lib/helper.py | Adds dependency-repo reconstruction helpers for gitdb/smmap from merged history. |
| smmap/tox.ini | Adds smmap tox configuration for lint/test. |
| smmap/smmap/util.py | Adds smmap utility mapping/window primitives. |
| smmap/smmap/test/test_util.py | Adds smmap util tests. |
| smmap/smmap/test/test_tutorial.py | Adds a tutorial-style test for smmap usage. |
| smmap/smmap/test/test_mman.py | Adds smmap memory manager tests. |
| smmap/smmap/test/test_buf.py | Adds smmap buffer tests (including perf-oriented test). |
| smmap/smmap/test/lib.py | Adds smmap test support utilities. |
| smmap/smmap/test/init.py | Initializes smmap test package. |
| smmap/smmap/buf.py | Adds smmap buffer implementation. |
| smmap/smmap/init.py | Adds smmap package init and re-exports. |
| smmap/setup.py | Adds smmap packaging entrypoint. |
| smmap/setup.cfg | Adds smmap flake8 configuration. |
| smmap/SECURITY.md | Points smmap security reporting to GitPython’s policy. |
| smmap/README.md | Adds smmap README in-repo. |
| smmap/MANIFEST.in | Adds smmap manifest. |
| smmap/Makefile | Adds smmap convenience Makefile targets. |
| smmap/LICENSE | Adds smmap license text. |
| smmap/doc/source/tutorial.rst | Vendors smmap tutorial documentation. |
| smmap/doc/source/intro.rst | Vendors smmap intro documentation. |
| smmap/doc/source/index.rst | Vendors smmap documentation index. |
| smmap/doc/source/conf.py | Vendors smmap Sphinx config. |
| smmap/doc/source/changes.rst | Vendors smmap changelog. |
| smmap/doc/source/api.rst | Vendors smmap API docs. |
| smmap/doc/Makefile | Vendors smmap doc Makefile. |
| smmap/doc/make.bat | Vendors smmap doc Windows build script. |
| smmap/doc/.gitignore | Adds smmap doc build ignores. |
| smmap/.gitignore | Adds smmap repo ignores. |
| smmap/.coveragerc | Adds smmap coverage configuration. |
| setup.py | Restricts root package discovery to git packages for GitPython distribution. |
| README.md | Updates dev instructions for editable installs in the monorepo. |
| pyproject.toml | Updates mypy/pyright/ruff excludes and paths for in-tree gitdb/smmap. |
| init-tests-after-clone.sh | Removes submodule initialization step (no longer applicable). |
| gitdb/setup.py | Adds gitdb packaging entrypoint. |
| gitdb/SECURITY.md | Points gitdb security reporting to GitPython’s policy. |
| gitdb/requirements.txt | Adds gitdb runtime dependency constraints for smmap. |
| gitdb/README.rst | Adds gitdb README in-repo. |
| gitdb/MANIFEST.in | Adds gitdb manifest. |
| gitdb/Makefile | Adds gitdb convenience Makefile targets. |
| gitdb/LICENSE | Adds gitdb license text. |
| gitdb/gitdb/utils/encoding.py | Adds gitdb encoding helpers. |
| gitdb/gitdb/utils/init.py | Initializes gitdb utils package. |
| gitdb/gitdb/typ.py | Adds gitdb type constants. |
| gitdb/gitdb/test/test_util.py | Adds gitdb utility tests. |
| gitdb/gitdb/test/test_stream.py | Adds gitdb stream tests (incl. regression coverage). |
| gitdb/gitdb/test/test_pack.py | Adds gitdb pack tests. |
| gitdb/gitdb/test/test_example.py | Adds gitdb example/tutorial tests. |
| gitdb/gitdb/test/test_base.py | Adds gitdb base type tests. |
| gitdb/gitdb/test/performance/test_stream.py | Vendors gitdb performance stream tests. |
| gitdb/gitdb/test/performance/test_pack.py | Vendors gitdb performance pack tests. |
| gitdb/gitdb/test/performance/test_pack_streaming.py | Vendors gitdb pack streaming performance tests. |
| gitdb/gitdb/test/performance/lib.py | Adds gitdb performance test base. |
| gitdb/gitdb/test/performance/init.py | Initializes gitdb performance test package. |
| gitdb/gitdb/test/lib.py | Adds gitdb test utilities and fixtures. |
| gitdb/gitdb/test/db/test_ref.py | Adds gitdb ReferenceDB tests. |
| gitdb/gitdb/test/db/test_pack.py | Adds gitdb PackedDB tests. |
| gitdb/gitdb/test/db/test_mem.py | Adds gitdb MemoryDB tests. |
| gitdb/gitdb/test/db/test_loose.py | Adds gitdb LooseObjectDB tests. |
| gitdb/gitdb/test/db/test_git.py | Adds gitdb GitDB tests. |
| gitdb/gitdb/test/db/lib.py | Adds shared gitdb DB test helpers. |
| gitdb/gitdb/test/db/init.py | Initializes gitdb DB test package. |
| gitdb/gitdb/test/init.py | Initializes gitdb test package. |
| gitdb/gitdb/exc.py | Adds gitdb exception types. |
| gitdb/gitdb/db/ref.py | Adds gitdb ReferenceDB implementation. |
| gitdb/gitdb/db/pack.py | Adds gitdb PackedDB implementation. |
| gitdb/gitdb/db/mem.py | Adds gitdb MemoryDB implementation. |
| gitdb/gitdb/db/loose.py | Adds gitdb LooseObjectDB implementation. |
| gitdb/gitdb/db/git.py | Adds gitdb GitDB compound implementation. |
| gitdb/gitdb/db/base.py | Adds gitdb DB base interfaces/compound DB logic. |
| gitdb/gitdb/db/init.py | Re-exports gitdb db modules. |
| gitdb/gitdb/const.py | Adds gitdb constants. |
| gitdb/gitdb/base.py | Adds gitdb core stream/info types. |
| gitdb/gitdb/init.py | Adds gitdb package init and re-exports. |
| gitdb/gitdb.pro | Vendors gitdb project metadata file. |
| gitdb/doc/source/tutorial.rst | Vendors gitdb tutorial docs. |
| gitdb/doc/source/intro.rst | Vendors gitdb intro docs. |
| gitdb/doc/source/index.rst | Vendors gitdb docs index. |
| gitdb/doc/source/conf.py | Vendors gitdb Sphinx config. |
| gitdb/doc/source/changes.rst | Vendors gitdb changelog. |
| gitdb/doc/source/api.rst | Vendors gitdb API docs. |
| gitdb/doc/source/algorithm.rst | Vendors gitdb algorithm discussion docs. |
| gitdb/doc/Makefile | Vendors gitdb doc Makefile. |
| gitdb/doc/.gitignore | Adds gitdb doc build ignores. |
| gitdb/AUTHORS | Adds gitdb authors list. |
| gitdb/.gitignore | Adds gitdb repo ignores. |
| gitdb/.coveragerc | Adds gitdb coverage configuration. |
| git/objects/fun.py | Ensures tree parsing works for bytearray by forcing bytes(...). |
| doc/source/intro.rst | Removes submodule init instructions from GitPython docs. |
| .pre-commit-config.yaml | Excludes gitdb/smmap paths from pre-commit hooks. |
| .gitmodules | Removes gitdb submodule configuration. |
| .github/workflows/pythonpackage.yml | Installs local smmap/gitdb in CI before running GitPython tests. |
| .github/workflows/dependencies.yml | Adds CI workflow to lint/test smmap and gitdb as standalone packages. |
| .github/workflows/cygwin-test.yml | Updates Cygwin CI to install local smmap/gitdb and removes submodule trust paths. |
| .github/workflows/codeql.yml | Ignores gitdb/smmap test paths for CodeQL analysis. |
| .github/workflows/alpine-test.yml | Updates Alpine CI to install local smmap/gitdb and removes submodule trust paths. |
| .github/dependabot.yml | Removes submodule update configuration from dependabot. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+30
to
+32
| def __contains__(self, sha): | ||
| return self.has_obj | ||
|
|
Comment on lines
+58
to
+59
| print(sys.stderr, "PDB Streaming: Wrote pack of size %i kb in %f s (%f kb/s)" % | ||
| (total_kb, elapsed, total_kb / (elapsed or 1)), sys.stderr) |
| @@ -0,0 +1,222 @@ | |||
| """Module containing a memory memory manager which provides a sliding window on a number of memory mapped files""" | |||
| The submodules must have been cloned for that to work, but that will already be the case if you have run `./init-tests-after-clone.sh`. You can use `pip list` to check which packages are installed editably and which are installed normally. | ||
| You can use `pip list` to check which packages are installed editably and which are installed normally. | ||
|
|
||
| To reiterate, this approach should only rarely be used. For most development it is preferable to allow the gitdb and smmap dependencices to be retrieved automatically from PyPI in their latest stable packaged versions. |
Comment on lines
+148
to
+152
| def set_ostream(self, stream): | ||
| """:raise TypeError: if the stream does not support the Sha1Writer interface""" | ||
| if stream is not None and not isinstance(stream, Sha1Writer): | ||
| raise TypeError("Output stream musst support the %s interface" % Sha1Writer.__name__) | ||
| return super().set_ostream(stream) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tasks
This section is for Byron only. Models continuing this PR must not add, remove, check, uncheck, rename, or reorder checkboxes here.
Everything below this line was generated by Codex GPT-5.
Created by Codex on behalf of Byron. Byron will review before this is ready to merge.
Reported issue
Use this information to plan for a subtree merge, combining gitdb and ssmap repositories into GitPython.
The requested ssmap repository is interpreted as smmap, matching the named GitPython dependency and the source repository discussed in the linked issues.
Summary
8ce61ad5cc4016bffaf25080bc0d69b3acbe8555intosmmap/.2da3232f9d58e7761e384ac6d32f7b1ed77a74a2intogitdb/.git,gitdb, andsmmapdistributions and import APIs.Context
This implements the subtree direction discussed in GitPython issues #511, #933, and #1716, gitdb issue #116, and smmap pull request #53.
Validation
git diff --check