Skip to content

(Experimental) Pickle: Consider Series identity when unpickling - #1922

Open
franzpoeschel wants to merge 7 commits into
openPMD:devfrom
franzpoeschel:try-fixing-pickle
Open

(Experimental) Pickle: Consider Series identity when unpickling#1922
franzpoeschel wants to merge 7 commits into
openPMD:devfrom
franzpoeschel:try-fixing-pickle

Conversation

@franzpoeschel

@franzpoeschel franzpoeschel commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

This tries fixing the bug reported in #1919 by @pordyna by storing and reading additionally the immutable internal Series memory location. If the location of an unpickled Series is identical to the memory location of a previously unpickled Series, then that Series is reused.

Notes:

  • The cache of unpickled Series objects is still thread_local at the moment. This keeps thread safety at the cost of restricting this bugfix to uses in the same thread.
    Suggestion: When unpickling from the same Series object, users should expect to share the same internal data references. So it might be fine to use a per-process cache instead of a per-thread cache.
  • The cache is never emptied, so this can be a memory leak. We could maybe add a scan operation that erases closed Series objects from the cache?
  • Since our pickling strategy is relatively basic, we will probably keep running into issues like this, as use cases get more complex.
  • tqdm as test dependency
  • check if we can automatically manage memory, this currently relies on Series::close()

# https://github.com/openPMD/openPMD-api/issues/1919
# The code is adapted from the reproducer in there.
try:
from tqdm.contrib.concurrent import process_map

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you add tqdm as a requirement for your CI? Otherwise, this will never be tested. You could maybe also rewrite it with Pool().map() from multiprocessing if you want to avoid the extra dependency.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet, I have added it as a TODO
I want to give the CI one try on the test without fix anyway, so it will come up latest at that point

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytest has @skipif that we can use for missing imports.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree we can probably reproduce this with multiprocessing from the Python stdlib

Comment on lines +40 to +43
struct unpickled_series
{
std::map<uintptr_t, Series> m_series_by_former_id;
std::shared_mutex m_mutex;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An idea:
Usually when we create series in Python, the Python interpreter owns the lifetime. Now after pickle, there is a C++ object owning the lifetime.

There might be a way to attach the extra data/counter/id to the Python object dynamically and reference it (or the series that is de-serialized first) between each other in Python.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants