(Experimental) Pickle: Consider Series identity when unpickling - #1922
(Experimental) Pickle: Consider Series identity when unpickling#1922franzpoeschel wants to merge 7 commits into
Conversation
| # https://github.com/openPMD/openPMD-api/issues/1919 | ||
| # The code is adapted from the reproducer in there. | ||
| try: | ||
| from tqdm.contrib.concurrent import process_map |
There was a problem hiding this comment.
Do you add tqdm as a requirement for your CI? Otherwise, this will never be tested. You could maybe also rewrite it with Pool().map() from multiprocessing if you want to avoid the extra dependency.
There was a problem hiding this comment.
Not yet, I have added it as a TODO
I want to give the CI one try on the test without fix anyway, so it will come up latest at that point
There was a problem hiding this comment.
pytest has @skipif that we can use for missing imports.
There was a problem hiding this comment.
I agree we can probably reproduce this with multiprocessing from the Python stdlib
| struct unpickled_series | ||
| { | ||
| std::map<uintptr_t, Series> m_series_by_former_id; | ||
| std::shared_mutex m_mutex; |
There was a problem hiding this comment.
An idea:
Usually when we create series in Python, the Python interpreter owns the lifetime. Now after pickle, there is a C++ object owning the lifetime.
There might be a way to attach the extra data/counter/id to the Python object dynamically and reference it (or the series that is de-serialized first) between each other in Python.
This tries fixing the bug reported in #1919 by @pordyna by storing and reading additionally the immutable internal Series memory location. If the location of an unpickled Series is identical to the memory location of a previously unpickled Series, then that Series is reused.
Notes:
thread_localat the moment. This keeps thread safety at the cost of restricting this bugfix to uses in the same thread.Suggestion: When unpickling from the same
Seriesobject, users should expect to share the same internal data references. So it might be fine to use a per-process cache instead of a per-thread cache.Seriesobjects from the cache?Series::close()