Skip to content

fix(python): read consumer metadata without the consumer lock - #3888

Open
ethanlin01x wants to merge 2 commits into
apache:masterfrom
ethanlin01x:fix/python-consumer-metadata-deadlock
Open

fix(python): read consumer metadata without the consumer lock#3888
ethanlin01x wants to merge 2 commits into
apache:masterfrom
ethanlin01x:fix/python-consumer-metadata-deadlock

Conversation

@ethanlin01x

@ethanlin01x ethanlin01x commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Relates to #3776 (Found while reviewing)

Rationale

The synchronous getters on the Python IggyConsumer took the consumer mutex with blocking_lock() while holding the GIL, and consume_messages holds that mutex for the whole consumption run. Reading an attribute during consumption hung the interpreter; reading one from a callback panicked inside the Tokio runtime. Neither is recoverable from Python.

consume = consumer.consume_messages(handle, shutdown_event)
print(consumer.name())   # never returns

What changed?

None of those getters need exclusive access.

name, stream and topic never change after construction, so the Python wrapper snapshots them. The partition id and offsets do change, but already live behind Arcs in the Rust SDK, which now hands them out as IggyConsumerState via IggyConsumer::state(). The wrapper keeps a clone and reads them as atomic loads, never taking the lock.

Additive on the Rust side. On the Python side stream() and topic() now return the identifier directly instead of a PyResult.

Local Execution

  • Passed
  • Pre-commit hooks ran

AI Usage

  1. Which tools? Claude
  2. Scope of usage? help implement and write PR description
  3. How did you verify the generated code works correctly? Ran the Python test suite against a real server
  4. Can you explain every line of the code if asked? Yes, all the changes are checked by the human.

@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 67.74194% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.79%. Comparing base (c0c7493) to head (004e7cb).

Files with missing lines Patch % Lines
core/sdk/src/clients/consumer.rs 56.52% 20 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3888      +/-   ##
============================================
- Coverage     83.84%   83.79%   -0.05%     
  Complexity     1358     1358              
============================================
  Files          1212     1212              
  Lines        166843   166869      +26     
  Branches     134306   134454     +148     
============================================
- Hits         139890   139830      -60     
+ Misses        23312    23256      -56     
- Partials       3641     3783     +142     
Components Coverage Δ
Rust Core 84.65% <56.52%> (+0.03%) ⬆️
Java SDK 66.67% <ø> (ø)
C# SDK 74.99% <ø> (-1.56%) ⬇️
Python SDK 90.14% <100.00%> (+<0.01%) ⬆️
PHP SDK 84.48% <ø> (ø)
Node SDK 95.84% <ø> (ø)
Go SDK 68.32% <ø> (ø)
Files with missing lines Coverage Δ
foreign/python/src/client.rs 99.85% <100.00%> (+<0.01%) ⬆️
foreign/python/src/consumer.rs 81.81% <100.00%> (-0.43%) ⬇️
core/sdk/src/clients/consumer.rs 71.05% <56.52%> (-0.72%) ⬇️

... and 74 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ethanlin01x
ethanlin01x force-pushed the fix/python-consumer-metadata-deadlock branch from 8d0d07d to b73cf67 Compare August 15, 2026 09:28
@ethanlin01x
ethanlin01x marked this pull request as ready for review August 15, 2026 09:41
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 15, 2026
@ethanlin01x

Copy link
Copy Markdown
Contributor Author

/request-review @hubcio

@ethanlin01x

Copy link
Copy Markdown
Contributor Author

/ready

@github-actions
github-actions Bot requested a review from hubcio August 15, 2026 10:39
@ethanlin01x
ethanlin01x force-pushed the fix/python-consumer-metadata-deadlock branch from 7b7a74e to 5763540 Compare August 20, 2026 13:19
The synchronous getters on IggyConsumer took the consumer mutex with
blocking_lock() while holding the GIL, and consume_messages holds that mutex
for the whole consumption run. Reading an attribute during consumption hung the
interpreter; reading one from a callback panicked inside the Tokio runtime.

None of those getters need exclusive access. The name, stream and topic are
fixed at construction, and the partition id and offsets live behind Arcs that
IggyConsumerState now exposes as a cloneable view. IggyConsumer owns that state
and delegates to it, so the Python wrapper reads metadata without the lock.

Found while reviewing apache#3776.
@ethanlin01x
ethanlin01x force-pushed the fix/python-consumer-metadata-deadlock branch from 5763540 to e9c641e Compare August 20, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants