Serialize the HTTP/2 send path under a dedicated lock (thread-safety race)

[Dextheking1]

Closes encode/httpx#3566

Root cause

httpx.Client(http2=True) shares a single connection across threads (HTTP/2 multiplexing). That connection is httpcore.HTTP2Connection (plus the async twin), which wraps one shared h2.H2Connection state machine. The receive path (_read_lock) and socket writes (_write_lock) were already serialized, but the send path was not: concurrent threads could interleave

• get_next_available_stream_id() → duplicate stream IDs handed out (StreamIDTooLowError: 2411 is lower than 2413),
• send_headers() → HPACK encode iterating dynamic_entries while another thread mutates it (RuntimeError: deque mutated during iteration — the first traceback in the linked issue),
• open_outbound_streams iterating h2.streams while another thread adds/removes streams (dictionary changed size during iteration, KeyError).

h2's docs require get_next_available_stream_id() to be immediately followed by the matching send_headers() — impossible under concurrency without a lock. Serializing in httpx would kill multiplexing, so the fix has to live here.

The fix

One new lock, _send_lock (threading.Lock / anyio.AsyncLock), held in handle_request around stream-ID allocation, _events[stream_id] registration, _send_request_headers, and _send_request_body. The NoAvailableStreamIDError → ConnectionNotAvailable handling and the except BaseException cleanup (_response_closed, semaphore release) keep their exact original semantics — the send block simply moved inside the existing cleanup try.

Lock-order audit: _send_lock is acquired only in handle_request, outermost relative to _read_lock/_write_lock/_state_lock; the max-streams semaphore is acquired before it and its holder never waits on a waiter. No new deadlock.

Evidence

Fail-before / pass-after with an in-memory fake HTTP/2 transport (real client- and server-side h2 state machines, server-side guarded by its own lock so only client-side httpcore code is under test):

case	pre-fix	post-fix
new regression test: 8 threads × 50 requests (half with 64 B bodies)	327/400 fail — LocalProtocolError(StreamIDTooLowError), KeyError	400/400 pass (×4 runs)
GET 8×300	33/2400 ok	2400/2400 (×3)
POST 16×100, half with 64 B bodies	8–1191/1600 ok	1600/1600 (×10)
POST 16×400	—	6400/6400 (×3)
existing tests: sync http2 + connection + pool (37), async http2 + connection (38)	75 passed	75 passed

New regression test: tests/_sync/test_http2_thread_safety.py.

[Priyankm23]

Hi @Dextheking1 / maintainers!

A quick note regarding the CI failure in this PR:
The failure at Build package & docs is caused by twine==6.1.0 in requirements.txt, which fails to recognize wheel Metadata-Version: 2.5 generated by newer hatchling. Bumping twine to 7.0.0 resolves the build failure.

Also, regarding the concurrency design:
Holding AsyncLock across _send_request_body means that if one async request is uploading or streaming a large body, all other async requests on that connection are blocked from sending until that upload completes. This creates a bottleneck and defeats HTTP/2 multiplexing in async mode.

I have implemented an alternative approach on my fork that:

1. Uses AsyncThreadLock / ThreadLock (threading.Lock in sync, zero-cost no-op in async) so async requests can continue to multiplex and upload bodies freely without any lock contention.
2. Synchronizes all shared in-memory _h2_state operations (stream ID allocation, headers, data, end_stream, acknowledge_received_data, and buffer draining) with zero network I/O under lock.
3. Achieves 100% statement coverage across all 3,656 statements with all checks and tests passing.

Branch reference: https://github.com/Priyankm23/httpcore/tree/fix-http2-header-encoding-race (commit 7d48dcc).

I plan to open this as a PR against encode/httpcore as soon as the repository's temporary interaction limits are lifted, or maintainers are welcome to review the branch directly.