Skip to content

Let parallel stores to the persistent HTTP cache succeed - #66

Merged
FlorianRappl merged 4 commits into
AngleSharp:develfrom
michaelthyregod:fix/http-cache-parallel-stores
Nov 9, 2026
Merged

FlorianRappl merged 4 commits into
AngleSharp:develfrom
michaelthyregod:fix/http-cache-parallel-stores

Conversation

@michaelthyregod

@michaelthyregod michaelthyregod commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Every store to the persistent HTTP cache rewrote the origin.txt of its origin, so parallel loads from one origin collided on it: 51 of 64 parallel stores failed in a repro. Stores of one key could also collide on the swap. The IOException reached the page load.

  • Change: as discussed, the provider takes care of it instead of the handler. A small internal WriteQueue runs the writes of one key one after another, and HttpCache routes all its handler writes (stores, RemoveAsync, ClearAsync) through it, keyed by origin. Writes for different origins still run in parallel. There are no locks and no thread waits: a write simply awaits the previous one of its origin. PersistenceHttpCacheHandler is unchanged, and custom IHttpCacheHandlers never see two writes for one origin at the same time either.
  • Behaviour: no API change. A response whose store waits for another write of the same origin returns that much later, since the store is awaited as before. A failed write is still reported to its caller and does not stop the next one.

Tests: 64 parallel stores to one origin and 16 to one key all succeed and can be read back (both fail without the queue). For the queue itself: writes of one key run one after another, writes of other keys don't wait, and a failed write reports its error without stopping the next.

WriteQueue is meant to be reused, e.g., for localStorage later, in its own PR.

🤖 Generated with Claude Code

Every store rewrote the origin.txt of its origin, so parallel loads from one
origin collided on it (51 of 64 parallel stores failed in a repro), and stores
of one key could collide on the swap. The IOException reached the page load.

origin.txt is only written when missing, as its content follows from the folder
name. A store that still collides with another one is dropped and its temporary
file deleted: the other entry is as good, and like a failed read, a failed store
only means a cache miss later.
Comment thread CHANGELOG.md Outdated
@@ -1,3 +1,7 @@
# 1.3.2

- Fixed parallel stores to the persistent HTTP cache failing the page load with an `IOException` @michaelthyregod

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1.3.1

@FlorianRappl FlorianRappl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are too many and imho unnecessary changes.

If we worry about parallelism here we should put in a origin-specific queue (Task) which puts one change/write after another. The read should always work from memory. If one write is outstanding then no new write on the same origin is queued.

@michaelthyregod

Copy link
Copy Markdown
Contributor Author

Picking up the file-system part here. I looked at where Io writes to disk: the persistent stores from IoOptions.CreatePersistent are the cookie file (AdvancedCookieProvider → ICookieFileHandler), the HTTP cache (HttpCache → IHttpCacheHandler) and localStorage (StorageProviderFactory → IStorageHandler). The other file access (file: requests, file inputs, SaveToAsync) only reads, or writes once to a path the caller chooses.

All three follow the same shape: a handler with plain read/write, and a provider above it deciding when to write. So a general solution could be a small internal helper that ensures at most one write per key (origin or file) at a time, used by each provider, so handlers never have to care about parallel writes. #65 already does this for the cookie file. For #66 that would mean dropping the handler changes and letting HttpCache use the helper. localStorage could follow in its own PR; I haven't verified yet whether it's hit in parallel in practice.

Would you like it that way? If so, I'd rework #66 to introduce the helper and use it in HttpCache.

@FlorianRappl

Copy link
Copy Markdown
Contributor

I think that's a great suggestion. Let's do it like that!

@FlorianRappl FlorianRappl added this to the v1.3.1 milestone Nov 9, 2026
HttpCache now queues its handler writes per origin (WriteQueue), so handlers never see parallel writes; the changes to PersistenceHttpCacheHandler are undone.
@michaelthyregod

Copy link
Copy Markdown
Contributor Author

Reworked as discussed: the writes now go through a small WriteQueue in HttpCache, one origin at a time, and the handler is unchanged. Description updated.

@FlorianRappl FlorianRappl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@FlorianRappl FlorianRappl added the fs label Nov 9, 2026
@FlorianRappl
FlorianRappl merged commit 9bfa7d0 into AngleSharp:devel Nov 9, 2026
5 checks passed
@michaelthyregod
michaelthyregod deleted the fix/http-cache-parallel-stores branch November 15, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants