test: SSE conformance leg for the gateway e2e suite (+ graceful 501 degradation) - #22
Conversation
Description - Add test_stream_delivers_lifecycle to the env-gated gateway e2e suite: consumes job.events() under a daemon-thread watchdog (120s hard cap) and asserts the v1 stream contract — prompt StatusChange snapshot, OutputReady before the terminal frame, iteration ending on its own at a terminal StatusChange, and agreement with the poll-authoritative state. Duplicate frames are tolerated (snapshot semantics). - Stub server can now answer the events endpoint with 501 not_implemented (events_not_implemented state flag). - Pin today's client behavior on a 501 events endpoint: job.events() raises the protocol-level ApiError (code not_implemented), sync and async. - Cover the remaining backstop combination: a stream that ends cleanly without a terminal frame falls back to polling and ends on a synthetic terminal StatusChange (clean-end/poll-running and timeout/poll-terminal were already covered). How tested - uv run pytest: 105 passed, 4 skipped (integration leg skips without COMFY_BASE_URL/COMFY_API_KEY, proving the gating). - uv run ruff check / format --check, uv run mypy src: clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M2mBv9d3kfp7SmZ9uMc4tR
Description - A 501 from the events endpoint now ends job.events() iteration silently (yield nothing) instead of raising, on both the sync and async paths. The spec declares 501 a contract-legal response for surfaces without SSE, and the SDK is poll-first with SSE as enhancement — wait()/result() remain fully functional, so a hard raise punished users for a server's legal choice. Any other ApiError from the stream still raises. - Update the 501 pin tests to the new behavior; the previous raise-ApiError expectation is documented in the test docstring. How tested - uv run pytest: 105 passed, 4 skipped (env-gated integration leg). - uv run ruff check / format --check, uv run mypy src: clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M2mBv9d3kfp7SmZ9uMc4tR
|
Warning
|
| Layer / File(s) | Summary |
|---|---|
Handle unsupported SSE responses src/comfy_sdk/jobs.py |
Synchronous and asynchronous event iterators return immediately for HTTP 501 responses, with the behavior documented for Job.events(). |
Model unavailable event endpoints tests/conftest.py |
The stub server can return a 501 JSON error instead of opening an SSE stream. |
Validate stream termination and lifecycle tests/test_events.py, tests/integration/test_gateway_e2e.py |
Tests cover clean stream closure with polling, synchronous and asynchronous unsupported-endpoint handling, and successful SSE lifecycle ordering. A tiny stream of tests keeps the flow neat and complete. |
Sequence Diagram(s)
sequenceDiagram
participant Client
participant EventsEndpoint
participant JobStatus
Client->>EventsEndpoint: connect to job events
EventsEndpoint-->>Client: SSE lifecycle frames
EventsEndpoint-->>Client: HTTP 501 when SSE is unavailable
Client-->>Client: end events iterator silently
Client->>JobStatus: poll authoritative job status
JobStatus-->>Client: terminal succeeded status
🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
| Check name | Status | Explanation |
|---|---|---|
| Linked Issues check | ✅ Passed | Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | ✅ Passed | Check skipped because no linked issues were found for this pull request. |
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
- Commit unit tests in branch
sundar/sse-conformance-leg
✨ Simplify code
- Create PR with simplified code
- Commit simplified code in branch
sundar/sse-conformance-leg
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/test_gateway_e2e.py`:
- Around line 143-146: Make the tight FIRST_FRAME_S assertion in the gateway
timing test independent of test ordering by adding or reusing a session-scoped
warm-up fixture that submits and awaits a throwaway job before the test runs.
Apply the fixture explicitly to the affected test(s), including the assertion
around the first-frame timing, while preserving the existing SSE contract
checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 0fb93e56-714e-4696-9934-84a6bc949029
📒 Files selected for processing (4)
src/comfy_sdk/jobs.pytests/conftest.pytests/integration/test_gateway_e2e.pytests/test_events.py
wei-hai
left a comment
There was a problem hiding this comment.
Found one test-correctness issue.
Description
The platform-gateway now implements
GET /jobs/{id}/events(SSE v1:statussnapshot/change frames,outputframes, 15s keepalives). This turns the manual staging smoke into a repeatable, env-gated conformance test, and pins the SDK's behavior when a surface answers the events endpoint with 501.Commit 1 — conformance leg + behavior pins
test_stream_delivers_lifecyclein the env-gated gateway e2e suite: submits the model-free workflow and consumesjob.events()under a daemon-thread watchdog (120s hard cap, so a wedged stream fails instead of hanging CI). Asserts: aStatusChangesnapshot arrives promptly after connect (generous 30s bound; ~0.4s observed live), anOutputReadyarrives before the terminal frame, iteration ends on its own at a terminalStatusChange(succeeded), andjob.refresh()agrees. Duplicate frames are tolerated (snapshot semantics — no counts asserted).events_not_implementedmode (501 +not_implementedErrorEnvelope) and the then-current behavior was pinned:events()raised the protocol-levelcomfy_low.errors.ApiError.StatusChange, no reconnect (clean-end/poll-running and idle-timeout/poll-terminal were already covered).Commit 2 — graceful 501 degradation
events()iteration silently (yield nothing) instead of raising, sync and async. Rationale: the spec declares 501 contract-legal for surfaces without SSE, and the SDK is poll-first with SSE as enhancement —wait()/result()remain fully functional, so a hard raise punished users for a server's legal choice. Any otherApiErrorfrom the stream still raises. Pin tests updated (old expectation documented in the docstring), including assertingwait()still completes and that there is no reconnect loop on 501.How tested
uv run pytest: 105 passed, 4 skipped — the 4 skips are the integration leg withoutCOMFY_BASE_URL/COMFY_API_KEY, proving the gating works.uv run ruff check/ruff format --check/mypy src: clean.🤖 Generated with Claude Code
https://claude.ai/code/session_01M2mBv9d3kfp7SmZ9uMc4tR