Skip to content

test: SSE conformance leg for the gateway e2e suite (+ graceful 501 degradation) - #22

Merged
wei-hai merged 2 commits into
mainfrom
sundar/sse-conformance-leg
Jul 30, 2026
Merged

test: SSE conformance leg for the gateway e2e suite (+ graceful 501 degradation)#22
wei-hai merged 2 commits into
mainfrom
sundar/sse-conformance-leg

Conversation

@sundar-svg

Copy link
Copy Markdown
Contributor

Description

The platform-gateway now implements GET /jobs/{id}/events (SSE v1: status snapshot/change frames, output frames, 15s keepalives). This turns the manual staging smoke into a repeatable, env-gated conformance test, and pins the SDK's behavior when a surface answers the events endpoint with 501.

Commit 1 — conformance leg + behavior pins

  • test_stream_delivers_lifecycle in the env-gated gateway e2e suite: submits the model-free workflow and consumes job.events() under a daemon-thread watchdog (120s hard cap, so a wedged stream fails instead of hanging CI). Asserts: a StatusChange snapshot arrives promptly after connect (generous 30s bound; ~0.4s observed live), an OutputReady arrives before the terminal frame, iteration ends on its own at a terminal StatusChange(succeeded), and job.refresh() agrees. Duplicate frames are tolerated (snapshot semantics — no counts asserted).
  • Stub server gains an events_not_implemented mode (501 + not_implemented ErrorEnvelope) and the then-current behavior was pinned: events() raised the protocol-level comfy_low.errors.ApiError.
  • Adds the one uncovered backstop combination: a stream ending cleanly without a terminal frame + poll already terminal → synthetic terminal StatusChange, no reconnect (clean-end/poll-running and idle-timeout/poll-terminal were already covered).

Commit 2 — graceful 501 degradation

  • A 501 from the events endpoint now ends events() iteration silently (yield nothing) instead of raising, sync and async. Rationale: the spec declares 501 contract-legal for surfaces without SSE, and the SDK is poll-first with SSE as enhancement — wait()/result() remain fully functional, so a hard raise punished users for a server's legal choice. Any other ApiError from the stream still raises. Pin tests updated (old expectation documented in the docstring), including asserting wait() still completes and that there is no reconnect loop on 501.

How tested

  • uv run pytest: 105 passed, 4 skipped — the 4 skips are the integration leg without COMFY_BASE_URL/COMFY_API_KEY, proving the gating works.
  • uv run ruff check / ruff format --check / mypy src: clean.
  • The equivalent stream consumption was verified manually against staging (snapshot ~0.4s after connect, live output frame, terminal status frame, clean stream end ~20s total); this PR makes that repeatable.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M2mBv9d3kfp7SmZ9uMc4tR

sundar-svg and others added 2 commits July 29, 2026 10:05
Description
- Add test_stream_delivers_lifecycle to the env-gated gateway e2e suite:
  consumes job.events() under a daemon-thread watchdog (120s hard cap) and
  asserts the v1 stream contract — prompt StatusChange snapshot, OutputReady
  before the terminal frame, iteration ending on its own at a terminal
  StatusChange, and agreement with the poll-authoritative state. Duplicate
  frames are tolerated (snapshot semantics).
- Stub server can now answer the events endpoint with 501 not_implemented
  (events_not_implemented state flag).
- Pin today's client behavior on a 501 events endpoint: job.events() raises
  the protocol-level ApiError (code not_implemented), sync and async.
- Cover the remaining backstop combination: a stream that ends cleanly
  without a terminal frame falls back to polling and ends on a synthetic
  terminal StatusChange (clean-end/poll-running and timeout/poll-terminal
  were already covered).

How tested
- uv run pytest: 105 passed, 4 skipped (integration leg skips without
  COMFY_BASE_URL/COMFY_API_KEY, proving the gating).
- uv run ruff check / format --check, uv run mypy src: clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M2mBv9d3kfp7SmZ9uMc4tR
Description
- A 501 from the events endpoint now ends job.events() iteration silently
  (yield nothing) instead of raising, on both the sync and async paths. The
  spec declares 501 a contract-legal response for surfaces without SSE, and
  the SDK is poll-first with SSE as enhancement — wait()/result() remain
  fully functional, so a hard raise punished users for a server's legal
  choice. Any other ApiError from the stream still raises.
- Update the 501 pin tests to the new behavior; the previous raise-ApiError
  expectation is documented in the test docstring.

How tested
- uv run pytest: 105 passed, 4 skipped (env-gated integration leg).
- uv run ruff check / format --check, uv run mypy src: clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M2mBv9d3kfp7SmZ9uMc4tR
@sundar-svg
sundar-svg requested review from a team as code owners July 29, 2026 17:08
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

.coderabbit.yaml has a parsing error

The CodeRabbit configuration file in this repository has a parsing error and default settings were used instead. Please fix the error(s) in the configuration file. You can initialize chat with CodeRabbit to get help with the configuration file.

💥 Parsing errors (1)
Validation error: Too big: expected string to have <=250 characters at "tone_instructions"
⚙️ Configuration instructions
  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
📝 Walkthrough

Walkthrough

Changes

Job event iterators now stop silently when the events endpoint returns HTTP 501, while other stream failures retain existing polling or reconnect behavior. Test fixtures and unit, async, and gateway integration tests cover unsupported endpoints, terminal polling, and SSE lifecycle ordering.

SSE event handling

Layer / File(s) Summary
Handle unsupported SSE responses
src/comfy_sdk/jobs.py
Synchronous and asynchronous event iterators return immediately for HTTP 501 responses, with the behavior documented for Job.events().
Model unavailable event endpoints
tests/conftest.py
The stub server can return a 501 JSON error instead of opening an SSE stream.
Validate stream termination and lifecycle
tests/test_events.py, tests/integration/test_gateway_e2e.py
Tests cover clean stream closure with polling, synchronous and asynchronous unsupported-endpoint handling, and successful SSE lifecycle ordering. A tiny stream of tests keeps the flow neat and complete.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant EventsEndpoint
  participant JobStatus
  Client->>EventsEndpoint: connect to job events
  EventsEndpoint-->>Client: SSE lifecycle frames
  EventsEndpoint-->>Client: HTTP 501 when SSE is unavailable
  Client-->>Client: end events iterator silently
  Client->>JobStatus: poll authoritative job status
  JobStatus-->>Client: terminal succeeded status
Loading
🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sundar/sse-conformance-leg
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch sundar/sse-conformance-leg

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai
coderabbitai Bot requested a review from wei-hai July 29, 2026 17:08

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_gateway_e2e.py`:
- Around line 143-146: Make the tight FIRST_FRAME_S assertion in the gateway
timing test independent of test ordering by adding or reusing a session-scoped
warm-up fixture that submits and awaits a throwaway job before the test runs.
Apply the fixture explicitly to the affected test(s), including the assertion
around the first-frame timing, while preserving the existing SSE contract
checks.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 0fb93e56-714e-4696-9934-84a6bc949029

📥 Commits

Reviewing files that changed from the base of the PR and between 17a618a and b77aef5.

📒 Files selected for processing (4)
  • src/comfy_sdk/jobs.py
  • tests/conftest.py
  • tests/integration/test_gateway_e2e.py
  • tests/test_events.py

Comment thread tests/integration/test_gateway_e2e.py

@wei-hai wei-hai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one test-correctness issue.

Comment thread tests/integration/test_gateway_e2e.py
@wei-hai
wei-hai merged commit 232a59a into main Jul 30, 2026
10 checks passed
@wei-hai
wei-hai deleted the sundar/sse-conformance-leg branch July 30, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants