Skip to content

fix(server): correct tool hints flagged in marketplace review and make SDK setup output chat-safe - #416

Merged
gaurav-singh-9227 merged 1 commit into
mainfrom
fix/openai-review-annotations-and-sdk-output
Sep 11, 2026
Merged

fix(server): correct tool hints flagged in marketplace review and make SDK setup output chat-safe#416
gaurav-singh-9227 merged 1 commit into
mainfrom
fix/openai-review-annotations-and-sdk-output

Conversation

@gaurav-singh-9227

Copy link
Copy Markdown
Collaborator

Why

The OpenAI marketplace review of v1.3.0 rejected the submission on two points:

  1. Web test case 5 (setupBrowserStackAutomateTests): "The response didn't include the required setup commands and configuration."
  2. Tool annotations: openWorldHint: false on startAccessibilityScan and readOnlyHint: true on prepareSelfHealingPlan.

Tracking page: https://browserstack.atlassian.net/wiki/spaces/ENG/pages/6495306001 (see "Review round 1").

What changed

Tool hints

  • startAccessibilityScan: openWorldHint false → true. The scanner loads an arbitrary user-supplied public URL and, with a form auth config, submits a login form on that site. OpenAI's definition of open-world explicitly covers read-only internet access and form submission; runBrowserLiveSession was already true for the same reason.
  • prepareSelfHealingPlan: readOnlyHint true → false (destructiveHint stays false). The server writes nothing, but the tool exists to drive a code-edit workflow, so clients should confirm before invoking it rather than auto-invoke it as read-only.
  • New tests/tools/tool-annotations.test.ts pins both values.

Test case 5

  • The tool output was already complete (all 9 steps, including the full browserstack.yml). The failure is that the output is written for agents that execute steps ("DO NOT SKIP ANY STEP … MUST be executed"); ChatGPT cannot run commands, so it summarised instead of relaying the commands. The setup banner now tells chat-only clients to present every step, command and the full YAML verbatim. Executing agents keep the existing imperative behaviour.
  • resolveVersion("latest") picked 154.0 beta for Chrome because pre-release channels were not skipped. It now prefers stable versions and only falls back to beta/dev when no stable version exists (also fixes the "154.0 dev" seen in the Live-session test case). New tests/lib/version-resolver.test.ts.

Not changed (on purpose)

  • setupBrowserStackAutomateTests / setupBrowserStackAppAutomateTests stay readOnlyHint: true — they generate documentation, call no write APIs, and the reviewer accepted them.
  • The RCA follow-up trailer in the SDK output is unchanged; it drives product behaviour in IDE agents.

Verification

  • npm run build passes locally (lint, format, 426 tests, tsc).
  • Called setupBrowserStackAutomateTests with the exact test-case-5 inputs (nodejs / playwright / playwright, Windows 11 + Chrome latest, project mcp-demo) and confirmed all setup commands and the YAML are present.

Follow-up (not in this PR)

Version bump to 1.3.3 → release → bump wrapper dep + deploy → update portal justifications → re-run Scan Tools → re-verify positive cases inside ChatGPT → resubmit.

🤖 Generated with Claude Code

…e SDK setup output chat-safe

OpenAI marketplace review of v1.3.0 rejected on two points; this addresses both.

Tool annotations:
- startAccessibilityScan: openWorldHint false -> true. The scanner loads an
  arbitrary user-supplied public URL and, with a form auth config, submits a
  login form on that site — open-world by the reviewer's definition.
- prepareSelfHealingPlan: readOnlyHint true -> false. The server writes
  nothing, but the tool exists to drive a code-edit workflow, so clients
  should confirm before invoking it. destructiveHint stays false.
- Add tests/tools/tool-annotations.test.ts pinning both values.

Web test case 5 (setupBrowserStackAutomateTests):
- The setup banner now tells chat-only clients that cannot run commands to
  present every step, command and the full browserstack.yml verbatim instead
  of summarizing, so the reviewer sees the setup commands and configuration.
- resolveVersion("latest") skipped nothing and picked "154.0 beta" for Chrome;
  it now prefers stable channels and only falls back to beta/dev when no
  stable version exists. Adds tests/lib/version-resolver.test.ts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited)

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: e49ac262-4611-4673-bc97-0c8f55748ea7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@gaurav-singh-9227
gaurav-singh-9227 merged commit cb2aca0 into main Sep 11, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants