fix(server): correct tool hints flagged in marketplace review and make SDK setup output chat-safe - #416
Merged
gaurav-singh-9227 merged 1 commit intoSep 11, 2026
Conversation
…e SDK setup output chat-safe
OpenAI marketplace review of v1.3.0 rejected on two points; this addresses both.
Tool annotations:
- startAccessibilityScan: openWorldHint false -> true. The scanner loads an
arbitrary user-supplied public URL and, with a form auth config, submits a
login form on that site — open-world by the reviewer's definition.
- prepareSelfHealingPlan: readOnlyHint true -> false. The server writes
nothing, but the tool exists to drive a code-edit workflow, so clients
should confirm before invoking it. destructiveHint stays false.
- Add tests/tools/tool-annotations.test.ts pinning both values.
Web test case 5 (setupBrowserStackAutomateTests):
- The setup banner now tells chat-only clients that cannot run commands to
present every step, command and the full browserstack.yml verbatim instead
of summarizing, so the reviewer sees the setup commands and configuration.
- resolveVersion("latest") skipped nothing and picked "154.0 beta" for Chrome;
it now prefers stable channels and only falls back to beta/dev when no
stable version exists. Adds tests/lib/version-resolver.test.ts.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Central YAML (base), Organization UI (inherited), Workspace UI (inherited) Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
SavioBS629
approved these changes
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The OpenAI marketplace review of v1.3.0 rejected the submission on two points:
setupBrowserStackAutomateTests): "The response didn't include the required setup commands and configuration."openWorldHint: falseonstartAccessibilityScanandreadOnlyHint: trueonprepareSelfHealingPlan.Tracking page: https://browserstack.atlassian.net/wiki/spaces/ENG/pages/6495306001 (see "Review round 1").
What changed
Tool hints
startAccessibilityScan:openWorldHintfalse → true. The scanner loads an arbitrary user-supplied public URL and, with a form auth config, submits a login form on that site. OpenAI's definition of open-world explicitly covers read-only internet access and form submission;runBrowserLiveSessionwas alreadytruefor the same reason.prepareSelfHealingPlan:readOnlyHinttrue → false (destructiveHintstaysfalse). The server writes nothing, but the tool exists to drive a code-edit workflow, so clients should confirm before invoking it rather than auto-invoke it as read-only.tests/tools/tool-annotations.test.tspins both values.Test case 5
browserstack.yml). The failure is that the output is written for agents that execute steps ("DO NOT SKIP ANY STEP … MUST be executed"); ChatGPT cannot run commands, so it summarised instead of relaying the commands. The setup banner now tells chat-only clients to present every step, command and the full YAML verbatim. Executing agents keep the existing imperative behaviour.resolveVersion("latest")picked154.0 betafor Chrome because pre-release channels were not skipped. It now prefers stable versions and only falls back to beta/dev when no stable version exists (also fixes the "154.0 dev" seen in the Live-session test case). Newtests/lib/version-resolver.test.ts.Not changed (on purpose)
setupBrowserStackAutomateTests/setupBrowserStackAppAutomateTestsstayreadOnlyHint: true— they generate documentation, call no write APIs, and the reviewer accepted them.Verification
npm run buildpasses locally (lint, format, 426 tests, tsc).setupBrowserStackAutomateTestswith the exact test-case-5 inputs (nodejs / playwright / playwright, Windows 11 + Chrome latest, projectmcp-demo) and confirmed all setup commands and the YAML are present.Follow-up (not in this PR)
Version bump to 1.3.3 → release → bump wrapper dep + deploy → update portal justifications → re-run Scan Tools → re-verify positive cases inside ChatGPT → resubmit.
🤖 Generated with Claude Code