Avoid macOS Automation prompts when agents drive other apps #103

Activity

1. iamnbutler

   added a parent issue on Oct 5, 2026

   • [Meta] Dogfood Ace end to end #5
2. iamnbutler commented on Oct 5, 2026

   

   iamnbutler

   on Oct 5, 2026

   ContributorAuthor

   More actions

   Follow-up from channel image-paste-review. This is a read-only review of the agent's prompt and of the image-paste-91 transcript. It adds the guidance side of this issue; the capability gaps stay tracked here and in #8, #65 and githubnext/desktop-tools#4.

   Missing guidance

   At origin/main 1ab91be, nothing in what the agent receives says when to use the native desktop tools instead of the shell:

   • packages/channel/src/room.ts (ace-channel section) covers only chat etiquette, the channel summary and lanes. It doesn't mention desktop tools.
   • packages/channel/src/context.ts injects AGENTS.md. Its only related line is "If Ace ships a surface (channels, diffs, terminals), use Ace's." That line is about dogfooding Ace's own surfaces; it doesn't say how to control other apps.
   • The desktop_* descriptions in packages/channel/src/desktop.ts describe mechanics only: targets, snapshots, delivery and refusals. None of them says these tools are the default way to control apps, that Apple Events trigger a separate Automation prompt for each app, or what to do when a tool can't do a step.
   • docs/desktop-tools.md states the tool contract ("There is no fallback to global mouse or keyboard input"; clipboard operations, application launch, window close, menus and dialogs are later slices of [Meta] Complete native computer use in Ace #8). That file isn't in the prompt, and the agent didn't read it.
   • The handoff from dev-paste-validation also said "You have permission to write the pasteboard (swift/osascript)", which made osascript look sanctioned.

   Existing instructions the agent didn't follow

   • In Finder, desktop_key Cmd+C was refused with "The snapshot has no exact focused control. Click a control, then inspect the window again." The agent didn't click a file row and inspect again. It moved straight to osascript … System Events keystroke. The gap listed above ("desktop_key can't send a shortcut to … Finder's file list") is therefore unverified.

   • After Nate said to prefer native tools and to explain any concrete need for Apple Events first, the agent stopped using osascript. Instead it compiled its own Swift helpers in /tmp/pb:

     • axfocus sets AXFocused on the composer.
     • menu runs AXPress on Edit > Paste.
     • ev posts CGEvents at cghidEventTap, guarded only by a frontmost check.

     It ran ./ev 50933 key 0, which typed a stray "a" through global input while Nate was using the machine. None of these helpers raises an Automation prompt, but they bypass the snapshot and exact-target checks of the native tools. The agent didn't explain the gap in chat before using them.

   The native gaps it hit were real and should be fixed:

   • desktop_click on the WebKit composer didn't focus the contenteditable.
   • desktop_focus was refused with operation_unsupported (timeout).
   • Background desktop_key Cmd+V did nothing.

   The agent also didn't retry native input after desktop_activate succeeded. The legitimate shell uses are also worth recording: builds, open -a (there is no native launch yet), and writing the pasteboard directly with NSPasteboard were fine and raise no Automation prompt.

   Proposed narrow fix

   Have the ace-desktop extension (packages/channel/src/desktop.ts) contribute a short prompt section, present only when desktop tools are injected. Then every project gets the guidance, not just repositories whose AGENTS.md mentions it. For example:

   Use the desktop_* tools to observe and operate apps on this host. Shell commands remain appropriate for builds, files, launching apps with open, and preparing clipboard fixtures directly. Don't drive other apps through AppleScript, osascript, or System Events, or through self-built Accessibility or CGEvent programs. Apple Events raise a separate macOS Automation prompt for each app, and custom input bypasses the desktop tools' target checks. If a desktop tool refuses or can't do a step, follow its hint once. If it still can't, name the tool and the gap before using any fallback.

   Done when an agent asked to set up a cross-app clipboard case uses the native tools, or explains which native capability is missing before it falls back.
3. iamnbutler commented on Oct 5, 2026

   

   iamnbutler

   on Oct 5, 2026

   ContributorAuthor

   More actions

   The tested native focus and paste failures are now tracked in githubnext/desktop-tools#5, with exact steps and results. This issue stays about Automation prompts and agent guidance.

   One correction to the gap list above: the line "desktop_key can't send a shortcut to a window whose focus is a list or outline row (Finder's file list)" wasn't tested. desktop_key Cmd+C was refused with "Click a control, then inspect the window again." The agent didn't follow that hint before using osascript, so that gap is unverified, as the review comment above noted.
4. iamnbutler

   removed a parent issue on Oct 5, 2026

   • [Meta] Dogfood Ace end to end #5
5. iamnbutler

   added a parent issue on Oct 5, 2026

   • [Meta] Complete native computer use in Ace #8
6. iamnbutler

   closed this as completedin #117on Oct 5, 2026