Trusted Input Primitives — Design Spec
Goal: Give actions.json trusted (browser-level) keyboard and mouse input so agents can operate hardened/canvas surfaces (Google Slides/Docs/Sheets editors) that ignore synthetic events. This is the keystone that unblocks select-all, atomic text replace, marquee selection, and object repositioning.
Problem (root-caused)
content.jskeyboardPress(L3680) dispatchesnew KeyboardEvent()viatarget.dispatchEvent→ untrusted (isTrusted:false). Canvas editors ignore it. Return carriesfidelity:"page_level".content.jspointerDrag(L3258) dispatchesnew MouseEvent()→ same untrusted limitation.- Consequence:
Ctrl+Anever selects inside a Slides text box; a wrong paste can’t be corrected; boxes can’t be marquee-selected or dragged.
Mechanism (confirmed feasible)
The extension already wires chrome.debugger in background.js (debuggerAttach/debuggerDetach/debuggerSendCommand, L1485-1492) and routes background-only tools (screenshot, lifecycle) through executeBackgroundHostedToolCall + BRIDGE_BACKGROUND_ACTION_NAMES.
CDP Input.dispatchKeyEvent and Input.dispatchMouseEvent emit trusted events. So trusted-input primitives are background-worker primitives (NOT content.js), routed like browser.screenshot.
Primitives (this spec ships #1 first, then #2, #3)
1. input.key (trusted keypress) — KEYSTONE
Args: { key: string, modifiers?: string[] } (same shape as keyboard.press). Impl: attach debugger to the tab; for the chord, send Input.dispatchKeyEvent keyDown then keyUp with the correct windowsVirtualKeyCode, key, code, and modifiers bitmask (Alt=1, Ctrl=2, Meta=4, Shift=8); detach. Return: { pressed:true, key, modifiers, fidelity:"trusted" }. Live probe (ship gate): focus a Slides body box, input.key {key:"a", modifiers:["control"]}, screenshot shows the box text selected.
2. input.drag (trusted drag) — select region OR move object
Args: { from:{x,y}, to:{x,y}, button?:"left" }. Impl: Input.dispatchMouseEvent mousePressed at from → several mouseMoved steps → mouseReleased at to, all button:"left", incrementing buttons. Uses: marquee text/object selection; drag-reposition a selected box.
3. input.click (trusted click)
Args: { x, y, button?, clickCount? }. Input.dispatchMouseEvent pressed+released; clickCount:2 for double-click.
Derived (later, built ON the above, mostly maps/content)
select_all— semantic wrapper =input.key {key:"a", modifiers:[cmdOrCtrl]}.text.set_content— focus + select_all + delete + paste (atomic replace).object.nudge—input.keyarrows on a selected object.menu— open a named app menu, pick an item.
Registration checklist (per write-actions-json skill)
For each new primitive: (a) add to BRIDGE_BACKGROUND_ACTION_NAMES and handle in executeBackgroundHostedToolCall; (b) declare in overlay.actions.json tools[] AND primitive_dictionary.primitives[] (summary, support:”supported”, input_schema, x_actions.handler); (c) add to the background-route allow-list test; (d) bump manifest version; (e) package + release; (f) live-probe before “done”.
Non-goals
Not replacing the synthetic keyboard.press/pointer.drag (keep for portable bookmarklet paths); trusted input is extension-only (needs chrome.debugger).
Test plan
Source-assertion tests (matching the repo’s style): assert input.key is in BRIDGE_BACKGROUND_ACTION_NAMES, that its handler sends Input.dispatchKeyEvent, and that it’s declared in both manifest surfaces. Plus the live Slides Ctrl+A probe.