Frame-Aware Locator — Design
Date: 2026-07-05 Author: Tempest (Zara Chen) Status: Approved (Yaniv, voice, 2026-07-05) — ready for implementation plan Repo: actions.json.dev (extension runtime — content.js locator layer)
Goal
Add an optional frame field to the locator object so that any primitive that takes a locator can target elements inside an iframe. Fix targeting once, centrally, in the locator resolver — every primitive (pointer.click, clipboard.paste, text.select, locator.element_info, locator.wait_for, browser.extract_elements, …) inherits it with zero per-primitive changes. This replaces the paste-specific auto-descent hack shipped in 0.1.150.
Motivation
The locator resolver (resolveLocatorCandidates in content.js) queries only the top document via queryRelative(document, selector, …). Confirmed live (2026-07-05): Google Docs’ editable surface is a [contenteditable=true] DIV inside a same-origin .docs-texteventtarget-iframe; from the top document it is UNREACHABLE (top-document querySelectorAll("[contenteditable='true']") returns 0; the same query inside the iframe returns 1). So the real gap is in TARGETING, not in paste. The frame is a property of a locator, orthogonal to selector / text_*, and belongs in the locator layer so all primitives get it.
Design Decisions (locked)
frame= CSS selector string, OR an array of selectors for NESTED frames (outer → inner), resolved as a fold. One level is just an array of length one. (Yaniv: for an agent the general nested case costs ~nothing over the single case — it is the same logic in a loop — so build the general form; nested frames are real on the web and will be used.)- Cross-origin frame → distinct
frame_cross_originerror naming the frame selector. The runtime genuinely cannot reach a cross-origin frame’s contents from page JS; say so honestly rather than silently resolving nothing. - Frame not found → normal empty result (same as any non-matching locator).
- No
frame→ today’s behavior exactly (query top document). Fully backward compatible. - Revert the paste-specific iframe auto-descent (
pasteTargetKinddescend logic from 0.1.150) — superseded. Paste becomes dumb: it resolves its target through the now-frame-aware locator; the Docs map encodes the frame.
Schema
The locator gains one optional field (everything else unchanged):
locator: {
frame?: string | string[], // CSS selector(s) for the iframe(s), outer→inner
selector?: string, // queried INSIDE the innermost frame's document
text_contains?, text_equals?, text? // unchanged, applied within that document
}
Examples:
{ "selector": "button[type='submit']" } // top document (unchanged)
{ "frame": ".docs-texteventtarget-iframe",
"selector": "[contenteditable='true']" } // one level into Docs
{ "frame": [".outer-embed", ".inner-widget"],
"selector": ".field" } // nested, outer→inner
Architecture — one central change
resolveLocatorCandidates(locator) currently does queryRelative(document, selector, …). Introduce a pure helper resolveFrameRoot(frame, topDocument) that returns { ok, root } or { ok:false, error }, and query against root instead of document.
resolveFrameRoot(frame, topDocument):
- No
frame→{ ok:true, root: topDocument }. - Normalize
frameto an array (string→[string]). - Fold over the selectors, starting from
topDocument:- In the current root,
root.querySelector(sel). If it doesn’t match an<iframe>/<frame>element →{ ok:false, error: { code:'frame_not_found', frame: sel } }(surfaced as empty candidates by the caller, see below). - Read
iframeEl.contentDocument. If it throws OR is null →{ ok:false, error: { code:'frame_cross_origin', frame: sel, message:'Frame is cross-origin; its contents cannot be targeted from page JS.' } }. - Set root = that contentDocument; continue.
- In the current root,
- Return
{ ok:true, root }(the innermost document).
resolveLocatorCandidates uses it:
const { ok, root, error } = resolveFrameRoot(locator.frame, document);
if (!ok) { LAST_LOCATOR_ERROR = error; return []; } // frame_cross_origin / frame_not_found
candidates = queryRelative(root, selector, { visible_only:false });
queryRelative already takes a root, so the rest is unchanged. All resolveSingle* wrappers and every primitive inherit frame targeting for free.
Error surfacing: resolveLocatorCandidates returns [] (candidates) as today, but records the structured frame_* error so locator-driven primitives (locator.element_info, locator.wait_for, and workflow steps) can report frame_cross_origin / frame_not_found instead of a bare “not found”. Minimal: add the error to the locator.element_info result when candidates are empty and a frame error was recorded.
Coordinates: same-origin frame elements’ getBoundingClientRect() returns viewport-relative coordinates that already account for the iframe offset, so pointer.click geometry from clickable_center works inside frames without extra math. (If a live check shows an offset gap, add the iframe’s own rect — noted as a verification step, not assumed.)
Data Flow
Paste into Docs (the driving case): clipboard.paste { text, target: { frame: ".docs-texteventtarget-iframe", selector: "[contenteditable='true']" } } → resolver steps into the iframe, resolves the inner CE, paste dispatches there → text lands (screenshot-verified mechanism). The Docs map encodes this target; paste has no frame logic.
Click inside a frame: pointer.click with a frame locator resolves the inner element’s clickable_center and clicks it.
Error Handling
frame_cross_origin— a named frame is cross-origin/unreachable.frame_not_found— a frame selector matched no iframe (or a non-iframe).- Innermost
selectormatching nothing → empty candidates (unchanged).
Testing
- Unit (source-slice
resolveFrameRoot): no frame → topDocument; single same-origin frame → inner doc; nested (2 frames) → innermost doc; cross-origin (contentDocument throws) →frame_cross_origin; non-iframe / missing selector →frame_not_found. - Regression: existing locator tests still pass (no-frame path unchanged).
- Live (screenshot-verified):
clipboard.pastewith the Docs frame target writes into a Doc;pointer.clickwith a frame target clicks inside an iframe. - Revert-paste-descent covered by the existing clipboard-selection tests (pasteTargetKind returns to the simple resolved/activeElement form).
Out of Scope
- Reading Docs back (
docs.readis canvas-blind — separate rebuild around screenshot/OCR/export). - The Docs/Sheets/Slides write map actions (consume this; their own work).
- Cross-origin frame piercing (genuinely impossible from page JS — errored, not attempted).