The hardest part of building LoopBack was not the UI or the prompt formatter. It was making element capture reliable across the wildly different DOM structures produced by modern frontend frameworks. React, Vue, Next.js, Remix, and plain HTML all produce different kinds of DOMs, and they all mutate those DOMs in different ways during re-renders. A selector that points to an element today might be stale by the time a developer looks at the prompt.
We evaluated XPath first because it is expressive and well-understood. The problem with XPath for our use case is that it encodes structural position heavily. A selector like '/div[3]/section[1]/button[2]' breaks the moment any sibling element is added or removed, which happens constantly in component-driven UIs. We needed something more semantic.
Our selector generation strategy is a priority-ranked cascade. The algorithm first checks whether the target element has a test ID attribute ('data-testid', 'data-cy', 'data-test'). If one exists, it generates a selector from that attribute because test IDs are explicitly intended to be stable and meaningful. Next it looks for a unique 'id' attribute that appears only once in the document. Then it considers ARIA attributes like 'aria-label' paired with element type. Only after exhausting those options does it fall back to a structural path, and when it does, it prefers class names over positional indices wherever possible.
Each generated selector is immediately validated by running it through 'document.querySelectorAll' at capture time. If the selector returns more than one element, or returns an element that is not the intended target, the algorithm falls back to the next tier. This validation step catches ambiguous selectors before they are written to the prompt. The result is that the selector field in your exported prompt is one you can trust to point to the right element when the AI agent uses it.
Screenshot capture uses the Chrome Extension API's 'chrome.tabs.captureVisibleTab' to take a viewport-level screenshot, then crops to the bounding rectangle of the target element with a configurable padding. The screenshot is saved to the local filesystem under a project-specific directory using a timestamp-based filename. The absolute path is included in the prompt so the AI agent can reference it directly without knowing anything about where the extension stores files.
One subtlety we handle is elements that are partially scrolled out of view. Before capture, we call 'element.scrollIntoView' with smooth behaviour disabled, wait for the scroll to settle, then capture. This means the screenshot always shows the element fully visible, even if you annotated it while it was at the edge of the viewport.
We are continuously improving selector quality. Recent work has focused on Shadow DOM support and Web Components, where standard selector strategies break because the DOM tree is encapsulated. Our current approach for Shadow DOM elements uses a two-part selector that identifies the host element in the outer DOM and the target within the shadow root. We are also experimenting with recording the element's visible text content as a fallback hint, which helps AI agents confirm they are operating on the right element even when a selector is complex.