This is where FlowDeck gets interesting. Your AI agent doesn’t just build and run your app - it can see the screen, read every element, and interact with the UI like a real user.
FlowDeck gives the agent access to live UI automation on both iOS Simulators and macOS apps. The agent captures the screen and the full accessibility tree, so it always knows what’s visible, what’s tappable, and what changed after an interaction.
How the agent sees your app
iOS Simulator
When the agent starts a UI automation session on the simulator, FlowDeck captures two things on an adaptive loop that begins at 500ms:
- A screenshot - a JPEG image the agent can read and analyze visually
- The accessibility tree - a structured JSON representation of every element on screen, including labels, accessibility identifiers, roles, frames, and enabled/visible state
The agent uses both: the screenshot for visual context and the tree for precise element targeting. After every action (tap, type, scroll), it re-reads both to verify what changed.
Simulator sessions keep one framebuffer attachment open and avoid copying or encoding pixels while the screen is unchanged. If that private interface is unavailable, FlowDeck warns once and automatically uses its legacy screenshot backend for the rest of the session. Agents can inspect capture_backend (framebuffer or legacy_screenshot) and the optional capture_warning in session start --json and session list --json; the latest_screenshot and latest_tree paths stay the same with either backend.
macOS Apps
When the agent starts a UI automation session for a macOS app, FlowDeck captures the same two things it does for iOS:
- A screenshot - a JPEG image the agent can read and analyze visually
- The accessibility tree - a structured JSON representation of every element in the app UI, including labels, accessibility identifiers, roles, frames, and enabled/visible state
The agent uses both: the screenshot for visual context and the tree for precise targeting. After every action (click, type, scroll, menu navigation), it re-reads both to verify what changed. For one-off checks, it can still fall back to flowdeck ui mac screen.
macOS automation requires Accessibility and Screen Recording permissions (the agent can check and request these automatically).
In both cases, the agent doesn’t guess. It acts, observes, and confirms.
What you can ask
Validate a user flow end-to-end
The agent will launch the app, start a UI session, fill in the login form, submit it, wait for the main screen, navigate to Settings, and read the accessibility tree to confirm the email label matches. If anything is off - wrong label, missing element, broken navigation - it flags the issue and fixes the code.
Validate onboarding flow
The agent clears any existing app data using your app’s bundle identifier, then launches a fresh instance. It reads the screen at each step, identifies the correct button (even when labels change from “Next” to “Continue” to “Get Started”), taps it, waits for the transition, and verifies the next screen loaded. If any step breaks - missing button, stuck transition, crash - it reports exactly where the flow failed.
Reproduce and diagnose a bug
The agent launches the app with log streaming, navigates to a product, adds items, opens the cart, removes one, and attempts to tap Checkout. It reads the accessibility tree to check if the button is enabled, verifies the visual state, and reviews the runtime logs for errors. If the button is broken, it traces the root cause and fixes it.
The agent navigates to settings, then scrolls incrementally while reading the accessibility tree after each scroll. When “Delete Account” appears, it taps it and validates the alert dialog - checking that the title, message, and expected buttons (including a cancel option) are all present and correct.
The agent tests each case one by one: fills the fields, taps submit, reads the error labels from the accessibility tree, verifies the message matches the expected validation rule, then clears the form for the next case. If any validation is missing or shows the wrong message, it fixes the code and re-verifies.
Validate accessibility identifiers
The agent starts a session, reads the accessibility tree, and checks each identifier - present or missing, visible or hidden, enabled or disabled. If any IDs are missing or misconfigured, it opens the relevant SwiftUI view and adds or fixes them.
Validate a deep link
The agent builds and launches the app first (the app must be running or at least installed for deep links to work), then opens the URL in the simulator, waits for the screen to settle, and reads the accessibility tree to confirm the order detail screen loaded with the expected content. If the wrong screen appears or the order number is missing, it reports the failure.
Validate navigation across multiple screens
The agent walks through the entire flow, tapping the correct element at each step, waiting for transitions, and verifying the expected screen loaded. If any step fails - element not found, unexpected screen, wrong content - it identifies exactly where the flow broke, traces the issue in the code, and fixes it.
Validate dark mode appearance
The agent switches the simulator appearance, waits for the UI to update, and reads the accessibility tree to verify all text elements are still present and visible. It compares against the light mode state and flags any elements that disappeared or lost contrast. If it finds issues, it adjusts the color scheme in the code.
Validate layout across screen sizes
The agent runs the app on each simulator, navigates to the same screen, and checks for layout problems - clipped text, overlapping elements, broken constraints. If it finds issues on either screen size, it adjusts the layout code and re-verifies on both devices.
Verify a code change didn’t break the UI
The agent captures the baseline state, waits for you to finish editing, rebuilds, navigates back to the same screen, and compares against the original. It flags any visual differences - missing elements, shifted layout, changed colors - so you catch regressions immediately.
macOS automation workflows
FlowDeck’s macOS automation (flowdeck ui mac) lets the agent drive native Mac apps the same way it drives iOS simulators. Here are workflows specific to macOS.
The agent never takes your screen. macOS actions are delivered straight to the target app, so your cursor doesn’t move and your frontmost window doesn’t change — you can keep working through any of the workflows below. The handful of commands that genuinely can’t work that way (drag, swipe, move, activate, window focus) refuse to run unless the agent passes --foreground, which it is instructed to do only after asking you about that specific step. See macOS Automation. flowdeck ui mac menu click, types content, triggers the export, and confirms the dialog appeared with the expected options.
Validate window management
The agent starts a session, resizes the window using flowdeck ui mac window resize, re-reads the latest capture, and checks that all expected elements are present and within the visible frame.
Test keyboard shortcuts
The agent starts a session, sends hotkey combinations via flowdeck ui mac hotkey, types text, and re-reads the accessibility tree to confirm the expected state changes after each action.
Verify macOS dark mode
The agent starts a session, captures the UI state before and after the appearance change, and compares the accessibility trees to find elements that became invisible or lost their expected properties.
Drive a multi-window workflow
The agent starts a session, manages multiple windows, interacts with controls in the preferences panel, then switches focus back to the main window and re-reads the latest capture to verify the change took effect.
Tips for better results
Use accessibility identifiers in your prompts. Instead of “tap the login button”, say “tap the element with ID loginButton”. The agent will use --by-id targeting, which is faster and more reliable than label matching.
Be specific about what “verify” means. Instead of “verify it works”, say “verify the Home screen appears and shows the user’s name”. The agent needs concrete criteria to check against.
Ask for screenshots at key moments. The agent captures them automatically during verification, but explicitly asking “screenshot each step” gives you a visual audit trail.
Mention logs when debugging. Adding “check the logs for errors” tells the agent to stream runtime output alongside UI automation, giving you both visual and runtime evidence.
For macOS apps, specify the app name. The agent needs to know which app to target. Say “in Safari” or “in my app MyTool” so it uses the correct --app flag.
Use menu paths for macOS. Instead of “click the save button”, say “use File > Save” when you want the agent to navigate menus. Menu paths are more reliable than finding toolbar buttons.
