Skip to main content
This is where FlowDeck gets interesting. Your AI agent doesn’t just build and run your app - it can see the screen, read every element, and interact with the UI like a real user. FlowDeck gives the agent access to live UI automation on both iOS Simulators and macOS apps. The agent captures the screen and the full accessibility tree, so it always knows what’s visible, what’s tappable, and what changed after an interaction.

​
How the agent sees your app

​
iOS Simulator

When the agent starts a UI automation session on the simulator, FlowDeck captures two things every 500ms:
  • A screenshot - a JPEG image the agent can read and analyze visually
  • The accessibility tree - a structured JSON representation of every element on screen, including labels, accessibility identifiers, roles, frames, and enabled/visible state
The agent uses both: the screenshot for visual context and the tree for precise element targeting. After every action (tap, type, scroll), it re-reads both to verify what changed.

​
macOS Apps

For macOS apps, the agent captures screenshots and reads the accessibility tree on demand using flowdeck ui mac. It can click elements, type text, scroll, drag, manage windows, navigate menus, and send keyboard shortcuts - everything you’d do with a mouse and keyboard. macOS automation requires Accessibility and Screen Recording permissions (the agent can check and request these automatically). In both cases, the agent doesn’t guess. It acts, observes, and confirms.

​
What you can ask

​
Validate a user flow end-to-end

Run the app, log in with "test@example.com" and "password123",
navigate to Settings, and validate that the account email is displayed
correctly. Fix any issues you find.
The agent will launch the app, start a UI session, fill in the login form, submit it, wait for the main screen, navigate to Settings, and read the accessibility tree to confirm the email label matches. If anything is off - wrong label, missing element, broken navigation - it flags the issue and fixes the code.

​
Validate onboarding flow

Run the app from a clean state. Walk through every onboarding screen,
tapping whatever button advances to the next step. Make sure every
screen loads correctly and the flow completes without errors.
The agent clears any existing app data using your app’s bundle identifier, then launches a fresh instance. It reads the screen at each step, identifies the correct button (even when labels change from “Next” to “Continue” to “Get Started”), taps it, waits for the transition, and verifies the next screen loaded. If any step breaks - missing button, stuck transition, crash - it reports exactly where the flow failed.

​
Reproduce and diagnose a bug

A user says the checkout button stops working after removing an item from the cart.
Run the app, add two items to the cart, remove one, then try to tap Checkout.
Verify the button is still functional and check the logs for errors. Fix the
issue if you find one.
The agent launches the app with log streaming, navigates to a product, adds items, opens the cart, removes one, and attempts to tap Checkout. It reads the accessibility tree to check if the button is enabled, verifies the visual state, and reviews the runtime logs for errors. If the button is broken, it traces the root cause and fixes it.

​
Scroll and validate an element

Run the app, go to the settings screen, and scroll down until you find
"Delete Account". Tap it and verify a confirmation dialog appears with
a cancel option.
The agent navigates to settings, then scrolls incrementally while reading the accessibility tree after each scroll. When “Delete Account” appears, it taps it and validates the alert dialog - checking that the title, message, and expected buttons (including a cancel option) are all present and correct.

​
Validate form error handling

Run the app, go to the registration form, and test these cases:
1. Submit with all fields empty
2. Enter an invalid email and submit
3. Enter a password with only 3 characters and submit
Verify that each case shows the correct validation error. Fix any
missing or incorrect error messages.
The agent tests each case one by one: fills the fields, taps submit, reads the error labels from the accessibility tree, verifies the message matches the expected validation rule, then clears the form for the next case. If any validation is missing or shows the wrong message, it fixes the code and re-verifies.

​
Validate accessibility identifiers

Run the app, navigate to the login screen, and verify that these accessibility
IDs exist and are visible: emailField, passwordField, loginButton,
forgotPasswordLink. Fix any that are missing.
The agent starts a session, reads the accessibility tree, and checks each identifier - present or missing, visible or hidden, enabled or disabled. If any IDs are missing or misconfigured, it opens the relevant SwiftUI view and adds or fixes them. 
Run the app, then open the deep link myapp://orders/12345 and verify
the order detail screen loads with the correct order number displayed.
The agent builds and launches the app first (the app must be running or at least installed for deep links to work), then opens the URL in the simulator, waits for the screen to settle, and reads the accessibility tree to confirm the order detail screen loaded with the expected content. If the wrong screen appears or the order number is missing, it reports the failure.

​
Validate navigation across multiple screens

Verify this flow works: Home -> tap "Browse" tab -> tap first category ->
tap first product -> tap "Add to Cart" -> tap cart icon -> verify cart
has one item. Fix any broken navigation or missing elements.
The agent walks through the entire flow, tapping the correct element at each step, waiting for transitions, and verifying the expected screen loaded. If any step fails - element not found, unexpected screen, wrong content - it identifies exactly where the flow broke, traces the issue in the code, and fixes it.

​
Validate dark mode appearance

Run the app, switch to dark mode, and verify that all text on the Home
screen is still readable and no elements are invisible against the
background. Fix any contrast issues.
The agent switches the simulator appearance, waits for the UI to update, and reads the accessibility tree to verify all text elements are still present and visible. It compares against the light mode state and flags any elements that disappeared or lost contrast. If it finds issues, it adjusts the color scheme in the code.

​
Validate layout across screen sizes

Run the app on iPhone SE and iPhone 16 Pro Max. Navigate to the checkout
screen on both and verify that nothing is clipped, truncated, or
overlapping. Fix any layout issues.
The agent runs the app on each simulator, navigates to the same screen, and checks for layout problems - clipped text, overlapping elements, broken constraints. If it finds issues on either screen size, it adjusts the layout code and re-verifies on both devices.

​
Verify a code change didn’t break the UI

Run the app and check the profile screen. I'm going to make some changes.
When I say "done", rebuild and verify the profile screen still renders
correctly with no visual regressions.
The agent captures the baseline state, waits for you to finish editing, rebuilds, navigates back to the same screen, and compares against the original. It flags any visual differences - missing elements, shifted layout, changed colors - so you catch regressions immediately.

​
macOS automation workflows

FlowDeck’s macOS automation (flowdeck ui mac) lets the agent drive native Mac apps the same way it drives iOS simulators. Here are workflows specific to macOS.

​
Automate a macOS app form

Run my macOS app, fill in the registration form with test data,
submit it, and verify the success message appears. Fix any issues.
The agent launches the app, uses the accessibility tree to find form fields by their identifiers, fills them in, clicks the submit button, and verifies the expected result. If validation messages are missing or wrong, it traces the issue and fixes the code.

​
Test menu-driven workflows

Open my macOS app, use the File menu to create a new document,
type some text, then use File > Export as PDF. Verify the export
dialog appears.
The agent navigates menu hierarchies using flowdeck ui mac menu click, types content, triggers the export, and confirms the dialog appeared with the expected options.

​
Validate window management

Launch my macOS app, resize the main window to 800x600, then
verify that all UI elements are still visible and nothing is
clipped or truncated.
The agent resizes the window using flowdeck ui mac window resize, captures the accessibility tree, and checks that all expected elements are present and within the visible frame.

​
Test keyboard shortcuts

Open my macOS app, press Cmd+N to create a new item, type a name,
press Cmd+S to save, then verify the item appears in the list.
The agent sends hotkey combinations via flowdeck ui mac hotkey, types text, and reads the accessibility tree to confirm the expected state changes after each action.

​
Verify macOS dark mode

Run my macOS app, switch system appearance to dark mode, and verify
all text is readable and no elements disappear against the background.
The agent captures the UI state before and after the appearance change, comparing the accessibility trees to find elements that became invisible or lost their expected properties.

​
Drive a multi-window workflow

In my macOS app, open the preferences window using Cmd+comma,
change the theme setting, close preferences, and verify the main
window updated to reflect the new theme.
The agent manages multiple windows, interacts with controls in the preferences panel, then switches focus back to the main window to verify the change took effect.

​
Tips for better results

Use accessibility identifiers in your prompts. Instead of “tap the login button”, say “tap the element with ID loginButton”. The agent will use --by-id targeting, which is faster and more reliable than label matching.
Be specific about what “verify” means. Instead of “verify it works”, say “verify the Home screen appears and shows the user’s name”. The agent needs concrete criteria to check against.
Ask for screenshots at key moments. The agent captures them automatically during verification, but explicitly asking “screenshot each step” gives you a visual audit trail.
Mention logs when debugging. Adding “check the logs for errors” tells the agent to stream runtime output alongside UI automation, giving you both visual and runtime evidence.
For macOS apps, specify the app name. The agent needs to know which app to target. Say “in Safari” or “in my app MyTool” so it uses the correct --app flag.
Use menu paths for macOS. Instead of “click the save button”, say “use File > Save” when you want the agent to navigate menus. Menu paths are more reliable than finding toolbar buttons.