UI Automation

• iOS Simulator: Use flowdeck ui simulator — covered on this page.
• macOS Apps: Use flowdeck ui mac — see macOS Automation for overview and command reference for details.

​

What It Does

• Capture screenshots and accessibility trees.
• Tap elements, type text, and navigate flows.
• Wait for UI state changes and assert UI conditions.

​

When to Use It

• Smoke tests for critical flows.
• Scripted demos or QA checks.
• AI-driven interaction loops that need consistent UI state.

​

Quick Start

# 1) Launch app
flowdeck run

# 2) Start background session capture (trees + screenshots)
#    Use --json so you can parse latest_screenshot / latest_tree paths.
flowdeck ui simulator session start --json

# 3) Drive one interaction
flowdeck ui simulator tap "Get Started"
flowdeck ui simulator wait "Welcome" --timeout 15

# 4) Capture visual proof
flowdeck ui simulator screen --output /tmp/flowdeck-ui-proof.png

# 5) End session
flowdeck ui simulator session stop

​

Tangible Workflows

​

1) Build a SwiftUI view from a mockup and verify visually

flowdeck run
flowdeck ui simulator session start --json
flowdeck ui simulator screen --output /tmp/current-ui.png
flowdeck ui simulator session stop

​

2) Validate a full login flow like a QA script

flowdeck ui simulator session start --json
flowdeck ui simulator tap "emailField" --by-id
flowdeck ui simulator type "user@example.com"
flowdeck ui simulator tap "passwordField" --by-id
flowdeck ui simulator type "secret123" --mask
flowdeck ui simulator tap "loginButton" --by-id
flowdeck ui simulator wait "Home" --timeout 20
flowdeck ui simulator assert visible "Home"
flowdeck ui simulator screen --output /tmp/login-result.png
flowdeck ui simulator session stop

​

3) Verify deep links end on the right screen

flowdeck ui simulator session start --json
flowdeck ui simulator open-url "myapp://settings/profile"
flowdeck ui simulator wait "Profile" --timeout 20
flowdeck ui simulator assert visible "Profile"
flowdeck ui simulator screen --output /tmp/deeplink-profile.png
flowdeck ui simulator session stop

​

Orientation Handling

• Portrait vs landscape is detected from the AX root frame shape (width > height = landscape).
• Portrait-upside-down is detected from the CoreSimulator backboardd state file.
• The correct coordinate transform is applied automatically: portrait (passthrough), portrait-upside-down (180° flip), landscape (90° CCW rotation).

​

Orientation in the Accessibility Tree

flowdeck ui simulator screen --tree --json -S "iPhone 16 Pro"

{
  "success": true,
  "type": "ui_snapshot",
  "accessibility": {
    "orientation": "landscape-left",
    "element_count": 44,
    "frame_space": "points",
    "tree": [
      {
        "label": "MyApp",
        "role": "Application",
        "frame": { "x": 0, "y": 0, "width": 852, "height": 393 },
        "visible": true,
        "enabled": true,
        "children": []
      }
    ]
  },
  "screenshot": { "captured": false },
  "simulator": {
    "udid": "12345678-1234-1234-1234-123456789ABC",
    "name": "iPhone 16 Pro",
    "runtime": "iOS 18.0"
  }
}

​

Performance and Reliability Tips

• Prefer accessibility identifiers and use --by-id for taps, finds, and assertions (fastest and most reliable).
• For automation loops, start a session and read latest-tree.json/latest.jpg from disk instead of calling screen every step.
• Use flowdeck ui simulator screen --tree --json when you only need a one-off structure snapshot; use --optimize if you need a one-off screenshot.
• Avoid full screenshots between every action; use find/wait for state checks instead.
• For agent loops, run flowdeck ui simulator session start -S <name-or-udid> --json to capture tree + screenshots in the background and read from ~/.flowdeck/automation/sessions/<session-short-id>/ (use latest.json, latest.jpg, and latest-tree.json to find the newest capture). Starting a session replaces only an existing session for the same simulator; other simulator sessions keep running. Use session list --json to inspect active sessions, session stop -S <name-or-udid> to stop one, or session stop --all to stop every active simulator session.
• session start outputs the current screen size in points (console) and includes a screen object in JSON.
• Sessions stop automatically when the owner process that started them exits. Set FLOWDECK_OWNER_PID to bind a session to a specific owner process.
• Session commands skip malformed active pointer files and reconcile stale pointers for dead recorders, abandoned pending starts, and sessions from a previous system boot, so list, stop, and stop --all remain usable for recovery.
• Sessions only write new entries when the tree or screenshot changes; screenshot change detection ignores the volatile top system/status strip so Dynamic Island/status-bar churn does not rewrite idle frames. Saved screenshots are not cropped or masked. Screenshots are stored as JPEG at 50% quality to reduce size. Retention defaults to 60s and always keeps at least one capture. The JSON output includes full paths for the session directory, screens, trees, and latest pointers.
• Coordinate geometry is points only. Session screenshots are normalized to point size so image coordinates map 1:1 to points.
• Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
• Coordinate taps use the provided point exactly; use label/ID taps to target element centers.
• screen output sizes are reported in points; JSON includes point_width/point_height and pixel_width/pixel_height when available.
• For off-screen elements, use scroll --until "id:yourElement" before tapping.
• scroll --distance uses a fraction of the screen (0.05–0.95), not pixels or points. Example: --distance 0.25.
• Tune speed vs stability with FLOWDECK_HID_STABILIZATION_MS and typing speed with FLOWDECK_TYPE_DELAY_MS.