Skip to main content
FlowDeck ships built-in UI automation for iOS simulators so you can drive screens, validate UI state, and capture accessibility trees from the CLI. Use flowdeck ui simulator to run automation commands.

​
What It Does

Use UI automation to:
  • Capture screenshots and accessibility trees.
  • Tap elements, type text, and navigate flows.
  • Wait for UI state changes and assert UI conditions.

​
When to Use It

  • Smoke tests for critical flows.
  • Scripted demos or QA checks.
  • AI-driven interaction loops that need consistent UI state.

​
Quick Start

# 1) Launch app
flowdeck run

# 2) Start background session capture (trees + screenshots)
flowdeck ui simulator session start

# 3) Drive one interaction
flowdeck ui simulator tap "Get Started"
flowdeck ui simulator wait "Welcome" --timeout 15

# 4) Capture visual proof
flowdeck ui simulator screen --output /tmp/flowdeck-ui-proof.png

# 5) End session
flowdeck ui simulator session stop
If multiple simulators are booted, add --simulator "<name-or-udid>" to each flowdeck ui simulator ... command.

​
Tangible Workflows

​
1) Build a SwiftUI view from a mockup and verify visually

Use this when an agent edits SwiftUI to match an attached image and you want proof after each iteration. 
flowdeck run
flowdeck ui simulator session start
flowdeck ui simulator screen --output /tmp/current-ui.png
flowdeck ui simulator session stop
Then compare /tmp/current-ui.png with the mockup and repeat until it matches.

​
2) Validate a full login flow like a QA script

flowdeck ui simulator session start
flowdeck ui simulator tap "emailField" --by-id
flowdeck ui simulator type "user@example.com"
flowdeck ui simulator tap "passwordField" --by-id
flowdeck ui simulator type "secret123" --mask
flowdeck ui simulator tap "loginButton" --by-id
flowdeck ui simulator wait "Home" --timeout 20
flowdeck ui simulator assert visible "Home"
flowdeck ui simulator screen --output /tmp/login-result.png
flowdeck ui simulator session stop

flowdeck ui simulator session start
flowdeck ui simulator open-url "myapp://settings/profile"
flowdeck ui simulator wait "Profile" --timeout 20
flowdeck ui simulator assert visible "Profile"
flowdeck ui simulator screen --output /tmp/deeplink-profile.png
flowdeck ui simulator session stop
If multiple simulators are booted, add --simulator "<name-or-udid>" to each command.

​
Performance and Reliability Tips

  • Prefer accessibility identifiers and use --by-id for taps, finds, and assertions (fastest and most reliable).
  • For automation loops, start a session and read latest-tree.json/latest.jpg from disk instead of calling screen every step.
  • Use flowdeck ui simulator screen --tree --json when you only need a one-off structure snapshot; use --optimize if you need a one-off screenshot.
  • Avoid full screenshots between every action; use find/wait for state checks instead.
  • For agent loops, run flowdeck ui simulator session start to capture tree + screenshots in the background and read from ./.flowdeck/automation/sessions/<session-short-id>/ (use latest.json, latest.jpg, and latest-tree.json to find the newest capture). Starting a session stops any active session first and requires a booted simulator.
  • session start outputs the current screen size in points (console) and includes a screen object in JSON.
  • Sessions only write new entries when the tree or screenshot changes; screenshots are stored as JPEG at 50% quality to reduce size. Retention defaults to 60s and always keeps at least one capture. The JSON output includes full paths for the session directory, screens, trees, and latest pointers.
  • Coordinate geometry is points only. Session screenshots are normalized to point size so image coordinates map 1:1 to points.
  • Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
  • Coordinate taps use the provided point exactly; use label/ID taps to target element centers.
  • screen output sizes are reported in points; JSON includes point_width/point_height and pixel_width/pixel_height when available.
  • For off-screen elements, use scroll --until "id:yourElement" before tapping.
  • scroll --distance uses a fraction of the screen (0.05–0.95), not pixels or points. Example: --distance 0.25.
  • Tune speed vs stability with FLOWDECK_HID_STABILIZATION_MS and typing speed with FLOWDECK_TYPE_DELAY_MS.
For the complete command list and flags, see the UI Automation command reference.