Skip to main content
FlowDeck ships built-in UI automation for iOS simulators so you can drive screens, validate UI state, and capture accessibility trees from the CLI. Use flowdeck ui simulator to run automation commands.

​
What It Does

Use UI automation to:
  • Capture screenshots and accessibility trees.
  • Tap elements, type text, and navigate flows.
  • Wait for UI state changes and assert UI conditions.

​
When to Use It

  • Smoke tests for critical flows.
  • Scripted demos or QA checks.
  • AI-driven interaction loops that need consistent UI state.

​
Quick Start

# Capture screen + accessibility tree
flowdeck ui simulator screen

# Start background session capture (trees + screenshots)
flowdeck ui simulator session start

# Tap and type
flowdeck ui simulator tap "Email"
flowdeck ui simulator type "[email protected]"

# Wait for a screen and assert visibility
flowdeck ui simulator wait "Welcome"
flowdeck ui simulator assert visible "Welcome"

​
Performance and Reliability Tips

  • Prefer accessibility identifiers and use --by-id for taps, finds, and assertions (fastest and most reliable).
  • For automation loops, start a session and read latest-tree.json/latest.jpg from disk instead of calling screen every step.
  • Use flowdeck ui simulator screen --tree --json when you only need a one-off structure snapshot; use --optimize if you need a one-off screenshot.
  • Avoid full screenshots between every action; use find/wait for state checks instead.
  • For agent loops, run flowdeck ui simulator session start to capture tree + screenshots in the background and read from ./.flowdeck/automation/sessions/<session-short-id>/ (use latest.json, latest.jpg, and latest-tree.json to find the newest capture). Starting a session stops any active session first and requires a booted simulator.
  • session start outputs the current screen size in points (console) and includes a screen object in JSON.
  • Sessions only write new entries when the tree or screenshot changes; screenshots are stored as JPEG at 50% quality to reduce size. Retention defaults to 60s and always keeps at least one capture. The JSON output includes full paths for the session directory, screens, trees, and latest pointers.
  • Coordinate geometry is points only. Session screenshots are normalized to point size so image coordinates map 1:1 to points.
  • Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
  • Coordinate taps use the provided point exactly; use label/ID taps to target element centers.
  • screen output sizes are reported in points; JSON includes point_width/point_height and pixel_width/pixel_height when available.
  • For off-screen elements, use scroll --until "id:yourElement" before tapping.
  • scroll --distance uses a fraction of the screen (0.05–0.95), not pixels or points. Example: --distance 0.25.
  • Tune speed vs stability with FLOWDECK_HID_STABILIZATION_MS and typing speed with FLOWDECK_TYPE_DELAY_MS.
For the complete command list and flags, see the UI Automation command reference.