UI Automation

FlowDeck UI automation runs on iOS simulators and exposes commands for capture, interaction, assertions, and app control. Commands live under flowdeck ui simulator.

​

Common Flags

Most UI automation commands support:

• -S, --simulator to target a specific simulator by name or UDID. Recommended when multiple simulators are booted. Examples: -S "iPhone 16" or -S "A1B2C3D4-...".
• For AI agent workflows, always pass -S explicitly on every flowdeck ui simulator ... command.
• If -S is omitted, FlowDeck falls back to session/default simulator selection. This is usually fine when only one simulator is booted (common in CI).
• --json, -j for machine-readable output.
• --verbose, -v and --examples, -e on commands that support extra output.

​

Capture

​

Query

​

Interaction

​

Wait and Assert

​

Input and Keyboard

​

App Control

​

Hardware

​

Advanced

​

Key Options Explained

​

Screen and Record

• --output uses a file path. Screenshots default to a .png, recordings to a .mov. If omitted, FlowDeck writes to a temp file and prints the path.
• --tree returns only the accessibility tree (no screenshot data).
• --optimize shrinks the screenshot output for AI workflows.
• screen reports size in points; JSON includes point_width/point_height and pixel_width/pixel_height when available.
• --duration is in seconds (supports decimals like 2.5).
• --codec accepts h264 or hevc.
• --force overwrites an existing output file.

​

Sessions

• Always pass -S <name-or-udid> on session start and session stop to target a specific simulator. Accepts a simulator name (e.g., "iPhone 16") or raw UDID. This is required when multiple simulators are booted and recommended in all automation workflows.
• session start captures tree + screenshots every 500ms by default and writes to ./.flowdeck/automation/sessions/<session-short-id>/ (the short ID is the first 8 characters of the session UUID).
• Starting a session stops any active session first and requires a booted simulator.
• --interval-ms and --retention-seconds tune capture frequency and retention (default 60s). Retention always keeps at least one capture.
• Sessions capture screenshots on each interval; entries are written only when the tree or screenshot changes and screenshots are stored only when pixel content changes (JPEG at 50% quality).
• latest.json points to the most recent capture, latest.jpg symlinks to the latest screenshot, and latest-tree.json symlinks to the latest accessibility tree (no directory listing required).
• JSON output from session start includes full paths for the session directory, screens, trees, latest pointers, the UDID, and the current screen size in points (screen).
• session stop ends the active session and cleans up captured data. When -S is provided, it resolves the name to a UDID and verifies the active session matches before stopping (returns an error on mismatch). JSON output includes the stopped session’s UDID.
• Sessions also end automatically if the simulator disappears.

​

Find, Tap, and Double-Tap

• --by-id matches accessibility identifiers instead of labels.
• --by-role matches element roles (e.g. button, textField).
• --contains does a substring match against labels.
• --point expects x,y coordinates in points (matches normalized screenshots and tree output).
• Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
• Coordinate taps use the provided point exactly; use label/ID taps to target element centers.
• --geometry accepts points only.
• Session screenshots are normalized to point size so image coordinates map 1:1 to points.
• --duration on tap is a long-press hold time in seconds.

​

Swipe and Scroll

• swipe supports directions up, down, left, right or explicit --from x,y + --to x,y.
• --duration for swipe is in seconds (default 0.3).
• --distance sets swipe/scroll distance as a fraction of the screen (0.05–0.95), not pixels or points. Defaults: swipe 0.4, scroll 0.2.
  • Example: --distance 0.25 scrolls 25% of the screen height.
  • Invalid: --distance 300 (FlowDeck will reject values outside 0.05–0.95).
• --direction for scroll uses UP, DOWN, LEFT, or RIGHT (default DOWN).
• --speed is a 0-100 value (higher is faster).
• --until scrolls until an element is visible. Use id:myElement to match by accessibility identifier.
• --timeout for scroll --until is in milliseconds.

​

Pinch and Wait

• pinch uses a direction argument: in (zoom out) or out (zoom in).
• --scale overrides the zoom factor (defaults to 2.0 for out, 0.5 for in).
• --point expects x,y coordinates in points for the pinch center.
• --geometry accepts points only.
• --duration for pinch is in seconds.
• --timeout for wait is in seconds; --poll is in milliseconds.
• --gone, --enabled, and --stable change the wait condition (default is “exists”).

​

Text, Key, and App Control

• type --clear clears the focused field before typing.
• type --mask hides typed text in terminal output and JSON.
• erase --characters 5 deletes a specific number of characters; omit to clear all.
• key 40 sends a single HID keycode (for example, 40 = Enter).
• key --sequence 11,8,15 sends comma-separated HID keycodes.
• key --hold 1.0 holds a single key for the given seconds.
• open-url accepts https://... or custom schemes like myapp://path.
• clear-state requires a bundle identifier (for example com.example.app) and resets the simulator container for that app.
• rotate performs a two-finger rotation gesture (degrees, optional center/radius).

​

Set Appearance

• set-appearance accepts light or dark to switch the simulator’s appearance mode.
• If -S is omitted, the booted simulator is used.
• JSON output includes the applied appearance and target UDID.

​

Buttons and Touch

• button --hold 1.5 holds a hardware button for the given seconds.
• touch down and touch up expect x,y coordinates in points.
• Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
• --geometry accepts points only.

​

Performance and Reliability Tips

• Start a session before any UI work. Run flowdeck ui simulator session start -S <name-or-udid> --json, parse the JSON output to get latest_screenshot and latest_tree file paths, then use Read tool on those paths to see the screen and inspect elements.
• Verify after every action. After each tap/type/swipe, wait ~1 second, then re-read latest_screenshot to confirm the UI changed. Never chain actions without checking.
• Use accessibility identifiers and --by-id whenever possible; label matching is slower and more ambiguous.
• Before tapping an element, read latest_tree to confirm it exists and is visible.
• For automation loops, re-read latest-tree.json/latest.jpg from disk instead of issuing screen for every step. The session updates these files automatically.
• For one-off tree-only checks, use ui simulator screen --tree --json -S <name-or-udid> (no screenshot). Use --optimize when you do need a one-off screenshot.
• Use scroll --until "id:yourElement" to bring off-screen targets into view before tapping.
• Increase --poll for slow UIs to reduce load; decrease when you need faster detection.
• Tune input timing with environment variables:
  • FLOWDECK_HID_STABILIZATION_MS (default 25) for tap/gesture stability
  • FLOWDECK_TYPE_DELAY_MS (default 20) for typing speed

For a full help listing, run flowdeck ui simulator --help.