UI Automation

​

Common Flags

• -S, --simulator to target a specific simulator by name or UDID. Recommended when multiple simulators are booted. Examples: -S "iPhone 16" or -S "A1B2C3D4-...".
• For AI agent workflows, always pass -S explicitly on every flowdeck ui simulator ... command.
• If -S is omitted, FlowDeck falls back to session/default simulator selection. This is usually fine when only one simulator is booted (common in CI).
• --json, -j for machine-readable output.
• --verbose, -v and --examples, -e on commands that support extra output.

​

Capture

​

Query

​

Interaction

​

Wait and Assert

​

Input and Keyboard

​

App Control

​

Simulator Hardware Aliases

​

Advanced

​

Batch Flows

flowdeck ui simulator batch --steps '[
  {"action":"tap","target":"settings_button","by_id":true},
  {"action":"tap","target":"Reduce transparency"},
  {"action":"tap","target":"Done"},
  {"action":"tap","target":"Search"},
  {"action":"type","text":"London"},
  {"action":"tap","target":"London, England"},
  {"action":"assert","text":"11°"},
  {"action":"tap","target":"Precipitation"},
  {"action":"assert","text":"10.7 mm"}
]' -S "iPhone 16" --json

​

Batch Options

​

Step Actions

​

Key Options Explained

​

Screen and Record

• --output uses a file path. Screenshots default to a .png, recordings to a .mov. If omitted, FlowDeck writes to a temp file and prints the path.
• screen returns the accessibility tree by default and captures no image. The tree is the source of truth for what is on screen.
• --screenshot adds an image to that response. Use it when a value is drawn without accessibility data — custom graphics, charts — or when you need pixel-level layout.
• --interactive-elements returns only the actionable elements instead of the full tree, which is what you want when you are just looking for a tap target.
• --since-hash <hash> returns unchanged instead of the full tree when the screen still matches that hash, so a polling loop costs almost nothing while nothing moves.
• --optimize shrinks the screenshot output for AI workflows.
• --tree is deprecated and accepted only for compatibility. The tree already comes back without it.
• screen reports size in points; JSON includes point_width/point_height and pixel_width/pixel_height when available.
• --duration is in seconds (supports decimals like 2.5).
• --codec accepts h264 or hevc.
• --force overwrites an existing output file.

​

Sessions

• Always pass -S <name-or-udid> on session start and session stop to target a specific simulator. Accepts a simulator name (e.g., "iPhone 16") or raw UDID. This is required when multiple simulators are booted and recommended in all automation workflows.
• session start captures the accessibility tree and checks a persistent simulator framebuffer every 500ms by default. It writes to ~/.flowdeck/automation/sessions/<session-short-id>/ (the short ID is the first 8 characters of the session UUID).
• Starting a session stops and replaces only an existing session for the same simulator. Sessions for other simulators keep running concurrently.
• --interval-ms and --retention-seconds tune capture frequency and retention (default 60s). Retention always keeps at least one capture.
• The framebuffer backend skips pixel copying, JPEG encoding, and image writes while its version is unchanged. Entries are written only when the tree or screenshot body changes, and screenshots are stored at 50% JPEG quality. Screenshot change detection ignores the volatile top system/status strip, so Dynamic Island/status-bar churn does not keep rewriting idle frames; saved screenshots are not cropped or masked.
• FlowDeck automatically falls back to the legacy screenshot backend if the private framebuffer interface cannot attach or fails during the session. It prints one warning and keeps that backend for the rest of the session; no manual flag is required.
• latest.json points to the most recent capture, latest.jpg symlinks to the latest screenshot, and latest-tree.json symlinks to the latest accessibility tree (no directory listing required).
• JSON output from session start includes full paths for the session directory, screens, trees, latest pointers, the UDID, and the current screen size in points (screen). Key fields you can parse:
  • session_id — full UUID of the session
  • session_short_id — first 8 characters of the UUID (matches the directory name)
  • session_dir — absolute path to the session’s data directory
  • latest_screenshot — absolute path to the rolling latest.jpg file
  • latest_tree — absolute path to the rolling latest-tree.json file
  • capture_backend — framebuffer normally, or legacy_screenshot after automatic fallback
  • capture_warning — present only during fallback, with reason, message, and timestamp_ms
  • screens_dir, trees_dir, root, udid, interval_ms, retention_seconds, pid, owner_pid, screen
• session list --json reports every active session, including udid, recorder pid, owner_pid, age_seconds, and the live capture_backend/capture_warning state.
• Session commands skip malformed active pointer files and reconcile stale pointers for dead recorders, abandoned pending starts, and sessions from a previous system boot, so list, stop, and stop --all remain usable for recovery.
• session stop without -S stops the sole active session. If multiple sessions are active, pass -S <name-or-udid> to stop one or --all to stop every active session. JSON output includes the stopped session UDID(s).
• Sessions also end automatically if the simulator disappears or if the owner process that started the session exits. Set FLOWDECK_OWNER_PID to bind a session to a specific owner process.

​

Find, Tap, and Double-Tap

• Default (no flag): exact label match.
• --by-id: matches accessibility identifiers instead of labels.
• --by-role: matches element roles (e.g. button, textField).
• --contains: substring match against labels (only valid alone or with default label mode; do not combine with --by-id or --by-role).
• --point expects a single comma-separated value x,y in points (matches normalized screenshots and tree output).
  • Valid: --point 100,200 or --point "100,200".
  • Invalid: --point 100 200 (space-separated; two arguments will be rejected).
• Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
• Coordinate taps use the provided point exactly; use label/ID taps to target element centers.
• --geometry accepts points only.
• Session screenshots are normalized to point size so image coordinates map 1:1 to points.
• --duration on tap is a long-press hold time in seconds.

​

Swipe and Scroll

• swipe supports directions up, down, left, right or explicit --from x,y + --to x,y.
• --duration for swipe is in seconds (default 0.3).
• --distance sets swipe/scroll distance as a fraction of the screen (0.05–0.95), not pixels or points. Defaults: swipe 0.4, scroll 0.85 (≈ one viewport, so each scroll advances a full screen with a small overlap — no duplicated or skipped content).
  • Example: --distance 0.5 scrolls half the screen height.
  • Invalid: --distance 300 (FlowDeck will reject values outside 0.05–0.95).
• scroll returns moved and reached_end in its JSON. When reached_end is true the content didn’t move — you are at the top/bottom; stop scrolling.
• --direction for scroll uses UP, DOWN, LEFT, or RIGHT (default DOWN).
• --speed is a 0-100 value (higher is faster).
• --until scrolls until an element is visible, then returns the settled screen. Use id:myElement to match by accessibility identifier. It early-exits when the content stops moving, so it won’t scroll uselessly to the timeout at the end of a list.
• --timeout for scroll --until is in milliseconds (default 20000).
• tap, type, scroll, swipe, double-tap, and wait return the settled post-action screen (full tree + element refs + screen_hash) under a screen key by default — so you don’t need a separate snapshot after each action. Pass --no-screen for a terse result when you don’t need the tree.

​

Pinch and Wait

• pinch uses a direction argument: in (zoom out) or out (zoom in).
• --scale overrides the zoom factor (defaults to 2.0 for out, 0.5 for in).
• --point expects x,y coordinates in points for the pinch center.
• --geometry accepts points only.
• --duration for pinch is in seconds.
• --timeout for wait is in seconds; --poll is in milliseconds.
• --gone, --enabled, and --stable change the wait condition (default is “exists”).

​

Text, Key, and App Control

• type --clear clears the focused field before typing.
• type --mask hides typed text in terminal output and JSON.
• erase --characters 5 deletes a specific number of characters; omit to clear all.
• key 40 sends a single HID keycode (for example, 40 = Enter).
• key --sequence 11,8,15 sends comma-separated HID keycodes.
• key --hold 1.0 holds a single key for the given seconds.
• Pass either a positional keycode (e.g. key 40) or --sequence, not both.
• open-url accepts https://... or custom schemes like myapp://path.
• clear-state requires a bundle identifier (for example com.example.app) and resets the simulator container for that app.
• rotate <angle> performs a two-finger rotation gesture. <angle> is in degrees; positive values rotate clockwise, negative values rotate counterclockwise. Optional --point (center) and --radius.

​

Buttons, Appearance, and Touch

• Use flowdeck simulator button --hold 1.5 to hold a hardware button for the given seconds.
• Use flowdeck simulator appearance light or dark to switch appearance mode.
• Use flowdeck simulator orientation get or orientation set <value> for physical orientation.
• touch down and touch up expect x,y coordinates in points.
• Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
• --geometry accepts points only.

​

Performance and Reliability Tips

• Start a session before any UI work. Run flowdeck ui simulator session start -S <name-or-udid> --json, parse the JSON output to get latest_screenshot and latest_tree file paths, then use Read tool on those paths to see the screen and inspect elements.
• Verify after every action. After each tap/type/swipe, wait ~1 second, then re-read latest_screenshot to confirm the UI changed. Never chain actions without checking.
• Use accessibility identifiers and --by-id whenever possible; label matching is slower and more ambiguous.
• Before tapping an element, read latest_tree to confirm it exists and is visible.
• For automation loops, re-read latest-tree.json/latest.jpg from disk instead of issuing screen for every step. The session updates these files automatically.
• For one-off checks, use ui simulator screen --json -S <name-or-udid> — the tree comes back and no screenshot is taken. Add --screenshot (with --optimize) only when you actually need the image.
• Use scroll --until "id:yourElement" to bring off-screen targets into view before tapping.
• Increase --poll for slow UIs to reduce load; decrease when you need faster detection.
• Tune input timing with environment variables:
  • FLOWDECK_HID_STABILIZATION_MS (default 25) for tap/gesture stability
  • FLOWDECK_TYPE_DELAY_MS (default 20) for typing speed