> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowdeck.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# UI Automation

> Automate simulator interactions with FlowDeck's UI automation commands

FlowDeck UI automation runs on iOS simulators and exposes commands for capture, interaction, assertions, and app control. Commands live under `flowdeck ui simulator`.

## Common Flags

Most UI automation commands support:

* `-S, --simulator` to target a specific simulator by **name or UDID**. Recommended when multiple simulators are booted. Examples: `-S "iPhone 16"` or `-S "A1B2C3D4-..."`.
* For AI agent workflows, always pass `-S` explicitly on every `flowdeck ui simulator ...` command.
* If `-S` is omitted, FlowDeck falls back to session/default simulator selection. This is usually fine when only one simulator is booted (common in CI).
* `--json`, `-j` for machine-readable output.
* `--verbose`, `-v` and `--examples`, `-e` on commands that support extra output.

## Capture

| Command                      | Purpose                                    | Key Options                                                                        |
| ---------------------------- | ------------------------------------------ | ---------------------------------------------------------------------------------- |
| `ui simulator screen`        | Accessibility tree (screenshot on request) | `--screenshot`, `--interactive-elements`, `--since-hash`, `--optimize`, `--output` |
| `ui simulator session start` | Start background tree + screenshot capture | `-S`, `--interval-ms`, `--retention-seconds`                                       |
| `ui simulator session list`  | List active capture sessions               | `--json`                                                                           |
| `ui simulator session stop`  | Stop one or all active capture sessions    | `-S`, `--all`                                                                      |

## Query

| Command             | Purpose                          | Key Options                          |
| ------------------- | -------------------------------- | ------------------------------------ |
| `ui simulator find` | Locate elements by label/ID/role | `--by-id`, `--by-role`, `--contains` |

## Interaction

| Command                   | Purpose                               | Key Options                                                                   |
| ------------------------- | ------------------------------------- | ----------------------------------------------------------------------------- |
| `ui simulator tap`        | Tap element or coordinates            | `--point`, `--duration`, `--by-id`, `--no-screen`                             |
| `ui simulator double-tap` | Double tap element or coordinates     | `--point`, `--by-id`, `--no-screen`                                           |
| `ui simulator type`       | Type into focused field               | `--clear`, `--mask`, `--no-screen`                                            |
| `ui simulator swipe`      | Swipe by direction or coordinates     | `--from`, `--to`, `--duration`, `--distance`, `--no-screen`                   |
| `ui simulator scroll`     | Scroll content (gentler than swipe)   | `--direction`, `--speed`, `--distance`, `--until`, `--timeout`, `--no-screen` |
| `ui simulator pinch`      | Pinch to zoom in/out                  | `in`/`out`, `--scale`, `--point`, `--duration`                                |
| `ui simulator rotate`     | Rotate with a two-finger gesture      | `<angle>`, `--point`, `--radius`, `--duration`                                |
| `ui simulator back`       | Navigate back (edge swipe)            |                                                                               |
| `ui simulator batch`      | Run a whole UI flow in one invocation | `--steps`, `--file`, `--interactive-elements`                                 |

## Wait and Assert

| Command                        | Purpose                    | Key Options                                              |
| ------------------------------ | -------------------------- | -------------------------------------------------------- |
| `ui simulator wait`            | Wait for element state     | `--timeout`, `--poll`, `--gone`, `--enabled`, `--stable` |
| `ui simulator assert visible`  | Assert element is visible  | `--by-id`                                                |
| `ui simulator assert hidden`   | Assert element is hidden   | `--by-id`                                                |
| `ui simulator assert enabled`  | Assert element is enabled  | `--by-id`                                                |
| `ui simulator assert disabled` | Assert element is disabled | `--by-id`                                                |
| `ui simulator assert text`     | Assert element text        | `--expected`, `--contains`, `--by-id`                    |

## Input and Keyboard

| Command                      | Purpose                       | Key Options            |
| ---------------------------- | ----------------------------- | ---------------------- |
| `ui simulator erase`         | Erase text from focused field | `--characters`         |
| `ui simulator hide-keyboard` | Hide the on-screen keyboard   |                        |
| `ui simulator key`           | Send keyboard keycodes        | `--sequence`, `--hold` |

## App Control

| Command                    | Purpose                 | Key Options |
| -------------------------- | ----------------------- | ----------- |
| `ui simulator open-url`    | Open a URL or deep link |             |
| `ui simulator clear-state` | Clear app data/state    |             |

## Simulator Hardware Aliases

Simulator hardware and device-state controls are canonical under `flowdeck simulator`.

| Canonical Command       | Compatibility Alias           | Purpose                |
| ----------------------- | ----------------------------- | ---------------------- |
| `simulator button`      | `ui simulator button`         | Press hardware buttons |
| `simulator appearance`  | `ui simulator set-appearance` | Set light or dark mode |
| `simulator orientation` | `ui simulator orientation`    | Get or set orientation |

## Advanced

| Command                   | Purpose                   | Key Options |
| ------------------------- | ------------------------- | ----------- |
| `ui simulator touch down` | Touch down at coordinates | `x,y`, `-S` |
| `ui simulator touch up`   | Touch up at coordinates   | `x,y`, `-S` |

## Batch Flows

`ui simulator batch` runs a whole UI flow in a single invocation. It spans screens (label/id targets are re-found on the current screen), auto-scrolls to off-screen targets, and verifies values inline with `assert` — so navigation, input, and verification become one call with no per-action round-trips and no intermediate screenshots. It returns the final accessibility tree (or only the interactive elements with `--interactive-elements`).

```bash theme={null}
flowdeck ui simulator batch --steps '[
  {"action":"tap","target":"settings_button","by_id":true},
  {"action":"tap","target":"Reduce transparency"},
  {"action":"tap","target":"Done"},
  {"action":"tap","target":"Search"},
  {"action":"type","text":"London"},
  {"action":"tap","target":"London, England"},
  {"action":"assert","text":"11°"},
  {"action":"tap","target":"Precipitation"},
  {"action":"assert","text":"10.7 mm"}
]' -S "iPhone 16" --json
```

Steps can also be loaded from a file with `--file path/to/steps.json`.

### Batch Options

| Option                       | Short | Description                                                                     |
| ---------------------------- | ----- | ------------------------------------------------------------------------------- |
| `--steps <json>`             |       | JSON array of action steps                                                      |
| `--file <path>`              | `-f`  | Path to a JSON file of action steps                                             |
| `--simulator <name-or-udid>` | `-S`  | Target simulator name or UDID                                                   |
| `--interactive-elements`     |       | Return only interactive elements in the final snapshot instead of the full tree |
| `--json`                     | `-j`  | Output as JSON                                                                  |
| `--examples`                 | `-e`  | Show usage examples                                                             |

### Step Actions

| Action               | Fields                                          | Notes                                                                                              |
| -------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `tap` / `double_tap` | `target` (+ `by_id`), or `point` (`"x,y"`)      | Off-screen targets are auto-scrolled into view                                                     |
| `type`               | `text` (+ `clear`)                              | Types into the focused field                                                                       |
| `swipe` / `scroll`   | `direction` (+ `distance`)                      | `scroll` defaults to roughly one viewport                                                          |
| `wait`               | `target` (+ `condition`, `timeout_ms`, `by_id`) | Waits for an element state                                                                         |
| `key`                | `keycode`                                       | Sends a HID keycode                                                                                |
| `assert`             | `text` (+ `condition`)                          | Verifies `text` is visible; `condition: "absent"` asserts it is NOT present (auto-scrolls to find) |

The batch stops at the first failing step. The JSON response includes a `steps[]` array with per-step `success`/`error` and a `final` snapshot of the screen after the last step.

<Note>
  `batch` is the most efficient way to drive a multi-step flow from an agent: one call replaces many `tap`/`type`/`screen` round-trips, and inline `assert` steps verify state without screenshots.
</Note>

## Key Options Explained

### Screen and Record

* `--output` uses a file path. Screenshots default to a `.png`, recordings to a `.mov`. If omitted, FlowDeck writes to a temp file and prints the path.
* `screen` returns the accessibility tree by default and captures no image. The tree is the source of truth for what is on screen.
* `--screenshot` adds an image to that response. Use it when a value is drawn without accessibility data — custom graphics, charts — or when you need pixel-level layout.
* `--interactive-elements` returns only the actionable elements instead of the full tree, which is what you want when you are just looking for a tap target.
* `--since-hash <hash>` returns `unchanged` instead of the full tree when the screen still matches that hash, so a polling loop costs almost nothing while nothing moves.
* `--optimize` shrinks the screenshot output for AI workflows.
* `--tree` is deprecated and accepted only for compatibility. The tree already comes back without it.
* `screen` reports size in points; JSON includes `point_width`/`point_height` and `pixel_width`/`pixel_height` when available.
* `--duration` is in seconds (supports decimals like `2.5`).
* `--codec` accepts `h264` or `hevc`.
* `--force` overwrites an existing output file.

### Sessions

* **Always pass `-S <name-or-udid>`** on `session start` and `session stop` to target a specific simulator. Accepts a simulator name (e.g., `"iPhone 16"`) or raw UDID. This is required when multiple simulators are booted and recommended in all automation workflows.
* `session start` captures the accessibility tree and checks a persistent simulator framebuffer every 500ms by default. It writes to `~/.flowdeck/automation/sessions/<session-short-id>/` (the short ID is the first 8 characters of the session UUID).
* Starting a session stops and replaces only an existing session for the same simulator. Sessions for other simulators keep running concurrently.
* `--interval-ms` and `--retention-seconds` tune capture frequency and retention (default 60s). Retention always keeps at least one capture.
* The framebuffer backend skips pixel copying, JPEG encoding, and image writes while its version is unchanged. Entries are written only when the tree or screenshot body changes, and screenshots are stored at 50% JPEG quality. Screenshot change detection ignores the volatile top system/status strip, so Dynamic Island/status-bar churn does not keep rewriting idle frames; saved screenshots are not cropped or masked.
* FlowDeck automatically falls back to the legacy screenshot backend if the private framebuffer interface cannot attach or fails during the session. It prints one warning and keeps that backend for the rest of the session; no manual flag is required.
* `latest.json` points to the most recent capture, `latest.jpg` symlinks to the latest screenshot, and `latest-tree.json` symlinks to the latest accessibility tree (no directory listing required).
* JSON output from `session start` includes full paths for the session directory, screens, trees, latest pointers, the UDID, and the current screen size in points (`screen`). Key fields you can parse:
  * `session_id` — full UUID of the session
  * `session_short_id` — first 8 characters of the UUID (matches the directory name)
  * `session_dir` — absolute path to the session's data directory
  * `latest_screenshot` — absolute path to the rolling `latest.jpg` file
  * `latest_tree` — absolute path to the rolling `latest-tree.json` file
  * `capture_backend` — `framebuffer` normally, or `legacy_screenshot` after automatic fallback
  * `capture_warning` — present only during fallback, with `reason`, `message`, and `timestamp_ms`
  * `screens_dir`, `trees_dir`, `root`, `udid`, `interval_ms`, `retention_seconds`, `pid`, `owner_pid`, `screen`
* `session list --json` reports every active session, including `udid`, recorder `pid`, `owner_pid`, `age_seconds`, and the live `capture_backend`/`capture_warning` state.
* Session commands skip malformed active pointer files and reconcile stale pointers for dead recorders, abandoned pending starts, and sessions from a previous system boot, so `list`, `stop`, and `stop --all` remain usable for recovery.
* `session stop` without `-S` stops the sole active session. If multiple sessions are active, pass `-S <name-or-udid>` to stop one or `--all` to stop every active session. JSON output includes the stopped session UDID(s).
* Sessions also end automatically if the simulator disappears or if the owner process that started the session exits. Set `FLOWDECK_OWNER_PID` to bind a session to a specific owner process.

### Find, Tap, and Double-Tap

<Warning>
  `--by-id`, `--by-role`, and `--contains` are **mutually exclusive lookup modes**. Pass at most one. The default (none) is an **exact label match**. Only `--contains` can be combined with the default to do substring matching against labels.
</Warning>

* Default (no flag): exact label match.
* `--by-id`: matches accessibility identifiers instead of labels.
* `--by-role`: matches element roles (e.g. `button`, `textField`).
* `--contains`: substring match against labels (only valid alone or with default label mode; do not combine with `--by-id` or `--by-role`).
* `--point` expects a **single comma-separated value** `x,y` in points (matches normalized screenshots and tree output).
  * Valid: `--point 100,200` or `--point "100,200"`.
  * Invalid: `--point 100 200` (space-separated; two arguments will be rejected).
* Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
* Coordinate taps use the provided point exactly; use label/ID taps to target element centers.
* `--geometry` accepts `points` only.
* Session screenshots are normalized to point size so image coordinates map 1:1 to points.
* `--duration` on `tap` is a long-press hold time in seconds.

### Swipe and Scroll

* `swipe` supports directions `up`, `down`, `left`, `right` or explicit `--from x,y` + `--to x,y`.
* `--duration` for `swipe` is in seconds (default `0.3`).
* `--distance` sets swipe/scroll distance as a **fraction of the screen** (0.05–0.95), not pixels or points. Defaults: swipe `0.4`, scroll `0.85` (≈ one viewport, so each scroll advances a full screen with a small overlap — no duplicated or skipped content).
  * Example: `--distance 0.5` scrolls half the screen height.
  * Invalid: `--distance 300` (FlowDeck will reject values outside 0.05–0.95).
* `scroll` returns `moved` and `reached_end` in its JSON. When `reached_end` is `true` the content didn't move — you are at the top/bottom; stop scrolling.
* `--direction` for `scroll` uses `UP`, `DOWN`, `LEFT`, or `RIGHT` (default `DOWN`).
* `--speed` is a 0-100 value (higher is faster).
* `--until` scrolls until an element is visible, then returns the settled screen. Use `id:myElement` to match by accessibility identifier. It early-exits when the content stops moving, so it won't scroll uselessly to the timeout at the end of a list.
* `--timeout` for `scroll --until` is in **milliseconds** (default `20000`).
* `tap`, `type`, `scroll`, `swipe`, `double-tap`, and `wait` return the settled post-action screen (full tree + element refs + `screen_hash`) under a `screen` key by default — so you don't need a separate snapshot after each action. Pass `--no-screen` for a terse result when you don't need the tree.

<Warning>
  The iOS `scroll --timeout` is measured in **milliseconds**. The macOS counterpart (`flowdeck ui mac scroll --timeout`) is measured in **seconds**. Same flag name, different unit — never copy a value across platforms without converting.
</Warning>

### Pinch and Wait

* `pinch` uses a direction argument: `in` (zoom out) or `out` (zoom in).
* `--scale` overrides the zoom factor (defaults to `2.0` for `out`, `0.5` for `in`).
* `--point` expects `x,y` coordinates in points for the pinch center.
* `--geometry` accepts `points` only.
* `--duration` for `pinch` is in seconds.
* `--timeout` for `wait` is in seconds; `--poll` is in milliseconds.
* `--gone`, `--enabled`, and `--stable` change the wait condition (default is “exists”).

### Text, Key, and App Control

* `type --clear` clears the focused field before typing.
* `type --mask` hides typed text in terminal output and JSON.
* `erase --characters 5` deletes a specific number of characters; omit to clear all.
* `key 40` sends a single HID keycode (for example, `40` = Enter).
* `key --sequence 11,8,15` sends comma-separated HID keycodes.
* `key --hold 1.0` holds a single key for the given seconds.
* Pass either a positional keycode (e.g. `key 40`) **or** `--sequence`, not both.
* `open-url` accepts `https://...` or custom schemes like `myapp://path`.
* `clear-state` requires a bundle identifier (for example `com.example.app`) and resets the simulator container for that app.
* `rotate <angle>` performs a two-finger rotation gesture. `<angle>` is in degrees; positive values rotate **clockwise**, negative values rotate **counterclockwise**. Optional `--point` (center) and `--radius`.

### Buttons, Appearance, and Touch

* Use `flowdeck simulator button --hold 1.5` to hold a hardware button for the given seconds.
* Use `flowdeck simulator appearance light` or `dark` to switch appearance mode.
* Use `flowdeck simulator orientation get` or `orientation set <value>` for physical orientation.
* `touch down` and `touch up` expect `x,y` coordinates in points.
* Do not scale by @2x/@3x or device resolution; use the image coordinates directly.
* `--geometry` accepts `points` only.

## Performance and Reliability Tips

* **Start a session before any UI work.** Run `flowdeck ui simulator session start -S <name-or-udid> --json`, parse the JSON output to get `latest_screenshot` and `latest_tree` file paths, then use Read tool on those paths to see the screen and inspect elements.
* **Verify after every action.** After each tap/type/swipe, wait \~1 second, then re-read `latest_screenshot` to confirm the UI changed. Never chain actions without checking.
* Use accessibility identifiers and `--by-id` whenever possible; label matching is slower and more ambiguous.
* Before tapping an element, read `latest_tree` to confirm it exists and is visible.
* For automation loops, re-read `latest-tree.json`/`latest.jpg` from disk instead of issuing `screen` for every step. The session updates these files automatically.
* For one-off checks, use `ui simulator screen --json -S <name-or-udid>` — the tree comes back and no screenshot is taken. Add `--screenshot` (with `--optimize`) only when you actually need the image.
* Use `scroll --until "id:yourElement"` to bring off-screen targets into view before tapping.
* Increase `--poll` for slow UIs to reduce load; decrease when you need faster detection.
* Tune input timing with environment variables:
  * `FLOWDECK_HID_STABILIZATION_MS` (default 25) for tap/gesture stability
  * `FLOWDECK_TYPE_DELAY_MS` (default 20) for typing speed

For a full help listing, run `flowdeck ui simulator --help`.