> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowdeck.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# macOS UI Automation

> Automate macOS app interactions with FlowDeck's UI automation commands

FlowDeck macOS UI automation drives native macOS apps using the Accessibility framework and CGEvent-based input. Commands live under `flowdeck ui mac`.

## Common Flags

Most macOS UI automation commands support:

* `--app` to target a specific app by **name, bundle ID, or PID**. Required on most commands. Examples: `--app "Safari"`, `--app "com.apple.Safari"`, `--app "12345"`.
* `--json` for machine-readable output.
* `--examples` / `-e` to print usage examples for the command.
* `-h` / `--help` for the help screen.

Leaf action commands (`click`, `double-click`, `right-click`, `type`, `erase`, `key`, `hotkey`, `scroll`, `move`, `drag`, `swipe`, `find`, `list`, `wait`, `assert`, `launch`, `activate`, `quit`, `screen`, window/menu subcommands, permission commands) accept only `--json`, `--examples`/`-e`, and `-h` beyond their command-specific flags. They do **not** accept `-v`/`--verbose`.

Every command that can reach the cursor also takes `--foreground`. Actions run in the background by default — no cursor movement, no focus change — and `--foreground` opts out. `drag`, `swipe`, `move`, `activate`, and `window focus` have no background path and require it. See [macOS Automation](/cli/macos-automation) for the full rationale.

Session subcommands (`session start`, `session stop`) additionally accept `-v` / `--verbose`.

App resolution order:

1. Numeric string → PID
2. Contains a dot → bundle ID
3. Otherwise → fuzzy match against running app names

## Permissions

| Command                      | Purpose                                              | Key Options |
| ---------------------------- | ---------------------------------------------------- | ----------- |
| `ui mac check-permissions`   | Check Accessibility and Screen Recording permissions | `--json`    |
| `ui mac request-permissions` | Trigger system permission dialogs                    | `--json`    |

## Capture

| Command                | Purpose                                         | Key Options                                               |
| ---------------------- | ----------------------------------------------- | --------------------------------------------------------- |
| `ui mac screen`        | Screenshot + accessibility tree from app window | `--app`, `--tree`, `--mode`, `--output`, `--json`         |
| `ui mac session start` | Start background tree + screenshot capture      | `--app`, `--interval-ms`, `--retention-seconds`, `--json` |
| `ui mac session stop`  | Stop the active capture session                 | `--json`                                                  |

## Query

| Command                   | Purpose                          | Key Options                                   |
| ------------------------- | -------------------------------- | --------------------------------------------- |
| `ui mac find`             | Locate elements by label/ID/role | `--app`, `--by-id`, `--by-role`, `--contains` |
| `ui mac list apps`        | List running GUI apps            | `--include-agents`, `--include-system`        |
| `ui mac list windows`     | List app windows                 | `--app`                                       |
| `ui mac list screens`     | List connected displays          | `--json`                                      |
| `ui mac list permissions` | Check permission status          | `--json`                                      |

## Interaction

| Command               | Purpose                                              | Key Options                                                            |
| --------------------- | ---------------------------------------------------- | ---------------------------------------------------------------------- |
| `ui mac click`        | Click element or coordinates                         | `--app`, `--point`, `--by-id`, `--duration`                            |
| `ui mac double-click` | Double-click element or coordinates                  | `--app`, `--point`, `--by-id`                                          |
| `ui mac right-click`  | Right-click element or coordinates                   | `--app`, `--point`, `--by-id`                                          |
| `ui mac type`         | Type text into focused element                       | `--app`, `--clear`, `--delay-ms`, `--mask`                             |
| `ui mac erase`        | Erase text from focused field                        | `--app`, `--characters`, `--foreground`                                |
| `ui mac scroll`       | Scroll in a direction                                | `--app`, `--direction`, `--amount`, `--smooth`, `--until`, `--timeout` |
| `ui mac move`         | Move cursor to a point                               | `--point`, `--app`                                                     |
| `ui mac drag`         | Drag from one point to another (**foreground-only**) | `--app`, `--from`, `--to`, `--duration`, `--foreground`                |
| `ui mac swipe`        | Swipe in a direction (**foreground-only**)           | `--app`, `--direction`, `--distance`, `--duration`, `--foreground`     |

## Keyboard

| Command         | Purpose                        | Key Options                             |
| --------------- | ------------------------------ | --------------------------------------- |
| `ui mac key`    | Press a key by name or keycode | `--app`, `--name`, `--keycode`          |
| `ui mac hotkey` | Press a keyboard shortcut      | `--app`, combo argument (e.g., `cmd+s`) |

## Wait and Assert

| Command                  | Purpose                     | Key Options                                    |
| ------------------------ | --------------------------- | ---------------------------------------------- |
| `ui mac wait`            | Wait for element condition  | `--app`, `--condition`, `--timeout`, `--by-id` |
| `ui mac assert visible`  | Assert element is visible   | `--app`, `--by-id`                             |
| `ui mac assert hidden`   | Assert element is hidden    | `--app`, `--by-id`                             |
| `ui mac assert enabled`  | Assert element is enabled   | `--app`, `--by-id`                             |
| `ui mac assert disabled` | Assert element is disabled  | `--app`, `--by-id`                             |
| `ui mac assert text`     | Assert element text matches | `--app`, `--expected`, `--contains`, `--by-id` |

## App Lifecycle

| Command           | Purpose                                     | Key Options             |
| ----------------- | ------------------------------------------- | ----------------------- |
| `ui mac launch`   | Launch an app by bundle ID                  | `--bundle-id`           |
| `ui mac activate` | Bring an app to front (**foreground-only**) | `--app`, `--foreground` |
| `ui mac quit`     | Quit an app                                 | `--app`, `--force`      |

## Window Management

| Command                | Purpose                              | Key Options                        |
| ---------------------- | ------------------------------------ | ---------------------------------- |
| `ui mac window list`   | List app windows                     | `--app`                            |
| `ui mac window move`   | Move a window                        | `--app`, `--to`, `--index`         |
| `ui mac window resize` | Resize a window                      | `--app`, `--size`, `--index`       |
| `ui mac window focus`  | Focus a window (**foreground-only**) | `--app`, `--index`, `--foreground` |

## Menu Interaction

| Command             | Purpose                   | Key Options            |
| ------------------- | ------------------------- | ---------------------- |
| `ui mac menu list`  | List app menu bar items   | `--app`                |
| `ui mac menu click` | Click a menu item by path | `--app`, path argument |

***

## Key Options Explained

### Screen and Sessions

* `--output` sets the screenshot file path (PNG). If omitted, FlowDeck writes to a temp file and prints the path.
* `--tree` returns only the accessibility tree (no screenshot).
* `--mode` (boolean; no value) captures the full screen instead of the app window. Use when the app window is partially offscreen or you need surrounding context (Dock, menu bar, other windows).
* Screenshots require macOS 14+ and Screen Recording permission.
* Screenshot dimensions are in pixels.
* `session start` captures tree + screenshots every 500ms by default and writes to `~/.flowdeck/automation/mac-sessions/<session-short-id>/` (the short ID is the first 8 characters of the session UUID).
* `session stop` takes **no** `--app`. At most one macOS UI session is active at a time, so `stop` is unambiguous.
* Starting a session stops any active macOS session first.
* `--interval-ms` and `--retention-seconds` tune capture frequency and retention (default 60s). Retention always keeps at least one capture.
* Sessions write `latest.json`, `latest.jpg`, and `latest-tree.json` so automation can re-read the newest capture without listing the directory.
* JSON output from `session start` includes the session directory, app name, PID, screens directory, trees directory, and the `latest_screenshot` / `latest_tree` paths.

```bash theme={null}
flowdeck ui mac screen --app "Safari" --json
flowdeck ui mac screen --app "Safari" --output ./screen.png
flowdeck ui mac screen --app "Safari" --tree --json
flowdeck ui mac session start --app "Safari" --json
flowdeck ui mac session stop --json
```

### Find

* `--by-id` matches accessibility identifiers instead of labels.
* `--by-role` matches element roles (e.g., `button`, `textField`).
* `--contains` does a substring match against labels.
* Returns element role, center coordinates, enabled state, and text value.
* Suggestions are provided when no exact match is found.

```bash theme={null}
flowdeck ui mac find "Settings" --app "MyApp"
flowdeck ui mac find "settings_button" --app "MyApp" --by-id
flowdeck ui mac find "button" --app "MyApp" --by-role
flowdeck ui mac find "Log" --app "MyApp" --contains
```

### Click, Double-Click, and Right-Click

* `--by-id` treats the target as an accessibility identifier.
* `--point` expects `x,y` coordinates (screen-absolute, matching `find` output).
* Pass `<target>` **or** `--point`, not both. The command errors if you supply neither.
* `--duration` on `click` is a long-press hold time in seconds.
* `tap` and `double-tap` are hidden aliases for `click` and `double-click`.

```bash theme={null}
flowdeck ui mac click "Log In" --app "MyApp"
flowdeck ui mac click "login_button" --app "MyApp" --by-id
flowdeck ui mac click --point 120,340 --app "MyApp"
flowdeck ui mac right-click "item" --app "MyApp"
```

### Type and Erase

* `--clear` clears the focused field before typing (Select All + Delete).
* `--delay-ms` overrides the per-character typing delay.
* `--mask` hides the typed text in terminal output and JSON (useful for passwords).
* `erase --characters 5` deletes a specific number of characters; omit to clear all.

```bash theme={null}
flowdeck ui mac type "hello@example.com" --app "MyApp"
flowdeck ui mac type "secret123" --app "MyApp" --mask
flowdeck ui mac type "New Value" --app "MyApp" --clear
flowdeck ui mac erase --app "MyApp"
flowdeck ui mac erase --characters 5 --app "MyApp"
```

### Key and Hotkey

* `key --name` accepts key names: `return`, `escape`, `tab`, `delete`, `space`, `f1`–`f12`, arrow keys, etc.
* `key --keycode` accepts raw virtual keycodes.
* Pass `--name` **or** `--keycode`, not both. The command errors if you supply neither.
* `hotkey` accepts modifier+key combos: `cmd+s`, `cmd+shift+z`, `ctrl+alt+delete`.
* **Command-key combos are unreliable against a backgrounded app.** Whether a `cmd+...` combo lands depends on how the target app installs the shortcut, and a combo that doesn't land is still reported as success. Use `ui mac menu click "File > Save"` when the result matters — it goes through accessibility and always works in the background — or pass `--foreground` after asking the user.
* Supported modifiers: `cmd`/`command`, `shift`, `ctrl`/`control`, `alt`/`option`.

```bash theme={null}
flowdeck ui mac key --name return --app "MyApp"
flowdeck ui mac key --keycode 36 --app "MyApp"
flowdeck ui mac hotkey "ctrl+a" --app "MyApp"
flowdeck ui mac menu click "File > Save" --app "TextEdit"   # not hotkey "cmd+s"
```

### Scroll

* `--direction` is required: `up`, `down`, `left`, or `right`.
* `--amount` sets the scroll magnitude in discrete ticks (default: `3`).
* `--smooth` enables many small ticks with delays for smooth scrolling.
* `--until` scrolls repeatedly until the target element appears. Use `id:myElement` for accessibility ID matching.
* `--timeout` sets the deadline for `--until` in **seconds** (default: `30`).
* Scroll is performed at the center of the app's focused window.

<Note>
  iOS `ui simulator scroll --timeout` is in **milliseconds**; macOS `ui mac scroll --timeout` is in **seconds**. Same flag, different unit — never copy values across platforms.
</Note>

```bash theme={null}
flowdeck ui mac scroll --direction down --app "Safari"
flowdeck ui mac scroll --direction up --amount 10 --app "MyApp"
flowdeck ui mac scroll --direction down --amount 5 --smooth --app "MyApp"
flowdeck ui mac scroll --direction down --until "id:bottomButton" --app "MyApp"
flowdeck ui mac scroll --direction down --until "Save" --timeout 15 --app "MyApp"
```

### Move, Drag, and Swipe

* `move --point` expects `x,y` screen-absolute coordinates.
* `move --app` is **optional**. The cursor moves to the absolute screen point regardless of which app is focused; the flag only validates that the named app exists.
* `move`, `drag`, and `swipe` are **foreground-only**: they take the real cursor, so they require `--foreground` and exit non-zero without it.
* `drag --from` and `--to` expect `x,y` coordinates. `--duration` is in seconds (default: `0.5`).
* `swipe --direction` accepts `up`, `down`, `left`, `right`. `--distance` is in points (default: `200`).

```bash theme={null}
flowdeck ui mac move --point 500,300 --app "MyApp"
flowdeck ui mac drag --from 100,200 --to 400,500 --app "MyApp" --foreground
flowdeck ui mac drag --from 100,200 --to 400,500 --duration 1.0 --app "MyApp" --foreground
flowdeck ui mac swipe --direction up --app "MyApp" --foreground
flowdeck ui mac swipe --direction left --distance 400 --app "MyApp" --foreground
```

### Wait

* `--condition` accepts one of: `exists` (default), `gone`, `enabled`, `stable`. Invalid values are rejected with an error.
  * `exists`  — element is present in the accessibility tree
  * `gone`    — element is not present
  * `enabled` — element is present and `enabled = true`
  * `stable`  — element's frame stops moving between polls (use to wait for an animated transition to settle)
* `--timeout` is in **seconds** (default: `30`).
* `--by-id` treats the target as an accessibility identifier.

```bash theme={null}
flowdeck ui mac wait "Loading..." --app "MyApp"
flowdeck ui mac wait "Submit" --app "MyApp" --condition enabled --timeout 15
flowdeck ui mac wait "Toast" --app "MyApp" --condition gone
flowdeck ui mac wait "save_button" --app "MyApp" --by-id --condition enabled
```

### Assert

Assert element conditions with immediate pass/fail (no polling). Exits with failure if the assertion does not hold.

**Subcommands:**

| Subcommand          | Description                     |
| ------------------- | ------------------------------- |
| `visible <target>`  | Assert the element is visible   |
| `hidden <target>`   | Assert the element is hidden    |
| `enabled <target>`  | Assert the element is enabled   |
| `disabled <target>` | Assert the element is disabled  |
| `text <target>`     | Assert the element text matches |

**Common Options:**

| Option    | Description                              |
| --------- | ---------------------------------------- |
| `--by-id` | Treat target as accessibility identifier |
| `--app`   | Target app                               |

**Text Options:**

| Option              | Description                                        |
| ------------------- | -------------------------------------------------- |
| `--expected <text>` | Expected text value (required)                     |
| `--contains`        | Check whether the text contains the expected value |

```bash theme={null}
flowdeck ui mac assert visible "Profile" --app "MyApp"
flowdeck ui mac assert hidden "Spinner" --app "MyApp"
flowdeck ui mac assert enabled "Submit" --app "MyApp"
flowdeck ui mac assert disabled "Continue" --app "MyApp"
flowdeck ui mac assert text "Welcome" --app "MyApp" --expected "Hello"
flowdeck ui mac assert text "title_label" --app "MyApp" --by-id --expected "Dashboard" --contains
```

### App Lifecycle

* `launch` uses `--bundle-id`, **not** `--app`. It targets the installed bundle directly, not a running process. Discover bundle IDs with `ui mac list apps --json`.
* `activate --app` brings a running app to the foreground. It requires `--foreground`, because raising an app takes the user's screen. You rarely need it: every action command already reaches a background app.
* `quit --app` sends a graceful terminate. `--force` force-quits.

```bash theme={null}
flowdeck ui mac launch --bundle-id com.apple.Safari
flowdeck ui mac activate --app "Safari" --foreground
flowdeck ui mac quit --app "TextEdit"
flowdeck ui mac quit --app "MyApp" --force
```

### Window Management

* `--index` selects which window (default: `0`, the first/main window).
* `window move --to` expects `x,y` for the new origin.
* `window resize --size` expects `w,h` for the new dimensions.

```bash theme={null}
flowdeck ui mac window list --app "Safari" --json
flowdeck ui mac window move --app "Safari" --to 100,100
flowdeck ui mac window resize --app "Safari" --size 1200,800
flowdeck ui mac window focus --app "Safari" --index 1 --foreground
```

### Menu Interaction

* `menu list` shows the app's menu bar hierarchy with titles, shortcuts, and enabled state.
* `menu click` navigates a menu path using `>` separators.

```bash theme={null}
flowdeck ui mac menu list --app "TextEdit" --json
flowdeck ui mac menu click "File > Export as PDF" --app "TextEdit"
flowdeck ui mac menu click "Edit > Find > Find..." --app "Safari"
```

### List

* `list apps` shows running GUI applications with PID, name, and bundle ID.
* `--include-agents` includes background agent apps.
* `--include-system` includes system processes.
* `list windows` requires `--app`.
* `list screens` shows connected displays with frame and scale info.
* `list permissions` shows Accessibility and Screen Recording status.

```bash theme={null}
flowdeck ui mac list apps --json
flowdeck ui mac list apps --include-agents --include-system
flowdeck ui mac list windows --app "Safari" --json
flowdeck ui mac list screens --json
flowdeck ui mac list permissions --json
```

## Coordinate System

* All coordinates are **screen-absolute points** (matching what `find` returns).
* Multi-display setups use the global coordinate space (origin at primary display top-left).
* Do not apply Retina scaling; use coordinates as reported by `find` and `list screens`.
* Use `flowdeck ui mac list screens` to see display layouts and pixel densities.

### Coordinate Format

All point and size flags on macOS use **comma-separated** `x,y` (or `w,h` for sizes). Space-separated values are rejected with a live error:

```
Invalid coordinates: 400 300 is not supported on macOS
```

Applies to `--point`, `--from`, `--to`, and `--size` across `click`, `double-click`, `right-click`, `move`, `drag`, `window move`, and `window resize`.

```bash theme={null}
# Correct
flowdeck ui mac click --point 400,300 --app "MyApp"
flowdeck ui mac drag --from 100,100 --to 500,400 --app "MyApp" --foreground
flowdeck ui mac window resize --app "Safari" --size 1200,800

# Wrong — rejected
flowdeck ui mac click --point "400 300" --app "MyApp"
```

## Timing Tuning

Set these environment variables to adjust input timing:

* `FLOWDECK_HID_STABILIZATION_MS` adds settle time between input events (default: `25`)
* `FLOWDECK_TYPE_DELAY_MS` adds per-character typing delay (default: `1`)

## Performance and Reliability Tips

* **Start a session before any macOS UI work.** Run `flowdeck ui mac session start --app "MyApp" --json`, parse the JSON output to get `latest_screenshot` and `latest_tree`, then read those files to inspect the current UI.
* macOS UI sessions remain single-active and stop automatically when the owner process that started them exits. Set `FLOWDECK_OWNER_PID` to bind a session to a specific owner process.
* **Verify after every action.** After each click/type/scroll, wait briefly, then re-read `latest_screenshot` and `latest_tree` to confirm the UI changed.
* Use accessibility identifiers and `--by-id` whenever possible; label matching is slower and more ambiguous.
* For automation loops, re-read `latest-tree.json`/`latest.jpg` from disk instead of issuing `screen` for every step. The session updates these files automatically.
* Use `ui mac screen --tree --json --app "MyApp"` for one-off tree-only checks when you do not need a running session.

For a full help listing, run `flowdeck ui mac --help`.