> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowdeck.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# macOS Automation

> Overview of FlowDeck's macOS app UI automation

FlowDeck ships built-in UI automation for macOS apps so you can drive native interfaces, validate UI state, and capture accessibility trees from the CLI. Use `flowdeck ui mac` to run automation commands.

## What It Does

Use macOS automation to:

* Capture screenshots and accessibility trees from any running macOS app.
* Click elements, type text, scroll, drag, and navigate menus.
* Wait for UI state changes and assert UI conditions.
* Manage windows, launch/quit apps, and send keyboard shortcuts.

## When to Use It

* Smoke tests for macOS app flows.
* Scripted demos or QA checks on native Mac apps.
* AI-driven interaction loops that need consistent UI state.
* Automating menu-driven workflows or window management.

## Prerequisites

macOS automation requires two system permissions:

1. **Accessibility** — allows FlowDeck to read the accessibility tree and inject input.
2. **Screen Recording** — allows FlowDeck to capture window screenshots (macOS 14+).

Check and request permissions:

```bash theme={null}
flowdeck ui mac check-permissions
flowdeck ui mac request-permissions
```

After granting permissions in System Settings > Privacy & Security, restart your terminal for changes to take effect.

## Quick Start

```bash theme={null}
# 1) Check permissions
flowdeck ui mac check-permissions

# 2) List running apps
flowdeck ui mac list apps

# 3) Start background session capture (trees + screenshots).
#    Always pass --json so you can parse `latest_screenshot` and
#    `latest_tree` paths from the response.
flowdeck ui mac session start --app "Safari" --json

# 4) Drive one interaction (background by default: no cursor capture, no focus steal)
flowdeck ui mac click "Downloads" --app "Safari"

# 5) Capture visual proof
flowdeck ui mac screen --app "Safari" --output /tmp/flowdeck-mac-ui-proof.png

# 6) End session
flowdeck ui mac session stop
```

<Note>
  The `--app` flag accepts an app name (e.g., `"Safari"`), bundle ID (e.g., `"com.apple.Safari"`), or PID (e.g., `"12345"`). It is required on most commands.
</Note>

## Background Mode (the default)

Action commands drive the app **without moving your cursor or stealing focus**, and you don't pass a flag to get that. Events are delivered straight to the target app through the accessibility layer (and window-targeted system events for coordinate clicks), so the app doesn't need to be frontmost — you keep working while automation runs.

```bash theme={null}
# These run against a backgrounded app; your foreground window keeps focus
flowdeck ui mac click "saveButton" --app "MyApp" --by-id
flowdeck ui mac type "hello@example.com" --app "MyApp"
flowdeck ui mac scroll --direction down --app "MyApp"
flowdeck ui mac hotkey "cmd+s" --app "MyApp"
```

To type into a field, focus it first with a tap, then type:

```bash theme={null}
flowdeck ui mac click "emailField" --app "MyApp" --by-id
flowdeck ui mac type "user@example.com" --app "MyApp"
```

<Note>
  Both element targets and `--point` coordinates work in the background. An accessibility id or label is still the more robust choice, because coordinates break when the layout shifts.
</Note>

## Foreground Mode (`--foreground`)

`--foreground` opts out: events go to the global event tap, which moves your real cursor and brings the app forward. Five commands have no background path and **refuse to run** without it:

| Command        | Why it can't run in the background                            |
| -------------- | ------------------------------------------------------------- |
| `drag`         | Continuous pointer gesture; the recognizer needs a key window |
| `swipe`        | Same as `drag`                                                |
| `move`         | Moving the pointer *is* taking the pointer                    |
| `activate`     | Raising an app *is* taking the screen                         |
| `window focus` | Same as `activate`                                            |

```bash theme={null}
flowdeck ui mac drag --from 100,200 --to 400,500 --app "MyApp" --foreground
```

<Warning>
  If you are driving FlowDeck from an agent, treat the refusal as a prompt to **ask the user**, not to re-run with `--foreground`. A background action that failed to find its element is a targeting problem — re-read the accessibility tree or use `scroll --until` — and there is deliberately no environment variable or config key that disables the gate for a whole session.
</Warning>

<Note>
  `--background` still parses but is a no-op, kept so existing scripts don't break. Earlier versions had this backwards: foreground was the default and `--background` was opt-in, so one forgotten flag interrupted whatever you were doing.
</Note>

## Tangible Workflows

### 1) Automate a macOS app form fill

Use this to fill out a form in a native macOS app and verify the result.

```bash theme={null}
flowdeck ui mac session start --app "MyApp"
flowdeck ui mac click "nameField" --app "MyApp" --by-id
flowdeck ui mac type "John Doe" --app "MyApp"
flowdeck ui mac click "emailField" --app "MyApp" --by-id
flowdeck ui mac type "john@example.com" --app "MyApp"
flowdeck ui mac click "submitButton" --app "MyApp" --by-id
flowdeck ui mac wait "Success" --app "MyApp" --timeout 10
flowdeck ui mac screen --app "MyApp" --output /tmp/form-result.png
flowdeck ui mac session stop
```

### 2) Drive a menu-based workflow

```bash theme={null}
flowdeck ui mac session start --app "TextEdit"
flowdeck ui mac menu list --app "TextEdit"
flowdeck ui mac menu click "File > Export as PDF" --app "TextEdit"
flowdeck ui mac wait "Save" --app "TextEdit" --timeout 5
flowdeck ui mac session stop
```

### 3) Window management

```bash theme={null}
flowdeck ui mac session start --app "Safari"
flowdeck ui mac window list --app "Safari"
flowdeck ui mac window move --app "Safari" --to 100,100
flowdeck ui mac window resize --app "Safari" --size 1200,800
flowdeck ui mac window focus --app "Safari" --index 1 --foreground
flowdeck ui mac session stop
```

### 4) Verify a build output visually

Build and run your macOS app, then capture its UI:

```bash theme={null}
flowdeck run -D "My Mac"
flowdeck ui mac session start --app "MyApp"
flowdeck ui mac screen --app "MyApp" --output /tmp/current-ui.png
flowdeck ui mac session stop
```

Compare `/tmp/current-ui.png` with the expected design and iterate.

## App Targeting

The `--app` flag resolves targets in this order:

1. **Numeric** — treated as a PID (e.g., `--app "12345"`)
2. **Contains a dot** — treated as a bundle ID (e.g., `--app "com.apple.Safari"`)
3. **Otherwise** — fuzzy-matched against running app names (e.g., `--app "Safari"`)

To see all running GUI apps:

```bash theme={null}
flowdeck ui mac list apps
flowdeck ui mac list apps --include-agents --include-system
```

## Differences from iOS Simulator Automation

| Feature                 | iOS (`ui simulator`)                          | macOS (`ui mac`)                                    |
| ----------------------- | --------------------------------------------- | --------------------------------------------------- |
| Target flag             | `-S` (simulator name/UDID)                    | `--app` (name/bundle ID/PID)                        |
| Click/tap               | `tap`                                         | `click` (`tap` is a hidden alias)                   |
| Right-click             | N/A                                           | `right-click`                                       |
| Drag                    | N/A                                           | `drag --from x,y --to x,y`                          |
| Cursor movement         | N/A                                           | `move --point x,y`                                  |
| Keyboard shortcuts      | N/A                                           | `hotkey "cmd+s"`                                    |
| Key input               | HID keycodes only                             | `--name` (e.g., `return`) or `--keycode`            |
| Menu interaction        | N/A                                           | `menu list` / `menu click`                          |
| Window management       | N/A                                           | `window list/move/resize/focus`                     |
| App lifecycle           | N/A                                           | `launch` / `activate` / `quit`                      |
| Sessions                | Background capture sessions                   | Background capture sessions                         |
| Assert                  | `assert visible/hidden/enabled/disabled/text` | `assert visible/hidden/enabled/disabled/text`       |
| Scroll                  | Fraction-based `--distance`                   | Tick-based `--amount`                               |
| Scroll `--timeout` unit | Milliseconds                                  | **Seconds**                                         |
| Scroll until            | `scroll --until`                              | `scroll --until`                                    |
| Swipe                   | Direction argument                            | `--direction` flag                                  |
| Multi-touch             | `pinch`, `rotate`                             | N/A                                                 |
| Hardware buttons        | `flowdeck simulator button home/lock/...`     | N/A                                                 |
| Deep links              | `open-url`                                    | N/A                                                 |
| Appearance              | `flowdeck simulator appearance light/dark`    | N/A                                                 |
| Coordinates             | Points (normalized)                           | Screen-absolute points                              |
| Coordinate format       | Comma-separated `x,y`                         | Comma-separated `x,y` (space-separated is rejected) |
| `-v` / `--verbose`      | Most commands                                 | **Only** `session start` / `session stop`           |
| `--examples` / `-e`     | Supported on every command                    | Supported on every command                          |
| Permissions             | None needed                                   | Accessibility + Screen Recording                    |

## Performance and Reliability Tips

* Prefer accessibility identifiers and use `--by-id` for clicks, finds, and assertions (fastest and most reliable).
* For automation loops, start a session and read `latest-tree.json`/`latest.jpg` from disk instead of calling `screen` every step.
* For agent loops, run `flowdeck ui mac session start --app "MyApp" --json` to capture tree + screenshots in the background and read from `~/.flowdeck/automation/mac-sessions/<session-short-id>/` (use `latest.json`, `latest.jpg`, and `latest-tree.json` to find the newest capture). Starting a session stops any active macOS UI session first.
* `session start` returns the session directory plus `latest_screenshot` and `latest_tree` paths in JSON.
* macOS UI sessions stop automatically when the owner process that started them exits. Set `FLOWDECK_OWNER_PID` to bind a session to a specific owner process.
* Use `flowdeck ui mac screen --tree --json --app "MyApp"` when you only need the accessibility tree (no screenshot).
* Coordinates are screen-absolute points matching the values returned by `find`. Do not scale by Retina factors.
* Use `flowdeck ui mac find` to discover element labels and IDs before interacting.
* For smooth scrolling, use `--smooth` on the `scroll` command.
* Tune input timing with environment variables:
  * `FLOWDECK_HID_STABILIZATION_MS` (default 25) for click/gesture stability
  * `FLOWDECK_TYPE_DELAY_MS` (default 1) for typing speed

<Tip>
  For the complete command list and flags, see the [macOS UI Automation command reference](/cli/commands/ui/mac-automation).
</Tip>