macOS Automation

FlowDeck ships built-in UI automation for macOS apps so you can drive native interfaces, validate UI state, and capture accessibility trees from the CLI. Use flowdeck ui mac to run automation commands.

What It Does

Use macOS automation to:

Capture screenshots and accessibility trees from any running macOS app.
Click elements, type text, scroll, drag, and navigate menus.
Wait for UI state changes and assert UI conditions.
Manage windows, launch/quit apps, and send keyboard shortcuts.

When to Use It

Smoke tests for macOS app flows.
Scripted demos or QA checks on native Mac apps.
AI-driven interaction loops that need consistent UI state.
Automating menu-driven workflows or window management.

Prerequisites

macOS automation requires two system permissions:

Accessibility — allows FlowDeck to read the accessibility tree and inject input.
Screen Recording — allows FlowDeck to capture window screenshots (macOS 14+).

Check and request permissions:

flowdeck ui mac check-permissions
flowdeck ui mac request-permissions

After granting permissions in System Settings > Privacy & Security, restart your terminal for changes to take effect.

Quick Start

# 1) Check permissions
flowdeck ui mac check-permissions

# 2) List running apps
flowdeck ui mac list apps

# 3) Start background session capture (trees + screenshots).
#    Always pass --json so you can parse `latest_screenshot` and
#    `latest_tree` paths from the response.
flowdeck ui mac session start --app "Safari" --json

# 4) Drive one interaction (background by default: no cursor capture, no focus steal)
flowdeck ui mac click "Downloads" --app "Safari"

# 5) Capture visual proof
flowdeck ui mac screen --app "Safari" --output /tmp/flowdeck-mac-ui-proof.png

# 6) End session
flowdeck ui mac session stop

The --app flag accepts an app name (e.g., "Safari"), bundle ID (e.g., "com.apple.Safari"), or PID (e.g., "12345"). It is required on most commands.

Background Mode (the default)

Action commands drive the app without moving your cursor or stealing focus, and you don’t pass a flag to get that. Events are delivered straight to the target app through the accessibility layer (and window-targeted system events for coordinate clicks), so the app doesn’t need to be frontmost — you keep working while automation runs.

# These run against a backgrounded app; your foreground window keeps focus
flowdeck ui mac click "saveButton" --app "MyApp" --by-id
flowdeck ui mac type "hello@example.com" --app "MyApp"
flowdeck ui mac scroll --direction down --app "MyApp"
flowdeck ui mac hotkey "cmd+s" --app "MyApp"

To type into a field, focus it first with a tap, then type:

flowdeck ui mac click "emailField" --app "MyApp" --by-id
flowdeck ui mac type "user@example.com" --app "MyApp"

Both element targets and --point coordinates work in the background. An accessibility id or label is still the more robust choice, because coordinates break when the layout shifts.

Foreground Mode (`--foreground`)

--foreground opts out: events go to the global event tap, which moves your real cursor and brings the app forward. Five commands have no background path and refuse to run without it:

Command	Why it can’t run in the background
`drag`	Continuous pointer gesture; the recognizer needs a key window
`swipe`	Same as `drag`
`move`	Moving the pointer is taking the pointer
`activate`	Raising an app is taking the screen
`window focus`	Same as `activate`

flowdeck ui mac drag --from 100,200 --to 400,500 --app "MyApp" --foreground

If you are driving FlowDeck from an agent, treat the refusal as a prompt to ask the user, not to re-run with --foreground. A background action that failed to find its element is a targeting problem — re-read the accessibility tree or use scroll --until — and there is deliberately no environment variable or config key that disables the gate for a whole session.

--background still parses but is a no-op, kept so existing scripts don’t break. Earlier versions had this backwards: foreground was the default and --background was opt-in, so one forgotten flag interrupted whatever you were doing.

Tangible Workflows

1) Automate a macOS app form fill

Use this to fill out a form in a native macOS app and verify the result.

flowdeck ui mac session start --app "MyApp"
flowdeck ui mac click "nameField" --app "MyApp" --by-id
flowdeck ui mac type "John Doe" --app "MyApp"
flowdeck ui mac click "emailField" --app "MyApp" --by-id
flowdeck ui mac type "john@example.com" --app "MyApp"
flowdeck ui mac click "submitButton" --app "MyApp" --by-id
flowdeck ui mac wait "Success" --app "MyApp" --timeout 10
flowdeck ui mac screen --app "MyApp" --output /tmp/form-result.png
flowdeck ui mac session stop

flowdeck ui mac session start --app "TextEdit"
flowdeck ui mac menu list --app "TextEdit"
flowdeck ui mac menu click "File > Export as PDF" --app "TextEdit"
flowdeck ui mac wait "Save" --app "TextEdit" --timeout 5
flowdeck ui mac session stop

3) Window management

flowdeck ui mac session start --app "Safari"
flowdeck ui mac window list --app "Safari"
flowdeck ui mac window move --app "Safari" --to 100,100
flowdeck ui mac window resize --app "Safari" --size 1200,800
flowdeck ui mac window focus --app "Safari" --index 1 --foreground
flowdeck ui mac session stop

4) Verify a build output visually

Build and run your macOS app, then capture its UI:

flowdeck run -D "My Mac"
flowdeck ui mac session start --app "MyApp"
flowdeck ui mac screen --app "MyApp" --output /tmp/current-ui.png
flowdeck ui mac session stop

Compare /tmp/current-ui.png with the expected design and iterate.

App Targeting

The --app flag resolves targets in this order:

Numeric — treated as a PID (e.g., --app "12345")
Contains a dot — treated as a bundle ID (e.g., --app "com.apple.Safari")
Otherwise — fuzzy-matched against running app names (e.g., --app "Safari")

To see all running GUI apps:

flowdeck ui mac list apps
flowdeck ui mac list apps --include-agents --include-system

Differences from iOS Simulator Automation

Feature	iOS (`ui simulator`)	macOS (`ui mac`)
Target flag	`-S` (simulator name/UDID)	`--app` (name/bundle ID/PID)
Click/tap	`tap`	`click` (`tap` is a hidden alias)
Right-click	N/A	`right-click`
Drag	N/A	`drag --from x,y --to x,y`
Cursor movement	N/A	`move --point x,y`
Keyboard shortcuts	N/A	`hotkey "cmd+s"`
Key input	HID keycodes only	`--name` (e.g., `return`) or `--keycode`
Menu interaction	N/A	`menu list` / `menu click`
Window management	N/A	`window list/move/resize/focus`
App lifecycle	N/A	`launch` / `activate` / `quit`
Sessions	Background capture sessions	Background capture sessions
Assert	`assert visible/hidden/enabled/disabled/text`	`assert visible/hidden/enabled/disabled/text`
Scroll	Fraction-based `--distance`	Tick-based `--amount`
Scroll `--timeout` unit	Milliseconds	Seconds
Scroll until	`scroll --until`	`scroll --until`
Swipe	Direction argument	`--direction` flag
Multi-touch	`pinch`, `rotate`	N/A
Hardware buttons	`flowdeck simulator button home/lock/...`	N/A
Deep links	`open-url`	N/A
Appearance	`flowdeck simulator appearance light/dark`	N/A
Coordinates	Points (normalized)	Screen-absolute points
Coordinate format	Comma-separated `x,y`	Comma-separated `x,y` (space-separated is rejected)
`-v` / `--verbose`	Most commands	Only `session start` / `session stop`
`--examples` / `-e`	Supported on every command	Supported on every command
Permissions	None needed	Accessibility + Screen Recording

Performance and Reliability Tips

Prefer accessibility identifiers and use --by-id for clicks, finds, and assertions (fastest and most reliable).
For automation loops, start a session and read latest-tree.json/latest.jpg from disk instead of calling screen every step.
For agent loops, run flowdeck ui mac session start --app "MyApp" --json to capture tree + screenshots in the background and read from ~/.flowdeck/automation/mac-sessions/<session-short-id>/ (use latest.json, latest.jpg, and latest-tree.json to find the newest capture). Starting a session stops any active macOS UI session first.
session start returns the session directory plus latest_screenshot and latest_tree paths in JSON.
macOS UI sessions stop automatically when the owner process that started them exits. Set FLOWDECK_OWNER_PID to bind a session to a specific owner process.
Use flowdeck ui mac screen --tree --json --app "MyApp" when you only need the accessibility tree (no screenshot).
Coordinates are screen-absolute points matching the values returned by find. Do not scale by Retina factors.
Use flowdeck ui mac find to discover element labels and IDs before interacting.
For smooth scrolling, use --smooth on the scroll command.
Tune input timing with environment variables:
- FLOWDECK_HID_STABILIZATION_MS (default 25) for click/gesture stability
- FLOWDECK_TYPE_DELAY_MS (default 1) for typing speed

For the complete command list and flags, see the macOS UI Automation command reference.

Overview

FlowDeck for AI

Interactive Mode (TUI)

Core Features

Command Reference

Advanced

Support

What It Does

When to Use It

Prerequisites

Quick Start

Background Mode (the default)

Foreground Mode (`--foreground`)

Tangible Workflows

1) Automate a macOS app form fill

2) Drive a menu-based workflow

3) Window management

4) Verify a build output visually

App Targeting

Differences from iOS Simulator Automation

Performance and Reliability Tips

​What It Does

​When to Use It

​Prerequisites

​Quick Start

​Background Mode (the default)

​Foreground Mode (--foreground)

​Tangible Workflows

​1) Automate a macOS app form fill

​2) Drive a menu-based workflow

​3) Window management

​4) Verify a build output visually

​App Targeting

​Differences from iOS Simulator Automation

​Performance and Reliability Tips

What It Does

When to Use It

Prerequisites

Quick Start

Background Mode (the default)

Foreground Mode (`--foreground`)

Tangible Workflows

1) Automate a macOS app form fill

2) Drive a menu-based workflow

3) Window management

4) Verify a build output visually

App Targeting

Differences from iOS Simulator Automation

Performance and Reliability Tips