macOS UI Automation

FlowDeck macOS UI automation drives native macOS apps using the Accessibility framework and CGEvent-based input. Commands live under flowdeck ui mac.

Common Flags

Most macOS UI automation commands support:

--app to target a specific app by name, bundle ID, or PID. Required on most commands. Examples: --app "Safari", --app "com.apple.Safari", --app "12345".
--json for machine-readable output.
--examples / -e to print usage examples for the command.
-h / --help for the help screen.

Leaf action commands (click, double-click, right-click, type, erase, key, hotkey, scroll, move, drag, swipe, find, list, wait, assert, launch, activate, quit, screen, window/menu subcommands, permission commands) accept only --json, --examples/-e, and -h beyond their command-specific flags. They do not accept -v/--verbose. Every command that can reach the cursor also takes --foreground. Actions run in the background by default — no cursor movement, no focus change — and --foreground opts out. drag, swipe, move, activate, and window focus have no background path and require it. See macOS Automation for the full rationale. Session subcommands (session start, session stop) additionally accept -v / --verbose. App resolution order:

Numeric string → PID
Contains a dot → bundle ID
Otherwise → fuzzy match against running app names

Permissions

Command	Purpose	Key Options
`ui mac check-permissions`	Check Accessibility and Screen Recording permissions	`--json`
`ui mac request-permissions`	Trigger system permission dialogs	`--json`

Capture

Command	Purpose	Key Options
`ui mac screen`	Screenshot + accessibility tree from app window	`--app`, `--tree`, `--mode`, `--output`, `--json`
`ui mac session start`	Start background tree + screenshot capture	`--app`, `--interval-ms`, `--retention-seconds`, `--json`
`ui mac session stop`	Stop the active capture session	`--json`

Query

Command	Purpose	Key Options
`ui mac find`	Locate elements by label/ID/role	`--app`, `--by-id`, `--by-role`, `--contains`
`ui mac list apps`	List running GUI apps	`--include-agents`, `--include-system`
`ui mac list windows`	List app windows	`--app`
`ui mac list screens`	List connected displays	`--json`
`ui mac list permissions`	Check permission status	`--json`

Interaction

Command	Purpose	Key Options
`ui mac click`	Click element or coordinates	`--app`, `--point`, `--by-id`, `--duration`
`ui mac double-click`	Double-click element or coordinates	`--app`, `--point`, `--by-id`
`ui mac right-click`	Right-click element or coordinates	`--app`, `--point`, `--by-id`
`ui mac type`	Type text into focused element	`--app`, `--clear`, `--delay-ms`, `--mask`
`ui mac erase`	Erase text from focused field	`--app`, `--characters`, `--foreground`
`ui mac scroll`	Scroll in a direction	`--app`, `--direction`, `--amount`, `--smooth`, `--until`, `--timeout`
`ui mac move`	Move cursor to a point	`--point`, `--app`
`ui mac drag`	Drag from one point to another (foreground-only)	`--app`, `--from`, `--to`, `--duration`, `--foreground`
`ui mac swipe`	Swipe in a direction (foreground-only)	`--app`, `--direction`, `--distance`, `--duration`, `--foreground`

Keyboard

Command	Purpose	Key Options
`ui mac key`	Press a key by name or keycode	`--app`, `--name`, `--keycode`
`ui mac hotkey`	Press a keyboard shortcut	`--app`, combo argument (e.g., `cmd+s`)

Wait and Assert

Command	Purpose	Key Options
`ui mac wait`	Wait for element condition	`--app`, `--condition`, `--timeout`, `--by-id`
`ui mac assert visible`	Assert element is visible	`--app`, `--by-id`
`ui mac assert hidden`	Assert element is hidden	`--app`, `--by-id`
`ui mac assert enabled`	Assert element is enabled	`--app`, `--by-id`
`ui mac assert disabled`	Assert element is disabled	`--app`, `--by-id`
`ui mac assert text`	Assert element text matches	`--app`, `--expected`, `--contains`, `--by-id`

App Lifecycle

Command	Purpose	Key Options
`ui mac launch`	Launch an app by bundle ID	`--bundle-id`
`ui mac activate`	Bring an app to front (foreground-only)	`--app`, `--foreground`
`ui mac quit`	Quit an app	`--app`, `--force`

Window Management

Command	Purpose	Key Options
`ui mac window list`	List app windows	`--app`
`ui mac window move`	Move a window	`--app`, `--to`, `--index`
`ui mac window resize`	Resize a window	`--app`, `--size`, `--index`
`ui mac window focus`	Focus a window (foreground-only)	`--app`, `--index`, `--foreground`

Command	Purpose	Key Options
`ui mac menu list`	List app menu bar items	`--app`
`ui mac menu click`	Click a menu item by path	`--app`, path argument

Key Options Explained

Screen and Sessions

--output sets the screenshot file path (PNG). If omitted, FlowDeck writes to a temp file and prints the path.
--tree returns only the accessibility tree (no screenshot).
--mode (boolean; no value) captures the full screen instead of the app window. Use when the app window is partially offscreen or you need surrounding context (Dock, menu bar, other windows).
Screenshots require macOS 14+ and Screen Recording permission.
Screenshot dimensions are in pixels.
session start captures tree + screenshots every 500ms by default and writes to ~/.flowdeck/automation/mac-sessions/<session-short-id>/ (the short ID is the first 8 characters of the session UUID).
session stop takes no --app. At most one macOS UI session is active at a time, so stop is unambiguous.
Starting a session stops any active macOS session first.
--interval-ms and --retention-seconds tune capture frequency and retention (default 60s). Retention always keeps at least one capture.
Sessions write latest.json, latest.jpg, and latest-tree.json so automation can re-read the newest capture without listing the directory.
JSON output from session start includes the session directory, app name, PID, screens directory, trees directory, and the latest_screenshot / latest_tree paths.

flowdeck ui mac screen --app "Safari" --json
flowdeck ui mac screen --app "Safari" --output ./screen.png
flowdeck ui mac screen --app "Safari" --tree --json
flowdeck ui mac session start --app "Safari" --json
flowdeck ui mac session stop --json

Find

--by-id matches accessibility identifiers instead of labels.
--by-role matches element roles (e.g., button, textField).
--contains does a substring match against labels.
Returns element role, center coordinates, enabled state, and text value.
Suggestions are provided when no exact match is found.

flowdeck ui mac find "Settings" --app "MyApp"
flowdeck ui mac find "settings_button" --app "MyApp" --by-id
flowdeck ui mac find "button" --app "MyApp" --by-role
flowdeck ui mac find "Log" --app "MyApp" --contains

Click, Double-Click, and Right-Click

--by-id treats the target as an accessibility identifier.
--point expects x,y coordinates (screen-absolute, matching find output).
Pass <target> or --point, not both. The command errors if you supply neither.
--duration on click is a long-press hold time in seconds.
tap and double-tap are hidden aliases for click and double-click.

flowdeck ui mac click "Log In" --app "MyApp"
flowdeck ui mac click "login_button" --app "MyApp" --by-id
flowdeck ui mac click --point 120,340 --app "MyApp"
flowdeck ui mac right-click "item" --app "MyApp"

Type and Erase

--clear clears the focused field before typing (Select All + Delete).
--delay-ms overrides the per-character typing delay.
--mask hides the typed text in terminal output and JSON (useful for passwords).
erase --characters 5 deletes a specific number of characters; omit to clear all.

flowdeck ui mac type "hello@example.com" --app "MyApp"
flowdeck ui mac type "secret123" --app "MyApp" --mask
flowdeck ui mac type "New Value" --app "MyApp" --clear
flowdeck ui mac erase --app "MyApp"
flowdeck ui mac erase --characters 5 --app "MyApp"

Key and Hotkey

key --name accepts key names: return, escape, tab, delete, space, f1–f12, arrow keys, etc.
key --keycode accepts raw virtual keycodes.
Pass --name or --keycode, not both. The command errors if you supply neither.
hotkey accepts modifier+key combos: cmd+s, cmd+shift+z, ctrl+alt+delete.
Command-key combos are unreliable against a backgrounded app. Whether a cmd+... combo lands depends on how the target app installs the shortcut, and a combo that doesn’t land is still reported as success. Use ui mac menu click "File > Save" when the result matters — it goes through accessibility and always works in the background — or pass --foreground after asking the user.
Supported modifiers: cmd/command, shift, ctrl/control, alt/option.

flowdeck ui mac key --name return --app "MyApp"
flowdeck ui mac key --keycode 36 --app "MyApp"
flowdeck ui mac hotkey "ctrl+a" --app "MyApp"
flowdeck ui mac menu click "File > Save" --app "TextEdit"   # not hotkey "cmd+s"

Scroll

--direction is required: up, down, left, or right.
--amount sets the scroll magnitude in discrete ticks (default: 3).
--smooth enables many small ticks with delays for smooth scrolling.
--until scrolls repeatedly until the target element appears. Use id:myElement for accessibility ID matching.
--timeout sets the deadline for --until in seconds (default: 30).
Scroll is performed at the center of the app’s focused window.

iOS ui simulator scroll --timeout is in milliseconds; macOS ui mac scroll --timeout is in seconds. Same flag, different unit — never copy values across platforms.

flowdeck ui mac scroll --direction down --app "Safari"
flowdeck ui mac scroll --direction up --amount 10 --app "MyApp"
flowdeck ui mac scroll --direction down --amount 5 --smooth --app "MyApp"
flowdeck ui mac scroll --direction down --until "id:bottomButton" --app "MyApp"
flowdeck ui mac scroll --direction down --until "Save" --timeout 15 --app "MyApp"

Move, Drag, and Swipe

move --point expects x,y screen-absolute coordinates.
move --app is optional. The cursor moves to the absolute screen point regardless of which app is focused; the flag only validates that the named app exists.
move, drag, and swipe are foreground-only: they take the real cursor, so they require --foreground and exit non-zero without it.
drag --from and --to expect x,y coordinates. --duration is in seconds (default: 0.5).
swipe --direction accepts up, down, left, right. --distance is in points (default: 200).

flowdeck ui mac move --point 500,300 --app "MyApp"
flowdeck ui mac drag --from 100,200 --to 400,500 --app "MyApp" --foreground
flowdeck ui mac drag --from 100,200 --to 400,500 --duration 1.0 --app "MyApp" --foreground
flowdeck ui mac swipe --direction up --app "MyApp" --foreground
flowdeck ui mac swipe --direction left --distance 400 --app "MyApp" --foreground

Wait

--condition accepts one of: exists (default), gone, enabled, stable. Invalid values are rejected with an error.
- exists — element is present in the accessibility tree
- gone — element is not present
- enabled — element is present and enabled = true
- stable — element’s frame stops moving between polls (use to wait for an animated transition to settle)
--timeout is in seconds (default: 30).
--by-id treats the target as an accessibility identifier.

flowdeck ui mac wait "Loading..." --app "MyApp"
flowdeck ui mac wait "Submit" --app "MyApp" --condition enabled --timeout 15
flowdeck ui mac wait "Toast" --app "MyApp" --condition gone
flowdeck ui mac wait "save_button" --app "MyApp" --by-id --condition enabled

Assert

Assert element conditions with immediate pass/fail (no polling). Exits with failure if the assertion does not hold. Subcommands:

Subcommand	Description
`visible <target>`	Assert the element is visible
`hidden <target>`	Assert the element is hidden
`enabled <target>`	Assert the element is enabled
`disabled <target>`	Assert the element is disabled
`text <target>`	Assert the element text matches

Common Options:

Option	Description
`--by-id`	Treat target as accessibility identifier
`--app`	Target app

Text Options:

Option	Description
`--expected <text>`	Expected text value (required)
`--contains`	Check whether the text contains the expected value

flowdeck ui mac assert visible "Profile" --app "MyApp"
flowdeck ui mac assert hidden "Spinner" --app "MyApp"
flowdeck ui mac assert enabled "Submit" --app "MyApp"
flowdeck ui mac assert disabled "Continue" --app "MyApp"
flowdeck ui mac assert text "Welcome" --app "MyApp" --expected "Hello"
flowdeck ui mac assert text "title_label" --app "MyApp" --by-id --expected "Dashboard" --contains

App Lifecycle

launch uses --bundle-id, not --app. It targets the installed bundle directly, not a running process. Discover bundle IDs with ui mac list apps --json.
activate --app brings a running app to the foreground. It requires --foreground, because raising an app takes the user’s screen. You rarely need it: every action command already reaches a background app.
quit --app sends a graceful terminate. --force force-quits.

flowdeck ui mac launch --bundle-id com.apple.Safari
flowdeck ui mac activate --app "Safari" --foreground
flowdeck ui mac quit --app "TextEdit"
flowdeck ui mac quit --app "MyApp" --force

Window Management

--index selects which window (default: 0, the first/main window).
window move --to expects x,y for the new origin.
window resize --size expects w,h for the new dimensions.

flowdeck ui mac window list --app "Safari" --json
flowdeck ui mac window move --app "Safari" --to 100,100
flowdeck ui mac window resize --app "Safari" --size 1200,800
flowdeck ui mac window focus --app "Safari" --index 1 --foreground

menu list shows the app’s menu bar hierarchy with titles, shortcuts, and enabled state.
menu click navigates a menu path using > separators.

flowdeck ui mac menu list --app "TextEdit" --json
flowdeck ui mac menu click "File > Export as PDF" --app "TextEdit"
flowdeck ui mac menu click "Edit > Find > Find..." --app "Safari"

List

list apps shows running GUI applications with PID, name, and bundle ID.
--include-agents includes background agent apps.
--include-system includes system processes.
list windows requires --app.
list screens shows connected displays with frame and scale info.
list permissions shows Accessibility and Screen Recording status.

flowdeck ui mac list apps --json
flowdeck ui mac list apps --include-agents --include-system
flowdeck ui mac list windows --app "Safari" --json
flowdeck ui mac list screens --json
flowdeck ui mac list permissions --json

Coordinate System

All coordinates are screen-absolute points (matching what find returns).
Multi-display setups use the global coordinate space (origin at primary display top-left).
Do not apply Retina scaling; use coordinates as reported by find and list screens.
Use flowdeck ui mac list screens to see display layouts and pixel densities.

Coordinate Format

All point and size flags on macOS use comma-separated x,y (or w,h for sizes). Space-separated values are rejected with a live error:

Invalid coordinates: 400 300 is not supported on macOS

Applies to --point, --from, --to, and --size across click, double-click, right-click, move, drag, window move, and window resize.

# Correct
flowdeck ui mac click --point 400,300 --app "MyApp"
flowdeck ui mac drag --from 100,100 --to 500,400 --app "MyApp" --foreground
flowdeck ui mac window resize --app "Safari" --size 1200,800

# Wrong — rejected
flowdeck ui mac click --point "400 300" --app "MyApp"

Timing Tuning

Set these environment variables to adjust input timing:

FLOWDECK_HID_STABILIZATION_MS adds settle time between input events (default: 25)
FLOWDECK_TYPE_DELAY_MS adds per-character typing delay (default: 1)

Performance and Reliability Tips

Start a session before any macOS UI work. Run flowdeck ui mac session start --app "MyApp" --json, parse the JSON output to get latest_screenshot and latest_tree, then read those files to inspect the current UI.
macOS UI sessions remain single-active and stop automatically when the owner process that started them exits. Set FLOWDECK_OWNER_PID to bind a session to a specific owner process.
Verify after every action. After each click/type/scroll, wait briefly, then re-read latest_screenshot and latest_tree to confirm the UI changed.
Use accessibility identifiers and --by-id whenever possible; label matching is slower and more ambiguous.
For automation loops, re-read latest-tree.json/latest.jpg from disk instead of issuing screen for every step. The session updates these files automatically.
Use ui mac screen --tree --json --app "MyApp" for one-off tree-only checks when you do not need a running session.

For a full help listing, run flowdeck ui mac --help.

Overview

FlowDeck for AI

Interactive Mode (TUI)

Core Features

Command Reference

Advanced

Support

macOS UI Automation

Common Flags

Permissions

Capture

Query

Interaction

Keyboard

Wait and Assert

App Lifecycle

Window Management

Menu Interaction

Key Options Explained

Screen and Sessions

Find

Click, Double-Click, and Right-Click

Type and Erase

Key and Hotkey

Scroll

Move, Drag, and Swipe

Wait

Assert

App Lifecycle

Window Management

Menu Interaction

List

Coordinate System

Coordinate Format

Timing Tuning

Performance and Reliability Tips

​Common Flags

​Permissions

​Capture

​Query

​Interaction

​Keyboard

​Wait and Assert

​App Lifecycle

​Window Management

​Menu Interaction

​Key Options Explained

​Screen and Sessions

​Find

​Click, Double-Click, and Right-Click

​Type and Erase

​Key and Hotkey

​Scroll

​Move, Drag, and Swipe

​Wait

​Assert

​App Lifecycle

​Window Management

​Menu Interaction

​List

​Coordinate System

​Coordinate Format

​Timing Tuning

​Performance and Reliability Tips

Common Flags

Permissions

Capture

Query

Interaction

Keyboard

Wait and Assert

App Lifecycle

Window Management

Menu Interaction

Key Options Explained

Screen and Sessions

Find

Click, Double-Click, and Right-Click

Type and Erase

Key and Hotkey

Scroll

Move, Drag, and Swipe

Wait

Assert

App Lifecycle

Window Management

Menu Interaction

List

Coordinate System

Coordinate Format

Timing Tuning

Performance and Reliability Tips