> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flowdeck.studio/llms.txt
> Use this file to discover all available pages before exploring further.

# UI Automation

> Interact with the simulator screen — tap, type, scroll, and take screenshots

FlowDeck can interact with the iOS simulator screen: tapping buttons, typing text, scrolling, swiping, and capturing screenshots. This powers both the AI agent and manual automation workflows.

## With the AI agent

UI automation is most powerful when driven by the agent. Describe the interaction you want and the agent handles the details:

```text theme={null}
Launch the app, navigate to the settings screen, and toggle the notifications switch.
```

```text theme={null}
Take a screenshot of the current simulator screen.
```

```text theme={null}
Fill in the login form with test credentials and tap Sign In. Tell me what happens.
```

```text theme={null}
Scroll down to the bottom of the feed and take a screenshot.
```

The agent uses the simulator's accessibility tree to find elements by label, role, or identifier — it doesn't rely on fixed coordinates, so it works even if the layout shifts.

## Screenshots

Capture a screenshot of the running simulator at any time:

* **From the agent:** ask for a screenshot in the conversation
* **From the UI:** click **Screenshot** in the simulator controls panel

Screenshots are saved to the session's output folder and shown inline in the conversation.

## How element targeting works

The agent reads the simulator's accessibility tree to find interactive elements. For best results, make sure your views have meaningful accessibility labels — the same labels that power VoiceOver also power automation.

<Tip>
  If the agent can't find a button, it's likely missing an accessibility label. Add `.accessibilityLabel("Settings")` to the view in SwiftUI, or set `accessibilityLabel` in UIKit.
</Tip>

## macOS automation

For macOS apps running on your Mac, the agent can interact with the app's window using macOS accessibility APIs. This requires Screen Recording permission and Accessibility permission for FlowDeck.

<Warning>
  macOS automation interacts with your real desktop, mouse, and keyboard. The agent will ask for your explicit approval before performing any macOS automation.
</Warning>

## Troubleshooting

### Agent can't find a UI element

Check that the element has an accessibility label or identifier. You can inspect the accessibility tree by asking the agent: "What elements are visible on screen right now?"

### Taps aren't registering

Make sure the simulator is booted and the app is in the foreground. Some animations need to finish before taps are registered — the agent handles this automatically with retries.

### Screenshots are blank or black

The simulator may still be booting or the app may not be in the foreground. Wait a moment and try again.