FlowDeck can interact with the iOS simulator screen: tapping buttons, typing text, scrolling, swiping, and capturing screenshots. This powers both the AI agent and manual automation workflows.
With the AI agent
UI automation is most powerful when driven by the agent. Describe the interaction you want and the agent handles the details:
The agent uses the simulator’s accessibility tree to find elements by label, role, or identifier — it doesn’t rely on fixed coordinates, so it works even if the layout shifts.
Screenshots
Capture a screenshot of the running simulator at any time:
- From the agent: ask for a screenshot in the conversation
- From the UI: click Screenshot in the simulator controls panel
Screenshots are saved to the session’s output folder and shown inline in the conversation.
How element targeting works
The agent reads the simulator’s accessibility tree to find interactive elements. For best results, make sure your views have meaningful accessibility labels — the same labels that power VoiceOver also power automation.
If the agent can’t find a button, it’s likely missing an accessibility label. Add .accessibilityLabel("Settings") to the view in SwiftUI, or set accessibilityLabel in UIKit.
macOS automation
For macOS apps running on your Mac, the agent can interact with the app’s window using macOS accessibility APIs. This requires Screen Recording permission and Accessibility permission for FlowDeck.
macOS automation interacts with your real desktop, mouse, and keyboard. The agent will ask for your explicit approval before performing any macOS automation.
Troubleshooting
Agent can’t find a UI element
Check that the element has an accessibility label or identifier. You can inspect the accessibility tree by asking the agent: “What elements are visible on screen right now?”
Taps aren’t registering
Make sure the simulator is booted and the app is in the foreground. Some animations need to finish before taps are registered — the agent handles this automatically with retries.
Screenshots are blank or black
The simulator may still be booting or the app may not be in the foreground. Wait a moment and try again. 