Interact with the simulator screen — tap, type, scroll, and take screenshots
FlowDeck can interact with the iOS simulator screen: tapping buttons, typing text, scrolling, swiping, and capturing screenshots. This powers both the AI agent and manual automation workflows.
UI automation is most powerful when driven by the agent. Describe the interaction you want and the agent handles the details:
Launch the app, navigate to the settings screen, and toggle the notifications switch.
Take a screenshot of the current simulator screen.
Fill in the login form with test credentials and tap Sign In. Tell me what happens.
Scroll down to the bottom of the feed and take a screenshot.
The agent uses the simulator’s accessibility tree to find elements by label, role, or identifier — it doesn’t rely on fixed coordinates, so it works even if the layout shifts.
The agent reads the simulator’s accessibility tree to find interactive elements. For best results, make sure your views have meaningful accessibility labels — the same labels that power VoiceOver also power automation.
If the agent can’t find a button, it’s likely missing an accessibility label. Add .accessibilityLabel("Settings") to the view in SwiftUI, or set accessibilityLabel in UIKit.
For macOS apps running on your Mac, the agent can interact with the app’s window using macOS accessibility APIs. This requires Screen Recording permission and Accessibility permission for FlowDeck.
macOS automation interacts with your real desktop, mouse, and keyboard. The agent will ask for your explicit approval before performing any macOS automation.
Check that the element has an accessibility label or identifier. You can inspect the accessibility tree by asking the agent: “What elements are visible on screen right now?”
Make sure the simulator is booted and the app is in the foreground. Some animations need to finish before taps are registered — the agent handles this automatically with retries.
The simulator may still be booting or the app may not be in the foreground. Wait a moment and try again.
Was this page helpful?
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"FlowDeck","description":"Flow","colors":{"primary":"#FF4407","light":"#FF4407","dark":"#FF4407"},"logo":{"light":"https://mintcdn.com/flowdeck/1mH37I1K6jA8b08h/images/Logo-BW-WhiteBg.png?fit=max\u0026auto=format\u0026n=1mH37I1K6jA8b08h\u0026q=85\u0026s=6130dfa97a692bdca791c30dd99fa3d7","dark":"https://mintcdn.com/flowdeck/1mH37I1K6jA8b08h/images/Logo-BW-BlackBg.png?fit=max\u0026auto=format\u0026n=1mH37I1K6jA8b08h\u0026q=85\u0026s=ca0c89fed26da5b5624b97fff0dd0211","href":"https://flowdeck.studio"},"favicon":{"light":"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePicture.png?fit=max\u0026auto=format\u0026n=WYbJRp5-oO6kcB2T\u0026q=85\u0026s=a542d6694e7f4b658d7283da721205c7","dark":"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePictureDark.png?fit=max\u0026auto=format\u0026n=WYbJRp5-oO6kcB2T\u0026q=85\u0026s=daf8cce9606d106c12c4640b9e0fe852"},"api":{},"appearance":{"default":"dark"},"background":{"color":{"light":"#FFFFFF","dark":"#000000"}},"navbar":{"primary":{"type":"button","label":"Get FlowDeck","href":"https://flowdeck.studio/"}},"search":{"prompt":"Search FlowDeck docs..."},"seo":{"indexing":"navigable"},"metadata":{"timestamp":false},"styling":{"latex":false},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"flowdeck","hasAudioNarration":false,"actualSubdomain":"flowdeck","customDomains":["docs.flowdeck.studio"],"gitSource":{"type":"github","owner":"flowdeck-dev","repo":"flowdeck","deployBranch":"stable","contentDirectory":"docs","isPrivate":true,"mountPath":"","label":"","_id":"6ad9380a8934a879d89251bf"},"chroma":{"liveDeploymentHistoryId":"6ad9388bec81d57a48f60f02","collectionId":"b792c53e-f802-443d-856b-9e8d47d66db3","schemaVersion":"v3","latestBuiltDeploymentHistoryId":"6ad9388bec81d57a48f60f02"},"feedback":{"thumbs":true},"entitlements":{},"buildId":"6ad93afc4b43520dc7b19f1e","clientVersion":"0.0.3964","searchFilterCounts":[{"language":"default","tag":"FlowDeck CLI","count":74},{"language":"default","tag":"Command Reference","count":37},{"language":"default","tag":"FlowDeck for Mac","count":20},{"language":"default","tag":"Simulator Commands","count":11},{"language":"default","tag":"Interactive Mode (TUI)","count":11},{"language":"default","tag":"Core Actions","count":10},{"language":"default","tag":"IDE Features","count":10},{"language":"default","tag":"Core Features","count":9},{"language":"default","tag":"FlowDeck for AI","count":6},{"language":"default","tag":"Advanced","count":6}]},"assistantConfig":{"enableAskAiButton":false,"enableStarterQuestions":false,"enableAskAiSearchBar":true},"hasChatPermissions":false,"hasWebMcpPermissions":false,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-v7-file-tree","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"pause-universal-search-reconcile","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"per-site-search-delta","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-private-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-nav-generations","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"byok-sep-25","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"vite-origin","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-agent-model-picker","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-updated-9-15","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"privacy":{"consentBanner":{"enabled":false}},"isAtlas":false,"googleFontsInlined":false},"customCss":[],"page":{"pagePath":"macos/ui-automation","slug":"/macos/ui-automation","pageMetadata":{"title":"UI Automation","description":"Interact with the simulator screen — tap, type, scroll, and take screenshots","href":"/macos/ui-automation"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Interact with the simulator screen — tap, type, scroll, and take screenshots\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={code:`code`,li:`li`,p:`p`,pre:`pre`,span:`span`,strong:`strong`,ul:`ul`,..._provideComponents(),...props.components},{CodeBlock,Heading,Tip,Warning}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Tip)_missingMdxReference(`Tip`,true);if(!Warning)_missingMdxReference(`Warning`,true);return _jsxs(_Fragment,{children:[_jsx(_components.p,{children:`FlowDeck can interact with the iOS simulator screen: tapping buttons, typing text, scrolling, swiping, and capturing screenshots. This powers both the AI agent and manual automation workflows.`}),`\n`,_jsx(Heading,{level:`2`,id:`with-the-ai-agent`,children:`With the AI agent`}),`\n`,_jsx(_components.p,{children:`UI automation is most powerful when driven by the agent. Describe the interaction you want and the agent handles the details:`}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Launch the app, navigate to the settings screen, and toggle the notifications switch.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Take a screenshot of the current simulator screen.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Fill in the login form with test credentials and tap Sign In. Tell me what happens.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`2`,language:`text`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`text`,children:_jsxs(_components.code,{language:`text`,numberOfLines:`2`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{children:`Scroll down to the bottom of the feed and take a screenshot.`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{})})]})})}),`\n`,_jsx(_components.p,{children:`The agent uses the simulator’s accessibility tree to find elements by label, role, or identifier — it doesn’t rely on fixed coordinates, so it works even if the layout shifts.`}),`\n`,_jsx(Heading,{level:`2`,id:`screenshots`,children:`Screenshots`}),`\n`,_jsx(_components.p,{children:`Capture a screenshot of the running simulator at any time:`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`From the agent:`}),` ask for a screenshot in the conversation`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`From the UI:`}),` click `,_jsx(_components.strong,{children:`Screenshot`}),` in the simulator controls panel`]}),`\n`]}),`\n`,_jsx(_components.p,{children:`Screenshots are saved to the session’s output folder and shown inline in the conversation.`}),`\n`,_jsx(Heading,{level:`2`,id:`how-element-targeting-works`,children:`How element targeting works`}),`\n`,_jsx(_components.p,{children:`The agent reads the simulator’s accessibility tree to find interactive elements. For best results, make sure your views have meaningful accessibility labels — the same labels that power VoiceOver also power automation.`}),`\n`,_jsx(Tip,{children:_jsxs(_components.p,{children:[`If the agent can’t find a button, it’s likely missing an accessibility label. Add `,_jsx(_components.code,{children:`.accessibilityLabel(\"Settings\")`}),` to the view in SwiftUI, or set `,_jsx(_components.code,{children:`accessibilityLabel`}),` in UIKit.`]})}),`\n`,_jsx(Heading,{level:`2`,id:`macos-automation`,children:`macOS automation`}),`\n`,_jsx(_components.p,{children:`For macOS apps running on your Mac, the agent can interact with the app’s window using macOS accessibility APIs. This requires Screen Recording permission and Accessibility permission for FlowDeck.`}),`\n`,_jsx(Warning,{children:_jsx(_components.p,{children:`macOS automation interacts with your real desktop, mouse, and keyboard. The agent will ask for your explicit approval before performing any macOS automation.`})}),`\n`,_jsx(Heading,{level:`2`,id:`troubleshooting`,children:`Troubleshooting`}),`\n`,_jsx(Heading,{level:`3`,id:`agent-can’t-find-a-ui-element`,children:`Agent can’t find a UI element`}),`\n`,_jsx(_components.p,{children:`Check that the element has an accessibility label or identifier. You can inspect the accessibility tree by asking the agent: “What elements are visible on screen right now?”`}),`\n`,_jsx(Heading,{level:`3`,id:`taps-aren’t-registering`,children:`Taps aren’t registering`}),`\n`,_jsx(_components.p,{children:`Make sure the simulator is booted and the app is in the foreground. Some animations need to finish before taps are registered — the agent handles this automatically with retries.`}),`\n`,_jsx(Heading,{level:`3`,id:`screenshots-are-blank-or-black`,children:`Screenshots are blank or black`}),`\n`,_jsx(_components.p,{children:`The simulator may still be booting or the app may not be in the foreground. Wait a moment and try again.`})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"UI Automation","description":"Interact with the simulator screen — tap, type, scroll, and take screenshots","href":"/macos/ui-automation"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"With the AI agent","slug":"with-the-ai-agent","depth":2,"children":[]},{"title":"Screenshots","slug":"screenshots","depth":2,"children":[]},{"title":"How element targeting works","slug":"how-element-targeting-works","depth":2,"children":[]},{"title":"macOS automation","slug":"macos-automation","depth":2,"children":[]},{"title":"Troubleshooting","slug":"troubleshooting","depth":2,"children":[{"title":"Agent can’t find a UI element","slug":"agent-can’t-find-a-ui-element","depth":3,"children":[]},{"title":"Taps aren’t registering","slug":"taps-aren’t-registering","depth":3,"children":[]},{"title":"Screenshots are blank or black","slug":"screenshots-are-blank-or-black","depth":3,"children":[]}]}],"codeExamples":{}},"jsLinks":[],"subdomain":"flowdeck","siteName":"FlowDeck","title":"UI Automation - FlowDeck","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-06-12T18:55:22.374Z","originalFileLocation":"macos/ui-automation.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"dark\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Interact with the simulator screen — tap, type, scroll, and take screenshots\">\u003cmeta data-mint-head name=\"application-name\" content=\"FlowDeck\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePicture.png?fit=max\u0026amp;auto=format\u0026amp;n=WYbJRp5-oO6kcB2T\u0026amp;q=85\u0026amp;s=a542d6694e7f4b658d7283da721205c7\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePicture.png?fit=max\u0026amp;auto=format\u0026amp;n=WYbJRp5-oO6kcB2T\u0026amp;q=85\u0026amp;s=a542d6694e7f4b658d7283da721205c7\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePicture.png?fit=max\u0026amp;auto=format\u0026amp;n=WYbJRp5-oO6kcB2T\u0026amp;q=85\u0026amp;s=a542d6694e7f4b658d7283da721205c7\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/png\" href=\"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePicture.png?fit=max\u0026amp;auto=format\u0026amp;n=WYbJRp5-oO6kcB2T\u0026amp;q=85\u0026amp;s=a542d6694e7f4b658d7283da721205c7\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePictureDark.png?fit=max\u0026amp;auto=format\u0026amp;n=WYbJRp5-oO6kcB2T\u0026amp;q=85\u0026amp;s=daf8cce9606d106c12c4640b9e0fe852\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePictureDark.png?fit=max\u0026amp;auto=format\u0026amp;n=WYbJRp5-oO6kcB2T\u0026amp;q=85\u0026amp;s=daf8cce9606d106c12c4640b9e0fe852\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/png\" href=\"https://mintcdn.com/flowdeck/WYbJRp5-oO6kcB2T/images/ProfilePictureDark.png?fit=max\u0026amp;auto=format\u0026amp;n=WYbJRp5-oO6kcB2T\u0026amp;q=85\u0026amp;s=daf8cce9606d106c12c4640b9e0fe852\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"UI Automation - FlowDeck\">\u003cmeta data-mint-head property=\"og:description\" content=\"Interact with the simulator screen — tap, type, scroll, and take screenshots\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"FlowDeck\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.flowdeck.studio/macos/ui-automation\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://flowdeck.mintlify.app/_mintlify/api/og/v2?division=IDE+Features\u0026amp;title=UI+Automation\u0026amp;description=Interact+with+the+simulator+screen+%E2%80%94+tap%2C+type%2C+scroll%2C+and+take+screenshots\u0026amp;theme=d4ee9523425cdf7b8cb44c77\u0026amp;signature=aa280961b8020980350a945edc05923a9c811d5e0122d905303d2cf7caee2b4d\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"UI Automation - FlowDeck\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Interact with the simulator screen — tap, type, scroll, and take screenshots\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://flowdeck.mintlify.app/_mintlify/api/og/v2?division=IDE+Features\u0026amp;title=UI+Automation\u0026amp;description=Interact+with+the+simulator+screen+%E2%80%94+tap%2C+type%2C+scroll%2C+and+take+screenshots\u0026amp;theme=d4ee9523425cdf7b8cb44c77\u0026amp;signature=aa280961b8020980350a945edc05923a9c811d5e0122d905303d2cf7caee2b4d\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"https://mintlify.s3.us-west-1.amazonaws.com/flowdeck/_generated/favicon/browserconfig.xml?v=3\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"FlowDeck\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#FF4407\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.flowdeck.studio/macos/ui-automation\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/macos/ui-automation.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.flowdeck.studio/#organization\",\"name\":\"FlowDeck\",\"url\":\"https://docs.flowdeck.studio\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/flowdeck/1mH37I1K6jA8b08h/images/Logo-BW-WhiteBg.png?fit=max\u0026auto=format\u0026n=1mH37I1K6jA8b08h\u0026q=85\u0026s=6130dfa97a692bdca791c30dd99fa3d7\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.flowdeck.studio#website\",\"name\":\"FlowDeck\",\"url\":\"https://docs.flowdeck.studio\",\"publisher\":{\"@id\":\"https://docs.flowdeck.studio/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.flowdeck.studio/macos/ui-automation#webpage\",\"url\":\"https://docs.flowdeck.studio/macos/ui-automation\",\"name\":\"UI Automation\",\"description\":\"Interact with the simulator screen — tap, type, scroll, and take screenshots\",\"dateModified\":\"2026-06-12T18:55:22.374Z\",\"isPartOf\":{\"@id\":\"https://docs.flowdeck.studio#website\"},\"breadcrumb\":{\"@id\":\"https://docs.flowdeck.studio/macos/ui-automation#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.flowdeck.studio/macos/ui-automation#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"IDE Features\",\"item\":\"https://docs.flowdeck.studio/macos/building\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"UI Automation\",\"item\":\"https://docs.flowdeck.studio/macos/ui-automation\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.flowdeck.studio/macos/ui-automation#article\",\"headline\":\"UI Automation\",\"name\":\"UI Automation\",\"description\":\"Interact with the simulator screen — tap, type, scroll, and take screenshots\",\"url\":\"https://docs.flowdeck.studio/macos/ui-automation\",\"mainEntityOfPage\":{\"@id\":\"https://docs.flowdeck.studio/macos/ui-automation#webpage\"},\"dateModified\":\"2026-06-12T18:55:22.374Z\",\"publisher\":{\"@id\":\"https://docs.flowdeck.studio/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.flowdeck.studio#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"FlowDeck\",\"creator\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","siteName":"FlowDeck","nav":{"global":null,"tabs":[{"tab":"FlowDeck for Mac","groups":[{"group":"Getting Started","pages":[{"title":"FlowDeck for Mac","href":"/macos/introduction"},{"title":"Install FlowDeck","href":"/macos/install"},{"title":"The Sample Project","href":"/macos/sample-project"},{"title":"Keyboard Shortcuts","href":"/macos/keyboard-shortcuts"}]},{"group":"Projects \u0026 Sessions","pages":[{"title":"Projects \u0026 Sessions","href":"/macos/projects-and-sessions"}]},{"group":"The AI Agent","pages":[{"title":"The AI Agent","href":"/macos/ai-agent"},{"title":"Coding Standards","href":"/macos/engineering-standards"},{"title":"Validation","href":"/macos/validation"},{"title":"Skills Manager","href":"/macos/skills-manager"}]},{"group":"IDE Features","pages":[{"title":"Build \u0026 Run","href":"/macos/building"},{"title":"Testing","href":"/macos/testing"},{"title":"Simulators \u0026 Devices","href":"/macos/simulators"},{"title":"Disk Cleanup","href":"/macos/clean"},{"title":"Console","href":"/macos/logging"},{"title":"Source Control","href":"/macos/source-control"},{"title":"Linear","description":"Connect your Linear account to FlowDeck","href":"/macos/linear"},{"title":"UI Automation","href":"/macos/ui-automation"},{"title":"Open Quickly (⌘K)","description":"One search field for every project, session, and action in FlowDeck","href":"/macos/command-palette"},{"title":"Settings \u0026 External Tools","href":"/macos/settings"}]},{"group":"Support","pages":[{"title":"Support","href":"/macos/support"}]}]},{"tab":"FlowDeck CLI","pages":[{"title":"FlowDeck CLI","href":"/introduction"}]}]},"title":"UI Automation - FlowDeck","buildId":"dpl_HcHmUS6i3UwswnTHJUEdcTtwfdTQ","deploymentId":"dpl_HcHmUS6i3UwswnTHJUEdcTtwfdTQ","dynamics":[],"chunks":[]}
