Interact with Elements

Learn how to interact with UI elements using AskUI’s powerful tools and actions

Once you’ve selected an element, you can interact with it using AskUI’s comprehensive set of actions and tools. This guide covers the different ways to interact with elements, from basic clicks to complex keyboard operations.

Available interaction methods vary depending on the agent type you’re using. Desktop agents support different interactions than mobile agents.

Basic Actions

The most common actions for interacting with elements using click(), type(), and mouse_move():

from askui import VisionAgent

with VisionAgent() as agent:
    # Click actions
    agent.click("Login button")  # Single click - see click() API reference
    agent.click("Submit", button="right")  # Right click
    agent.click("Open", button="middle")  # Middle click
    
    # Text input
    agent.type("[e-mail]")  # Type text
    
    # Mouse movement
    agent.mouse_move("Menu item")  # Move to element
    agent.mouse_move(100, 200)  # Move to coordinates

Keyboard Operations

Control keyboard input for complex interactions using keyboard():

with VisionAgent() as agent:
    # Basic keyboard input
    agent.keyboard("enter")  # Press Enter
    agent.keyboard("tab")  # Press Tab
    
    # Keyboard shortcuts
    agent.keyboard("a", modifier_keys=["control"])  # Ctrl+A
    agent.keyboard("c", modifier_keys=["control"])  # Ctrl+C
    agent.keyboard("v", modifier_keys=["control"])  # Ctrl+V

Built-in Tools

AskUI provides several built-in tools for different types of interactions:

Web Browser Tools

Control web browser operations:

from askui import VisionAgent

with VisionAgent() as agent:
    # Open new browser window
    agent.tools.webbrowser.open_new("https://askui.com")
    
    # Open in new tab
    agent.tools.webbrowser.open_new_tab("https://docs.askui.com")

Clipboard Tools

Manage clipboard operations:

with VisionAgent() as agent:
    # Copy and paste
    agent.tools.clipboard.copy("Text to copy")
    result = agent.tools.clipboard.paste()

Next Steps

• Learn about extracting data from UI elements
• Explore working with forms
• Review best practices for interactions