#The Basics

This page explains Midscene's vision-based approach, usage options, and key APIs to help you understand and use the GUI Agent. To try it first, go to Try the GUI Agent.

#A vision-based approach to automation

Midscene interprets screenshots, locates targets, performs actions, and observes the result. UI localization and actions do not depend on DOM data, selectors, or extra annotations. This has four main advantages:

• Serve the purpose of E2E testing: Operate software as a user would and verify complete workflows from their perspective. Act on visible text, icons, and layout, then check the rendered result, including selected states, error messages, and obscured elements.
• Work across UI stacks: HTML, Canvas, mobile, and desktop apps all appear as screenshots. Once a platform provides screenshots and actions, the same Agent APIs apply, reducing dependence on application internals.
• Align with mainstream model approaches: Gemini Computer Use and UI-TARS both use screenshot-driven interaction. Midscene's model adapters let users choose models through a consistent API and benefit from improvements in model capabilities.
• Affordable model costs: In the AppControlBench evaluation, Doubao Seed 2.1 Turbo ran 60 tasks and passed 58, with a total model API cost of about $0.59. Actual costs vary with the model, task, and cache usage.

This approach requires a model with UI localization capabilities; see supported models. Data extraction and page understanding can still include DOM data when needed; see the API reference.

#Choose how to use Midscene

Midscene provides a cross-platform GUI Agent for web, mobile, and desktop apps. It follows natural-language instructions to interact with interfaces, check results, and extract data. For testing, choose an integration approach based on your existing project setup and how you prefer to write test cases.

If you prefer YAML-based testing, use Midscene Test, an E2E testing framework for web, mobile, and desktop apps. Describe test steps and expected results in YAML, extend business operations with TypeScript, and let the framework handle setup, cleanup, retries, and execution reports.

If you already have a JavaScript/TypeScript test project or prefer writing tests in code, use the SDK to integrate the GUI Agent into frameworks such as Playwright Test or Vitest to add visual actions and assertions to your test cases.

#Key APIs

#Plan and interact

aiAct

aiAct accepts a goal described in natural language. It observes the interface, plans the next steps, locates the target elements, and executes the actions until the goal is complete. The prompt can also include assertions. Midscene verifies these assertions during execution and throws an error promptly if one fails.

aiAct is flexible and autonomous, so it works well when a task has multiple steps, conditional branches, or an uncertain execution path. During execution, aiAct continuously uses AI to plan from the latest interface state, which means it usually takes more time and tokens than an instant interaction.

Typical usage:

await agent.aiAct(
  'Search for headphones, add the first item to the cart, and confirm that the cart count changes to 1',
);

#Instant interactions

Instant interaction APIs perform one specified action. Their main job is to locate a UI element and execute a fixed operation on it.

These APIs do not plan a sequence of steps. A request such as “close the popup if it appears, then click the checkout button” should use aiAct; an instant interaction treats its prompt as a description of the target element, not as a workflow.

aiTap

aiTap locates and taps or clicks one element.

Typical usage:

await agent.aiTap('The checkout button in the shopping cart');

aiInput

aiInput locates an input field and enters a specified value. Its default replace mode clears the existing content before entering the new value.

Typical usage:

await agent.aiInput('The email address input', {
  value: 'user@example.com',
});

Other input modes are typeOnly, which preserves the existing content, and clear, which only clears the field.

Other instant interaction APIs include aiHover, aiClearInput, aiKeyboardPress, aiScroll, aiPinch, aiLongPress, aiDoubleClick, and aiRightClick. Platform support varies; see Planning and interaction in the API reference.

#Insight

Insight APIs observe the interface and return an analysis result without interacting with it. They use the current screenshot by default. On web pages, you can also pass domIncluded when the task requires DOM information that is not visible in the screenshot.

aiAssert

aiAssert checks a condition described in natural language. It resolves when the condition is true. When the condition is false, it throws an error that includes the reason returned by the model.

Typical usage:

await agent.aiAssert('The shopping cart contains one item and shows a subtotal');

aiQuery

aiQuery extracts structured data from the interface. Describe both the required data and its expected type or shape in the prompt.

Typical usage:

const items = await agent.aiQuery<
  Array<{ name: string; price: number }>
>('The products in the shopping cart, {name: string, price: number}[]');
// Example items value: [{ name: 'Wireless headphones', price: 99.9 }]

aiBoolean

aiBoolean answers a question about the interface and returns a boolean.

Typical usage:

const loginDialogVisible = await agent.aiBoolean(
  'Is the login dialog visible?',
);
// Example loginDialogVisible value: true

Related convenience methods include aiNumber for numbers and aiString or aiAsk for strings.

#Improve planning and localization when needed

• deepThink: enhance planning. Available on aiAct, it focuses on task decomposition and separates planning and element localization into different model calls. Use it for complex, multi-step tasks. See deepThink planning mode.
• deepLocate: enhance localization. Available on APIs that locate elements, such as aiAct and aiTap, it uses an additional model call to improve accuracy for small or visually ambiguous targets. See deep localization.

Both options are off by default. Enable them independently or combine them in aiAct. They add model-call overhead, so start with the defaults and enable them when you encounter planning or localization problems.

await agent.aiAct('Complete the checkout form and stop before placing the order', {
  deepThink: true,
});
await agent.aiTap('The cart icon in the upper-right corner', { deepLocate: true });

#Next steps

• Model Selection Guide: Learn when to use one Default model and when to combine models.
• Try the GUI Agent: Perform actions and check results in the Playground.
• Create a test project: Write and run YAML tests.
• API reference: Look up full options and platform support.