• English

      • Desktop environment setup

      Midscene automates desktop applications across Windows, macOS, and Linux using native mouse and keyboard controls or Remote Desktop Protocol (RDP), with built-in multi-display support. It works with Electron, Qt, native desktop apps, and cross-application workflows.

      This guide walks you through setting up the desktop environment required by Midscene Test, the JavaScript SDK, and Playground. You will first prepare the desktop environment and permissions, and then use Playground to verify screen capture, mouse and keyboard input, and model calls.

      Prepare the desktop environment

      Midscene requires screen capture and mouse/keyboard control permissions on the target desktop. Configure local desktop control or remote Windows desktop (RDP) below; if the required permissions and dependencies are already configured, skip directly to Verify in Playground.

      1. Local desktop permissions and dependencies

      • macOS: Grant two separate system permissions to the terminal or IDE that launches Playground or runs your scripts (such as Terminal, iTerm2, VS Code, Cursor, or WebStorm):
        1. Accessibility (for mouse and keyboard control): Enable your terminal or IDE in System Settings > Privacy & Security > Accessibility (see nut.js macOS setup).
        2. Screen Recording (for desktop screenshots): Enable your terminal or IDE in System Settings > Privacy & Security > Screen Recording. After granting access, completely quit and relaunch the terminal or IDE before starting Playground.
      • Windows: No extra setup is required for standard applications. However, Windows User Interface Privilege Isolation (UIPI) blocks non-elevated processes from injecting mouse clicks or keystrokes into windows running as Administrator (the cursor still moves to the target coordinates, but clicks and key presses are silently dropped). Prefer running the target app without Administrator privileges; if it must run elevated, launch the terminal or Node.js process running Midscene as Administrator as well (see Windows: clicks have no effect on some apps).
      • Linux (desktop GUI): Install ImageMagick for screen capture support.
      • Linux (headless server / CI): To run desktop automation on a headless Linux host (such as GitHub Actions), install Xvfb and ImageMagick, then enable headless mode to start a virtual display automatically:
      # Install dependencies
sudo apt-get install -y xvfb x11-xserver-utils imagemagick

      When using Xvfb, make sure the target application is started inside the same virtual display environment (not needed on machines with a physical display).

      2. Remote Windows desktop over RDP (optional)

      To automate a remote Windows desktop over RDP, install FreeRDP on the machine running Midscene and enable Remote Desktop on the target Windows host.

      Note that the Playground walkthrough below verifies local displays. To connect to a remote Windows session in code, see RDP options in the Desktop JavaScript SDK.

      Verify in Playground

      With Playground, you can verify screen capture, mouse and keyboard input, and model calls without writing code.

      Before starting, prepare your model configuration using a model API key or a ChatGPT subscription.

      Launch Playground

      Make sure Node.js (^20.19.0 || ^22.12.0 || >=24.0.0) is installed and both node -v and npx --version work in your terminal.

      1. Start the Desktop Playground CLI:
      npx --yes @midscene/computer-playground

      On headless Linux, enable a virtual display with the following command and run the target application in that same display:

      MIDSCENE_COMPUTER_HEADLESS_LINUX=true npx --yes @midscene/computer-playground
      1. In the Playground window that opens, click the gear icon in the top-right corner, paste your model configuration, and save.
      2. If your browser does not open automatically, visit the local Playground URL printed in the terminal, and keep the terminal process running.
      3. In the connection panel on the left, select the target Display to control, create the Agent, and place the target application window on that display.
      4. Confirm that the preview area shows the live screen of the selected display. If the preview is blank or fails to load, check the Screen Recording permission or Linux dependencies above.

      Run an action and verify the result

      Open a browser window on the target display, navigate to TodoMVC, and position the windows so that the Playground tab does not cover the TodoMVC page.

      1. In Playground, select aiAct, enter the following instruction, and run it (avoid moving the mouse or typing while the action is running):
      Add a todo named "Try Midscene", then mark it as completed.
      1. Once the completed todo item appears on the page, switch to aiAssert, enter the following assertion, and run it:
      The list contains a todo named "Try Midscene", and it is marked as completed.
      1. Confirm that the assertion passes and inspect the step-by-step screenshots in the results panel. At this point, desktop screen capture, mouse/keyboard input, and model calls have all been verified together.

      If execution fails, check the failed step and error message: for missing screenshots or unresponsive mouse/keyboard input, see the FAQ below; for model request errors, see Verify model connectivity.

      Clean up the example data

      After verification, delete the "Try Midscene" item in TodoMVC. If you plan to run the Midscene Test Desktop example next, switch to the All filter and make sure the list is completely empty (including completed items), as the example template checks for an empty list before running and does not overwrite existing items.

      See it in action

      Prompt (macOS): Open Safari and post a tweet announcing that Midscene supports AutoGLM. Use the AutoGLM video from the Downloads folder. (Full report)

      Use the desktop in tests or scripts

      Once screen capture and input are verified in Playground, you can integrate the desktop environment into your test or script project:

      Prepared itemHow to use it in your project
      Desktop permissions and dependenciesEnsure the terminal or IDE running your project has screen capture and input permissions; keep virtual display settings enabled on headless Linux.
      Target displaySet COMPUTER_DISPLAY_ID in the Midscene Test Desktop template, or pass the display ID via Desktop SDK options.
      Model configurationWrite your model settings to the project's .env file and load them as shown in the integration guide. Settings entered in Playground are stored only in the browser and are not automatically shared with your project.

      Choose your next step based on how you use Midscene:

      FAQ

      macOS: Script cannot move the mouse or press keys

      Controlling the mouse and keyboard on macOS requires Accessibility permissions. Open System Settings > Privacy & Security > Accessibility and make sure the toggle is enabled for the terminal or IDE running your script (such as Terminal, iTerm2, VS Code, Cursor, or WebStorm).

      If the toggle is already enabled but input still does not work, remove the app from the Accessibility list (using the - button), add it back, and restart the app—macOS occasionally caches stale permission signatures after app updates.

      Windows: Clicks have no effect on some apps

      If the mouse cursor moves to the right coordinates but clicks or keystrokes have no effect on a specific application window (while working normally in other apps like a browser or Notepad), check whether the target application is running as Administrator (elevated). Windows User Interface Privilege Isolation (UIPI) silently drops input events sent from a standard-privilege process to an elevated window without throwing an error.

      How to fix it:

      1. Recommended: Run the target application with standard user privileges (uncheck "Run this program as an administrator" in its shortcut or compatibility properties).
      2. Alternative: If the target application must run as Administrator, launch the terminal (such as PowerShell, Windows Terminal, or your IDE) running Midscene as Administrator as well so both processes share the same privilege level.

      Note: System-level shortcuts such as Win+Tab are handled directly by the Windows shell and still work even when UIPI blocks in-window input, which is why global shortcuts may work while clicks inside the app do not. Whenever the cursor reaches the right spot but clicks do nothing, check for privilege elevation first.

      Linux: Screenshots or interactions fail on a headless server

      Headless Linux environments (such as CI containers) have no physical display. Install Xvfb and ImageMagick, and enable headless mode when creating the Agent:

      sudo apt-get install -y xvfb x11-xserver-utils imagemagick

      Enable in code:

      const agent = await agentForComputer({ headless: true });

      Or enable via an environment variable:

      MIDSCENE_COMPUTER_HEADLESS_LINUX=true npx tsx example.ts

      Model calls fail

      Check the error message of the failed step and verify the model configuration in Playground settings or your project's .env file. Because device connectivity and model requests are independent, follow Verify model connectivity to troubleshoot API key, endpoint, or network issues. Note that CLI and project checks read your project's environment variables rather than browser-stored Playground settings.