In an exciting development for developers and tech enthusiasts, Google has introduced a public preview of the Gemini 2.5 Computer Use model. This model, integral to Project Mariner, features advanced agentic capabilities in AI Mode, allowing interactions with graphical user interfaces (GUIs), particularly in web browsers.
The Gemini 2.5 Computer Use model operates through a systematic loop that continues until a specific task is complete. The process begins when a request is sent to the model. This request includes crucial inputs such as the user request, a screenshot of the current environment, and a history of recent actions performed by the user. By analyzing these inputs, the model generates a response, typically in the form of a function call that represents UI actions like clicking or typing.
Once the model has generated a response, the corresponding client-side code executes the received action. After executing the action, a new screenshot of the GUI and the current URL are sent back to the Computer Use model as a function response, effectively restarting the loop. This process allows for a seamless interaction between the user and the AI, enhancing the overall user experience.
The Gemini 2.5 Computer Use model supports a wide range of UI actions, including navigating back and forward, searching the web, going to specific URLs, cursor hovering, keyboard combinations, scrolling, and drag-and-drop functionalities. Google has showcased this model's capabilities through two high-speed examples:
In the first scenario, the model retrieves details for any pet with a California residency from a specified link and adds them as guests in a spa CRM, scheduling a follow-up appointment with a specialist. The second example involves organizing tasks within a chaotic board for an art club by navigating to a specific website and ensuring that notes are correctly categorized.While the Gemini 2.5 Computer Use model is primarily optimized for web browsers, Google has indicated promising results on mobile platforms through an AndroidWorld benchmark. Although it is not yet optimized for desktop OS-level control, the model has demonstrated strong performance across web and mobile control benchmarks when compared to competitors like Claude and OpenAI, showcasing leading quality for browser control with minimal latency.
Built upon the visual understanding and reasoning capabilities of Gemini 2.5 Pro, this model powers various features within Project Mariner and supports the agentic capabilities of AI Mode. Google has utilized the model for internal UI testing, significantly speeding up software development processes. Additionally, an early access program is available for third-party developers interested in building assistants and workflow automation tools using this innovative model.
The Gemini 2.5 Computer Use model is currently accessible in public preview via the Gemini API in Google AI Studio and Vertex AI, marking a substantial step forward in AI technology and user interface automation.
