Could Computer-Using Agents Be AI’s Next Frontier? Anthropic Thinks So

Key Takeaways

• Anthropic’s Claude 3.5 Sonnet can now use a computer.
• Claude’s new ability lets the agent control a cursor and keyboard.
• The new feature reflects the evolution from AI that generates text to agents that perform actions.

Artificial Intelligence has made great strides in just a few years. But despite how smart contemporary agents are, their ability to actually do things remains limited.

Anthropic is among an emerging group of developers training agents to use computers, a notion that opens up new possibilities for the technology. 

Anthropic’s Computer Using AI

Launched in public beta on Tuesday, Oct. 22, the latest version of Anthropic’s Claude 3.5 Sonnet can control a computer’s cursor and input information via a virtual keyboard, “emulating the way people interact with their own computer,” the company said .

Going forward, users will be able to ask Claude to retrieve and modify documents, interact with websites, and use a potentially limitless array of software programs. 

In a sense, the new feature represents the logical progression of Anthropic’s focus on integration. Only now, with the right instructions, users can integrate any app they need. (Although it might not work with apps that restrict screenshot access for security reasons.) 

Alongside similar innovations from other developers, Claude’s new capabilities reflect an emerging trend in AI: the move from providing answers to performing actions.

From Answers to Action

Anyone who uses chatbots enough will be familiar with the frustration of being told the AI cannot perform a specific task but can explain how to do it with step-by-step instructions.

To address this issue, several startups have started building AI that can do things rather than simply answer questions. But most of these failed to impress. For example, Rabbit’s “large action model” was broadly labeled a flop.

However, when Microsoft introduced new Copilot capabilities, it celebrated the arrival of a new era of more autonomous AI agents that have evolved beyond the first generation of chatbots.

The Risks of AI Computer Use

Anthropic acknowledged that the technology comes with risks.

“Please be aware that computer use poses unique risks that are distinct from standard API features or chat interfaces,” the firm warned.

Users are advised not to grant model access to sensitive data such as login details to prevent them from being compromised. 

Anthropic also noted that Claude may inadvertently follow commands found on-screen, making it vulnerable to prompt injection attacks.