AI That Controls Your Computer - Clicking, Typing, Scrolling On It’s Own

by Sander Saar

Overview of various tasks the AI can execute, ranging from browsing and job searches to coding and phone automation. Overview of current performance limitations, quirky mistakes, and built-in restrictions around sensitive actions like transactions.

For agents: machine-readable ARVP doc at /public/creators/sander-saar/videos/video_tDGZHW9fGgU/arvp.json · Browse this video library

Chapters

  1. 0:00 Use Cases and Capabilities

    Overview of various tasks the AI can execute, ranging from browsing and job searches to coding and phone automation.

    On screen: A man in a brown jacket speaks directly to the camera against a split background showing a computer recording of automated web tasks.

  2. 0:42 Limitations and Safety Guardrails

    Overview of current performance limitations, quirky mistakes, and built-in restrictions around sensitive actions like transactions.

    On screen: The presenter speaks against a dark background listing restricted AI actions.

Visual moment index

What the footage shows, shot by shot. It is derived from the video itself, independent of the commentary. Timestamps link to the exact second.

0:00–0:08 A man in a brown jacket speaks directly to the camera against a split background showing a computer recording of automated web tasks. On screen: WHAT ARE THE BEST VIEW POINTS OF THE GOLDEN GATE BRIDGE? BATTERY SPENCER Computer use for orchestrating tasks Move to 472, 682 Left click Type best pla sunrise…

0:08–0:11 The presenter gestures enthusiastically while the background interface displays Claude automated actions interacting with a calendar app. On screen: Calendar File Edit View Window Help October 2024 Sun 13 Mon 14 Tue 15 Wed 16 Thu 17 Fri 18 Sat 19 Columbus Day Indigenous Peop... Sunrise at Battery Spencer…

0:11–0:18 A man speaks excitedly to the camera in the lower frame while an automated browser navigates a personal website and Anthropic's careers page above him.

0:19–0:21 The presenter continues explaining as the upper screen recording demonstrates Claude searching on Amazon for a book by Yuval Noah Harari. On screen: Anthropic Computer Use Stop Invite Deploy WebView repsrr.dev amazon.com Sapiens Audible Audible Audiobook 00 26.09 Free with Audible trial Available instantly…

0:22–0:24 The host gestures toward the screen recording showing the automated agent browsing vacation listings on Airbnb. On screen: Airbnb | Vacation rentals, cabins, beach houses, & more - Mozilla Firefox Airbnb Stays Experiences Airbnb your home Where Search destinations Check in Add…

0:24–0:28 The presenter speaks enthusiastically to the camera in front of a screen recording demonstrating Airbnb automated navigation. On screen: Toggle Screen Control Bar Airbnb | Vacation rentals, cabins, beach houses, & more - Mozilla Firefox Airbnb | Vacation rentals, cab... https://www.airbnb.com…

0:28–0:32 The presenter gestures while discussing Claude's coding capabilities over screen recordings of Claude generating a website and opening Visual Studio Code. On screen: Visual Studio Code Editing evolved Start Walkthroughs Recent Show welcome page on startup clipboard icon 2. Then I'll open VSCode and navigate to the…

0:32–0:36 The speaker points toward a demonstration on a monitor displaying terminal code and iPhone mirroring integration. On screen: @mckaywrigley

0:36–0:38 A man in a brown shirt speaks excitedly in front of a screen displaying iPhone mirroring and code execution. On screen: @mckaywrigley

0:38–0:45 The host speaks and gestures in front of a screen demoing Claude Computer Use controlling a Doom game session. On screen: @amasad Claude Computer Use Demo Security Alert! Never provide access to sensitive accounts or data, as malicious web content can hijack Claude's behavior Chat…

0:45–0:48 The host smiles and talks with an Anthropic article showing Yellowstone photos on screen behind him. On screen: A| Even while we were recording demonstrations of computer use for today's launch, we encountered some amusing errors. In one, Claude accidentally clicked to…

0:48–0:54 The presenter speaks against a dark background listing restricted AI actions. On screen: Even while we were recording demonstrations of computer use for today's launch, we encountered some amusing errors. In one, Claude accidentally clicked to stop…

0:54–0:58 The presenter concludes his explanation in front of a screen displaying a Claude Computer Use demo on Replit. On screen: + Use Template 1K 27 Copy link Anthropic Computer Use replit Oct 24, 2024 Forked from ryantmreplit / anthropic-computer-control Demo of using Anthropic's…

Transcript

0:00 Speaker A: Next few months are going to be insane. Speaker A: Here's an AI that just doesn't talk but actually controls your computer. Speaker A: Enter Anthropic's Claude computer use which is just mind blowing to follow around, clicking, typing, scrolling all on its own Reimagine job search. Speaker A: I ask Claude to navigate to my websites, learn about it and then find a suitable role at Anthropic or something as simple as find the latest you all know Harare's book on Amazon or to find a place on Airbnb for the weekend that's within one hour distance from la.

0:25 Speaker A: It literally gives you more hours in a day, not just for mundane tasks but also complex tasks like coding and building a website shown in in their demo, some pushing the boundaries with iPhone, mirroring to take control of your phone to use all your apps, or using as an agent in Minecraft to build and play for you. Speaker A: Just imagine adding voice to guide it all. Speaker A: It's still expensive and slow and sometimes makes amusing mistakes like adding a detour from the task to browse photos of Yellowstone. Speaker A: It also has been trained to refuse to do any transactions on your behalf like purchasing, sending messages, using logins, submitting forms, etc.

0:55 Speaker A: Let me know how would you use it and if you need a guide to get started.