AI researchers have used all sorts of video games, from Street Fighter to Pictionary, to test new models — often more for amusement than utility. But Anthropic said that Pokémon proved to be a useful benchmark for Claude 3.7 Sonnet, which can effectively “think” through the sorts of puzzles the game contains.

A new Twitch channel called ClaudePlaysPokemon uses Claude's LLM capabilities to see if the AI can become a true Pokemon Master.

As I write this, it’s hunting a Mankey outside of Viridian City in Pokémon Red. It’s using the new Claude 3.7 Sonnet “hybrid reasoning model” and shows its thinking in a side window. The stream is less interesting and slower-paced than the original Twitch Plays Pokémon.

Anthropic’s AI Claude 3.7 Sonnet played Pokémon Red on Twitch, showcasing its decision-making. Could AI agents play games for humans someday?