Google Gemini AI Plays Pokemon Live on Twitch

A software engineer took it upon themselves to program Google Gemini 2.5 Pro to play through Pokemon Blue. The journey through gen 1 is being broadcasted live on Twitch under the name ‘Gemini Plays Pokemon.’ 

They also used custom toolings, which allows for a minimap and visual screenshots – variables that are analyzed by Google’s Gemini AI to help frame its decision making. The goal of the project is to see how well Gemini 2.5 Experimental plays Pokemon. Google Gemini 2.5 Pro passed Rock Tunnel in Pokemon Blue in roughly 12 days. 

“This stream showcases an early prototype of the model navigating the game. Expect occasional offline periods and potential full resets as the experiment progresses. A better stream UI is also in the works,” the creator shared in the Twitch bio. 

The creator laid out how the project works:

Core Components

• Game Interface: The code connects to an mGBA emulator running Pokémon Blue via a socket connection, allowing it to:
• Take screenshots of the current game state
• Send button press commands (A, B, Up, Down, etc.)
• Retrieve game status data like player position, Pokémon party info, and map details
• Decision Making: The system processes game screenshots with a grid overlay and sends them (along with some game state extracted from RAM) to Google’s Gemini 2.5 Pro, which analyzes the information given and decides what to do next.

The Game Loop

• Capture a screenshot and retrieve game status data
• Process the image with a grid overlay to help with spatial reasoning
• Send the screenshot and game information to the AI model
• Parse the AI’s response to determine which button to press
• Execute the button press and wait for the game to update
• Repeat the process for the next frame

“Messages are currently summarized roughly every 100 actions to reduce input tokens used. The summary replaces those messages. This number is subject to change as the experiment progresses.”

The Google Gemini AI 2.5 Pro is currently playing through a visually enhanced version of Pokemon Blue – specifically, it is playing Danny-E 33’s Gen 2 Graphics Patch, “which updates the visuals to Generation 2’s style.” This explains the coloring of the in-game visuals. The software engineer responsible for this project is unaffiliated with Google.