Gamer: New AI Learns to Play Any Video Game
A new AI named Gamer can learn to play almost any video game, showing it can master complex tasks and interact across different digital worlds.
By Sci Herald Editorial Team ·
Based on research by Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren, first posted to arXiv in November 2026 — view the original study
TL;DR: A new AI named Gamer can learn to play almost any video game by watching the screen and using the keyboard and mouse like a human. This ability to master complex digital worlds is a major step toward creating robots and assistants that can operate effectively in our messy physical reality.
Key Facts
- The AI, called Gamer, was trained on tens of thousands of hours of human gameplay footage from a wide variety of video games.
- It interacts with games non-invasively, using only screen pixels as input and generating keyboard and mouse commands as output.
- The system uses a novel training method that combines learning from demonstrations with a form of trial-and-error learning to master long, complex tasks.
- A custom-built "cloud-edge inference engine" allows the AI to play games in real-time with low latency, a critical requirement for effective gameplay.
- Gamer demonstrated performance comparable to skilled human players in challenging 3D open-world games and competitive results on classic 2D game benchmarks.
From Pixels to Actions: An AI That Plays Like a Person
Imagine an AI that doesn't need special codes or inside information to play a game. It just watches the screen and uses the keyboard and mouse. That’s the core of Gamer, a new artificial intelligence system developed by a large team of researchers led by Run Luo, Kuan Zhang, and Qi Deng. Their work, first shared in November 2026, represents a significant leap toward creating a general-purpose AI that can operate across many different digital environments.
Most previous game-playing AIs were built for one specific game, using customized controls or access to the game’s internal code. Gamer is different. It’s a generalist agent—an AI built to handle long and complex tasks in a variety of interactive settings. Its goal is to interact with the digital world the only way a human can: by looking and clicking.
How Does the Gamer AI Actually Learn?
The training process for Gamer is a massive undertaking that happens in several stages, much like teaching a person a complex skill.
First, the AI undergoes continued pre-training (CPT). The researchers fed it a colossal dataset made from "tens of thousands of hours of device-level keyboard and mouse interaction" recorded from people playing games. This is the equivalent of a student spending years watching expert players, absorbing the basic mechanics of how to control a game. <figure class="my-10 overflow-hidden border border-slate-800 rounded-2xl bg-white text-slate-100 shadow-xl">
</figure>
The AI plays the games itself, and its actions are rewarded or penalized based on their success. The researchers developed a special method called Temporal-Aware GRPO to handle the unique challenges of video games, such as rewards that only come after a long sequence of actions (like finally defeating a boss after a 30-minute battle).
The Need for Speed: Playing Games in Real Time
One of the biggest technical hurdles wasn't just making an AI that can play games, but making one that can play them in real time. If an AI takes too long to think, the game state changes, and its action becomes useless or even harmful. This was a critical bottleneck for both deployment and further training.
To solve this, the team built a powerful inference engine called Streaming-vLLM. The researchers tested Gamer extensively.
Its most impressive feats were in "multiple challenging 3D open-world games," where it demonstrated "solid generalization and operational efficiency comparable to skilled players." While the source text doesn't name all the specific games, it highlights Honkai: Star Rail (HSR) as a key testing ground. HSR is a complex game that combines exploration, turn-based combat, and menu navigation—requiring the AI to master a wide array of skills within a single world. <figure class="my-10 overflow-hidden border border-slate-800 rounded-2xl bg-white text-slate-100 shadow-xl">
</figure>
Gamer also "achieves competitive results on a range of conventional 2D game benchmarks," proving its flexibility across different genres and eras of gaming. This cross-game performance is the strongest evidence that the AI is learning a general understanding of interaction, not just memorizing one game.
"These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms," Run Luo and colleagues write in the paper.
Why This Matters Beyond the Game Screen
This research is about far more than creating the ultimate automated gamer. Video games are a perfect, controlled testing ground for the much bigger challenge: building AI that can operate in the real world.
The skills required to play a modern video game—understanding a visual scene, making a plan, and executing precise physical actions—are the same foundational skills needed for a robot to navigate a kitchen or for a digital assistant to help you use complex software. The paper states that this approach "offers a path toward generalist agents that can interact across digital and physical worlds."
This work connects to other advances in AI that learns from watching and doing. For instance, research like SLA: AI Learns to Grab and Move Objects Like Humans and DreamX: AI Learns to Act Like People By Watching Videos is tackling similar problems of physical interaction. Gamer applies this concept to the rich, complex simulations of digital worlds.
The Bigger Picture
For years, a major goal in artificial intelligence has been to create a generalist agent—a single AI that can adapt and perform well on a wide variety of tasks, much like a human can. This is in stark contrast to most of today's AI, which are specialists excel in one narrow area but fail miserably at anything else.
The field of embodied AI focuses on agents that can interact with their environment, whether digital or physical. Gamer sits squarely in this domain. It tackles the monumental challenge of long-horizon tasks—goals that require many steps to complete, with actions that have consequences much later. Making a cup of coffee or completing a video game quest are both long-horizon tasks that require planning, memory, and adaptability.
Gamer represents significant progress by showing that a single model can learn a unified way to interact (via keyboard/mouse) and apply it across many domains. It also provides a blueprint for how to train such agents at a massive scale. However, the authors are clear about the limitations. The real physical world is far more complex and unpredictable than even the most detailed game. Transferring these skills from the digital to the physical realm remains a formidable challenge for future work.
Paper, Code and Resources
- Paper: Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent on arXiv (a preprint server; not yet peer reviewed)
- Code: RainBowLuoCS/Gamer on GitHub
- Project page: mihoyo-gamer.github.io
- Discussion: Hugging Face Papers — 103 community upvotes as of November 17, 2026
Frequently Asked Questions
What is a generalist AI agent?
A generalist AI agent is an artificial intelligence system designed to perform well across a wide range of different tasks, rather than being expert at just one. Gamer is a generalist agent for video games because it can learn to play many different types of games, from 2D classics to complex 3D open-world adventures.
Does the Gamer AI cheat to play the games?
No. According to the research, Gamer does not use any special access to the game’s internal code or state. It interacts with games "non-invasively," using only the screen pixels as input and generating standard keyboard and mouse commands as output—exactly how a human player does.
Could this AI be used for game testing?
Absolutely. The capabilities of Gamer could be very valuable for game developers. It could be used to automatically test games for bugs, balance difficulty by playing through levels countless times, or even provide a consistent benchmark for measuring the performance of other AI systems.
What's the next step for this research?
The immediate next steps involve scaling the technology further with more data and more complex games. The long-term goal, as stated by the researchers, is to use the lessons learned from digital worlds to build agents that can operate in the physical world. This means overcoming huge challenges like dealing with unpredictable real-world physics and the vast complexity of human environments.
Source: Read the original study