Gamer: New AI Learns to Play Any Video Game

A new AI named Gamer can learn to play almost any video game, showing it can master complex tasks and interact across different digital worlds.

By Sci Herald Editorial Team ·

Based on research by Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren, first posted to arXiv in November 2026 — view the original study

Gamer: New AI Learns to Play Any Video Game

TL;DR: A new AI named Gamer can learn to play almost any video game by watching the screen and using the keyboard and mouse like a human. This ability to master complex digital worlds is a major step toward creating robots and assistants that can operate effectively in our messy physical reality.

Key Facts

From Pixels to Actions: An AI That Plays Like a Person

Imagine an AI that doesn't need special codes or inside information to play a game. It just watches the screen and uses the keyboard and mouse. That’s the core of Gamer, a new artificial intelligence system developed by a large team of researchers led by Run Luo, Kuan Zhang, and Qi Deng. Their work, first shared in November 2026, represents a significant leap toward creating a general-purpose AI that can operate across many different digital environments.

Most previous game-playing AIs were built for one specific game, using customized controls or access to the game’s internal code. Gamer is different. It’s a generalist agent—an AI built to handle long and complex tasks in a variety of interactive settings. Its goal is to interact with the digital world the only way a human can: by looking and clicking.

Figure 4 : Sparse reasoning and cross-context memory in supervised fine-tuning. The shared autoregressive policy integrates visual observations, task memory, optional reasoning, and executable keyboard-mouse actions. Context memory compaction preserves task progress across long interactions, while a
Figure 4 : Sparse reasoning and cross-context memory in supervised fine-tuning. The shared autoregressive policy integrates visual observations, task memory, optional reasoning, and executable keyboard-mouse actions. Context memory compaction preserves task progress across long interactions, while a
Figure from the original paper by Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren (CC BY-NC-SA 4.0), via arXiv

How Does the Gamer AI Actually Learn?

The training process for Gamer is a massive undertaking that happens in several stages, much like teaching a person a complex skill.

First, the AI undergoes continued pre-training (CPT). The researchers fed it a colossal dataset made from "tens of thousands of hours of device-level keyboard and mouse interaction" recorded from people playing games. This is the equivalent of a student spending years watching expert players, absorbing the basic mechanics of how to control a game. <figure class="my-10 overflow-hidden border border-slate-800 rounded-2xl bg-white text-slate-100 shadow-xl">

Figure 2: CPT for unified keyboard-mouse control. Synchronized gameplay observations and device events are mapped to a shared action representation. Random key remapping reduces reliance on key identity, while repeat-aware loss decay reweights temporally persistent actions. The resulting control pr
Figure 2: CPT for unified keyboard-mouse control. Synchronized gameplay observations and device events are mapped to a shared action representation. Random key remapping reduces reliance on key identity, while repeat-aware loss decay reweights temporally persistent actions. The resulting control pr
Figure from the original paper by Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren (CC BY-NC-SA 4.0), via arXiv

</figure>

The AI plays the games itself, and its actions are rewarded or penalized based on their success. The researchers developed a special method called Temporal-Aware GRPO to handle the unique challenges of video games, such as rewards that only come after a long sequence of actions (like finally defeating a boss after a 30-minute battle).

The Need for Speed: Playing Games in Real Time

One of the biggest technical hurdles wasn't just making an AI that can play games, but making one that can play them in real time. If an AI takes too long to think, the game state changes, and its action becomes useless or even harmful. This was a critical bottleneck for both deployment and further training.

To solve this, the team built a powerful inference engine called Streaming-vLLM. The researchers tested Gamer extensively.

Its most impressive feats were in "multiple challenging 3D open-world games," where it demonstrated "solid generalization and operational efficiency comparable to skilled players." While the source text doesn't name all the specific games, it highlights Honkai: Star Rail (HSR) as a key testing ground. HSR is a complex game that combines exploration, turn-based combat, and menu navigation—requiring the AI to master a wide array of skills within a single world. <figure class="my-10 overflow-hidden border border-slate-800 rounded-2xl bg-white text-slate-100 shadow-xl">

Figure 1: Overview of Gamer. A generalist game agent framework operates across a broad range of visually and mechanically diverse games. At each step, the agent uses the current screen, task context, recent interaction history, long-term memory, and reasoning to generate native keyboard and mouse a
Figure 1: Overview of Gamer. A generalist game agent framework operates across a broad range of visually and mechanically diverse games. At each step, the agent uses the current screen, task context, recent interaction history, long-term memory, and reasoning to generate native keyboard and mouse a
Figure from the original paper by Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren (CC BY-NC-SA 4.0), via arXiv

</figure>

Gamer also "achieves competitive results on a range of conventional 2D game benchmarks," proving its flexibility across different genres and eras of gaming. This cross-game performance is the strongest evidence that the AI is learning a general understanding of interaction, not just memorizing one game.

"These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms," Run Luo and colleagues write in the paper.

Why This Matters Beyond the Game Screen

This research is about far more than creating the ultimate automated gamer. Video games are a perfect, controlled testing ground for the much bigger challenge: building AI that can operate in the real world.

The skills required to play a modern video game—understanding a visual scene, making a plan, and executing precise physical actions—are the same foundational skills needed for a robot to navigate a kitchen or for a digital assistant to help you use complex software. The paper states that this approach "offers a path toward generalist agents that can interact across digital and physical worlds."

This work connects to other advances in AI that learns from watching and doing. For instance, research like SLA: AI Learns to Grab and Move Objects Like Humans and DreamX: AI Learns to Act Like People By Watching Videos is tackling similar problems of physical interaction. Gamer applies this concept to the rich, complex simulations of digital worlds.

Figure 5 : Memory-linked long-horizon optimization with TA-GRPO . A long interaction may exceed the model’s context budget (top). Memory compaction divides online rollouts into finite segments whose task states remain connected by generated summaries (bottom). Segments at corresponding stages are co
Figure 5 : Memory-linked long-horizon optimization with TA-GRPO . A long interaction may exceed the model’s context budget (top). Memory compaction divides online rollouts into finite segments whose task states remain connected by generated summaries (bottom). Segments at corresponding stages are co
Figure from the original paper by Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren (CC BY-NC-SA 4.0), via arXiv

The Bigger Picture

For years, a major goal in artificial intelligence has been to create a generalist agent—a single AI that can adapt and perform well on a wide variety of tasks, much like a human can. This is in stark contrast to most of today's AI, which are specialists excel in one narrow area but fail miserably at anything else.

The field of embodied AI focuses on agents that can interact with their environment, whether digital or physical. Gamer sits squarely in this domain. It tackles the monumental challenge of long-horizon tasks—goals that require many steps to complete, with actions that have consequences much later. Making a cup of coffee or completing a video game quest are both long-horizon tasks that require planning, memory, and adaptability.

Gamer represents significant progress by showing that a single model can learn a unified way to interact (via keyboard/mouse) and apply it across many domains. It also provides a blueprint for how to train such agents at a massive scale. However, the authors are clear about the limitations. The real physical world is far more complex and unpredictable than even the most detailed game. Transferring these skills from the digital to the physical realm remains a formidable challenge for future work.

Figure 6 : Streaming-vLLM for persistent multi-environment inference. Each game client maintains separate short-term history, long-term memory, and logical KV-cache state. Paged allocation and memory-aware KV-cache reuse avoid repeated context processing, while incremental rotary position embedding
Figure 6 : Streaming-vLLM for persistent multi-environment inference. Each game client maintains separate short-term history, long-term memory, and logical KV-cache state. Paged allocation and memory-aware KV-cache reuse avoid repeated context processing, while incremental rotary position embedding
Figure from the original paper by Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Xiao Liu, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing Ren (CC BY-NC-SA 4.0), via arXiv

Paper, Code and Resources

Frequently Asked Questions

What is a generalist AI agent?

A generalist AI agent is an artificial intelligence system designed to perform well across a wide range of different tasks, rather than being expert at just one. Gamer is a generalist agent for video games because it can learn to play many different types of games, from 2D classics to complex 3D open-world adventures.

Does the Gamer AI cheat to play the games?

No. According to the research, Gamer does not use any special access to the game’s internal code or state. It interacts with games "non-invasively," using only the screen pixels as input and generating standard keyboard and mouse commands as output—exactly how a human player does.

Could this AI be used for game testing?

Absolutely. The capabilities of Gamer could be very valuable for game developers. It could be used to automatically test games for bugs, balance difficulty by playing through levels countless times, or even provide a consistent benchmark for measuring the performance of other AI systems.

What's the next step for this research?

The immediate next steps involve scaling the technology further with more data and more complex games. The long-term goal, as stated by the researchers, is to use the lessons learned from digital worlds to build agents that can operate in the physical world. This means overcoming huge challenges like dealing with unpredictable real-world physics and the vast complexity of human environments.

Source: Read the original study