Tencent Patents To Automate Live Game Stream Commentary Using AI

Live streaming has swiftly and thoroughly evolved over the last few decades, reaching new horizons thanks to the demand and availability of services like Twitch. Video games live streaming encapsulates most of the community, with users watching other streamers play their favorite games while delivering amusing commentary.

It seems live game streaming is about to get easier than ever before with a new patent published by Tencent. We came across the patent dubbed “COMMENTARY VIDEO GENERATION METHOD AND APPARATUS, SERVER, AND STORAGE MEDIUM,” which discusses automating commentary in live streams using the wonders of AI.

The patent notes that a game streamer needs to engage with the community by commenting on everything occurring while playing games, which can prove to be a monotonous task. It further states that manual commentary in live streaming requires a “long production process and high manual operation costs.”

Tencent‘s proposed system will utilize AI to generate commentary for streamers as they play the game automatically. It will employ “a computer vision technology, a speech processing technology, and a natural language processing” AI technology to seamlessly generate the commentary during live game streams.

The patent contains highly technical terms and jargon, so it is difficult to decipher in a reader-friendly manner. In short, it works “through online analysis on the game instruction frame, the commentary audio is generated, the game video is rendered, the commentary audio and the game video are aligned in time to generate the commentary video.”

While live streaming a game, any in-game activity, “game operation instruction,” performed in the game will be recorded and sent to the “commentary server,” which “can render the game in the commentary server 120 based on the received game operation instruction.” 

The patent elaborates on the commentary server module in detail; it is a back-end server behind any game being live-streamed online. The commentary server is designed to create online game commentary automatically using AI technology and push the live stream to the client devices viewing it online.

The further patent notes, “The commentary server automatically generates a corresponding commentary video stream during the game, and pushes the commentary video stream to a livestreaming terminal for playing, to improve the generation timeliness of the commentary video.”

It continues, “To generate the commentary video during the game in real time, in a possible implementation, online game video rendering and online analysis and commentary can be implemented through analysis on the game instruction frame.”

As stated above, Tencent’s system can generate commentary in real-time during the live game stream, which comes with its slew of benefits. “There is no need to wait for the game to be over to obtain and process the game video to generate the commentary video, thereby further improving the real-time performance and timeliness of the commentary video generation.”

It can consider various attributes of virtual objects in games before processing to create automated commentary. Some attributes include “location information, health point information, speed information, level information, skill information, feat information, equipment information, score information, etc.”

Meanwhile, the objects can refer to any virtual object that the user can control, for instance, a building, playable and non-playable characters, and the sort for automatic commentary generation during live game streams.

It is beneficial because it can automate many procedures behind the scenes during live streaming by utilizing artificial intelligence technologies. It will reduce the usage of computing resources and electricity, as per Tencent’s patent. The commentary will be created on the spot as the user plays the game on stream.

Additionally, the player could focus more on the game and leave the commentary to AI. The patent, if adopted, could transform the streaming service industry as we know it.

Video games of all genres, including “a sports game, a military simulation program, a multiplayer online battle arena (MOBA) game, a battle royale shooting game, a simulation game (SLG), etc.” can possibly use Tencent’s proposed method.

The patent sticks to using “a virtual online commentary scenario” as an example, meaning any multiplayer experience, like MOBA. Tencent explores a team fight situation in a MOBA game to elaborate on the working of the patent. The AI will use different variables to prioritize what to talk about during the game in live streaming. 

For instance, the number of virtual objects can determine what game event is selected to be commentated on by Artificial intelligence. “For example, if the game includes two team fights, where the team fight A has 3 virtual objects and the team fight B has 7 virtual objects, priority is given to the game event corresponding to the team fight B in selecting the game event.”

Tencent has been at the forefront of patenting innovative schemes alongside Sony and Microsoft to get ahead of its western competition. For instance, it only recently patented an ingenious Virtual Reality headset that seeks to solve some of the most prominent issues in the current VR devices and greatly increases immersion.

Tencent is highly popular for keeping several accredited games like PUBG, League of Legends, Ring of Elysium, Path of Exile, and more under its belt. The giant Chinese conglomerate is currently developing anticipated projects like the new AAA entry, Code: To Jin Yong.

What are your thoughts regarding Tencent seemingly tinkering with AI to automate gaming live streams in the future? Do you think automated live streams will aid streamers in focusing more on the gameplay only? Do let us know your views in the comments below.