Sony May Soon Add Controller Overlay Support For Recorded Gameplay

Sony Entertainment Interactive is among the biggest companies worldwide, and it is quite fast in patenting all the clever ideas. It has secured several alluring prospects such as bringing real-life locations into video games using AI and making quick-time events more user-friendly. However, the list only continues to grow from here on out.

Sony’s quest for enhancing the user experience continues further. We stumbled upon a newly published patent that is seeking to help newbies by officially integrating a controller overlay in recorded gameplay using machine learning models. The “spectator computer” cited in the patent means the viewer watching the recorded gameplay.

Major Takeaway

• Sony has published a new patent that seeks to incorporate a controller overlay for every recorded gameplay using machine learning models.
• The patent only appears to discuss controllers and leaves a mention of a keyboard out of the equation.
• The method will first receive the recorded gameplay and use AI to understand what button was pressed at a specific frame before showing the recorded gameplay with the controller overlay to the spectator’s computer.
• It will assist a viewer in how to play the game, letting newcomers see how a speedrunner or a pro completed a specific level.
• We may see the official controller overlay integration on PlayStation and Sony PC ports in the coming years.

The patent dubbed “CONTROLLER ACTION RECOGNITION FROM VIDEO FRAMES USING MACHINE LEARNING” will let users see which button is pressed at a specific frame. It is especially useful for new players because it will let them see how the controller is utilized by the recorder of the gameplay at every moment.

Sony describes the patent in the following words, “A machine learning model is used to receive a recorded video game for presentation on a spectator computer and to derive from the video identifications of controller operations during play of the game that resulted in the recorded video game.”

In easier words, the method will receive the recorded gameplay and apply machine learning models to implement a controller overlay. Then the dynamic controller overlay will be added to the footage, letting the spectators see every button pressed on the controller at a specific frame during the gameplay.

The patent continues, “Indications of identified controller operations may be presented with the recorded video game to assist a viewer in learning how to play the game.” It will improve the learning process by a mile since the users will be able to keep track of every controller action taking place alongside the gameplay.

Until now, we have usually seen walk-through videos of different games helping players learn how to complete a specific level. However, it is about to become easier, than ever before due to Sony’s latest concept. 

The system will help players wanting to learn how a level was completed by a professional or a speedrunner. Speedrunners will be able to better understand what button to press at each moment, making it a tad bit easier. That factor alone may cause the proposed system to be a popular demand for specific games in the future.

Sony elaborates, “such information about what controller buttons were pressed and when can be valuable to a player learning to play the computer game, making gaming more enjoyable for many types of gamers, from beginners to speed-runners.”

The technicalities of the patent discuss several machine learning training methods. However, all of them mention analyzing the recorded footage and applying the controller overlay frame by frame before sending it to the spectator computer. It will be automatically handled using AI and may prove to be quite reliable.

Multiple client platforms are noted in the patent, but they appear to be spectator computers that can view the recorded gameplay. On the other hand, server devices precisely mention PCs and PlayStation consoles. It makes sense for Sony to secure the discussed method for itself and the studios under it.

The patent states, “A server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, etc.” We can deduce that the presented system could at least be implemented in Sony games available on PlayStation consoles and PCs down the road. 

It is worth noting that controller overlays have been done manually in the past. For instance, it is reminiscent of YouTubers and Streamers usually adding a hand cam that lets us see how they use a controller while they play the game. Having an official integration using AI ought to make overlays more dynamic and beneficial to newbies.

A plethora of games developed by PlayStation Studios could utilize the alluring method. Platforming games or titles that require precise controller control would benefit the most through an official overlay integration.

However, we suggest taking it with a grain of salt since this is just a patent, and we may never see it implemented.

What are your thoughts about the possibility of Sony games featuring a controller overlay for recorded gameplay soon? Would you prefer such a technology to be incorporated? Do let us know your opinions in the comments below.

Similar Reads: Customizable Killcams Might Be Heading To Call Of Duty: Modern Warfare 2