Sony May Soon Add Controller Overlay Support For Recorded Gameplay

The overlay will show what buttons were pressed at a specific frame using AI.

Sony Entertainment Interactive is among the biggest companies worldwide, and it is quite fast in patenting all the clever ideas. It has secured several alluring prospects such as bringing real-life locations into video games using AI and making quick-time events more user-friendly. However, the list only continues to grow from here on out.

Sony’s quest for enhancing the user experience continues further. We stumbled upon a newly published patent that is seeking to help newbies by officially integrating a controller overlay in recorded gameplay using machine learning models. The “spectator computer” cited in the patent means the viewer watching the recorded gameplay.

Major Takeaway

  • Sony has published a new patent that seeks to incorporate a controller overlay for every recorded gameplay using machine learning models.
  • The patent only appears to discuss controllers and leaves a mention of a keyboard out of the equation.
  • The method will first receive the recorded gameplay and use AI to understand what button was pressed at a specific frame before showing the recorded gameplay with the controller overlay to the spectator’s computer.
  • It will assist a viewer in how to play the game, letting newcomers see how a speedrunner or a pro completed a specific level.
  • We may see the official controller overlay integration on PlayStation and Sony PC ports in the coming years.

The patent dubbed “CONTROLLER ACTION RECOGNITION FROM VIDEO FRAMES USING MACHINE LEARNING” will let users see which button is pressed at a specific frame. It is especially useful for new players because it will let them see how the controller is utilized by the recorder of the gameplay at every moment.

An illustration showing the machine learning applied on recorded footage and shown to the client device.

Sony describes the patent in the following words, “A machine learning model is used to receive a recorded video game for presentation on a spectator computer and to derive from the video identifications of controller operations during play of the game that resulted in the recorded video game.

In easier words, the method will receive the recorded gameplay and apply machine learning models to implement a controller overlay. Then the dynamic controller overlay will be added to the footage, letting the spectators see every button pressed on the controller at a specific frame during the gameplay.

A flow chart diagram showing the step by step procedure of how the system will produce a controller overlay for recorded gameplay.

The patent continues, “Indications of identified controller operations may be presented with the recorded video game to assist a viewer in learning how to play the game.” It will improve the learning process by a mile since the users will be able to keep track of every controller action taking place alongside the gameplay.

Until now, we have usually seen walk-through videos of different games helping players learn how to complete a specific level. However, it is about to become easier, than ever before due to Sony’s latest concept. 

The system will help players wanting to learn how a level was completed by a professional or a speedrunner. Speedrunners will be able to better understand what button to press at each moment, making it a tad bit easier. That factor alone may cause the proposed system to be a popular demand for specific games in the future.

Sony elaborates, “such information about what controller buttons were pressed and when can be valuable to a player learning to play the computer game, making gaming more enjoyable for many types of gamers, from beginners to speed-runners.

An example game controller with an example keys operation that may be learned by machine learning models from a game.

The technicalities of the patent discuss several machine learning training methods. However, all of them mention analyzing the recorded footage and applying the controller overlay frame by frame before sending it to the spectator computer. It will be automatically handled using AI and may prove to be quite reliable.

Illustration showing a sequence of video frames about controller operations and alternate video paths that can occur depending on controller operation.

Multiple client platforms are noted in the patent, but they appear to be spectator computers that can view the recorded gameplay. On the other hand, server devices precisely mention PCs and PlayStation consoles. It makes sense for Sony to secure the discussed method for itself and the studios under it.

The patent states, “A server or controller may be instantiated by a game console such as a Sony PlayStation®, a personal computer, etc.” We can deduce that the presented system could at least be implemented in Sony games available on PlayStation consoles and PCs down the road. 

It is worth noting that controller overlays have been done manually in the past. For instance, it is reminiscent of YouTubers and Streamers usually adding a hand cam that lets us see how they use a controller while they play the game. Having an official integration using AI ought to make overlays more dynamic and beneficial to newbies.

A plethora of games developed by PlayStation Studios could utilize the alluring method. Platforming games or titles that require precise controller control would benefit the most through an official overlay integration.

However, we suggest taking it with a grain of salt since this is just a patent, and we may never see it implemented.

What are your thoughts about the possibility of Sony games featuring a controller overlay for recorded gameplay soon? Would you prefer such a technology to be incorporated? Do let us know your opinions in the comments below.

Similar Reads: Customizable Killcams Might Be Heading To Call Of Duty: Modern Warfare 2

Did you find this helpful? Leave feedback below.

Thanks! Do share your feedback with us. ⚡

How can we make this post better? Your help would be appreciated. ✍

Subscribe to our newsletter and get up-to-speed gaming updates delivered to your inbox.

We don’t spam! Read more in our privacy policy.

Subscribe to our newsletter and get up-to-speed gaming updates delivered to your inbox.

We don’t spam! Read more in our privacy policy.


Shameer Sarfaraz is a Senior News Writer on eXputer who loves to keep up with the gaming and entertainment industries devoutly. He has a Bachelor's Degree in Computer Science and several years of experience reporting on games. Besides his passion for breaking news stories, Shahmeer loves spending his leisure time farming away in Stardew Valley. VGC, IGN, GameSpot, Game Rant, TheGamer, GamingBolt, The Verge, NME, Metro, Dot Esports, GameByte, Kotaku Australia, PC Gamer, and more have cited his articles.

Experience: 4+ Years || Education: Bachelor in Computer Science.

Related Articles

Join Our Community

Enjoyed this article? Discuss the latest gaming news, get expert help with guides and errors, and chat about all things gaming on eXputer Forums and Discord Server. Connect with fellow gamers who share your passion by becoming a part of eXputer's community.