Game Footage Analysis by AI
Live game streaming and play analytics is growing towards a trillion-dollar business. Yet much of the work is done manually and relies on huamn expertise to extract insights from footage of top players. Sensor AI partnered with game analytics companies to develop a solution to help reduce manual effort and cost of processing gameplay video, while improving the quality of insights uncovered from countless hours of online video game footage.
Business Overview
Modern game consoles have made it easy to share game footage through SNS.
Hobbyists and professional gamers spend countless hours making highlight play scenes on Twitch and Youtube for living.
Problem is, not only is the process painful and expensive, but many important details can be missed during manual editing.
Sensor AI partnered with companies specializing in live game streaming to produce AI-powered game analytics solutions.
We use custom-trained computer vision models to analyze play clips, labeling each frame with key events, actions, units, and/or items that are significant to the playing context.
We use multilingual speech-to-text to transcribe player audio to analyze what was said and use it to discover moments of interest (such as kills, nice play tricks, rare drop of items etc.).
Acoustic analysis also helped finding glitches, making early detection of bugs possible from thousands of hours of raw gameplay footages online.
Finally, a trained NLP language model analyzes the content and context of player conversations, distilling key topics mentioned to help editors understand the nature of the play, and know which key frames to look at for important events.
Below we will use Fortnite as an example to explain our solution:
Computer Vision Scene Analysis
Use custom trained model to tag frames for important events, actions, units, objects and/or performance automatically.
Produce Highlight Moments
Detects key events and extract adjacent frames to form a contiguous short clip, automatically combine important scenes made up of highlight moments.
Speech Analysis
Finding useful information from spoken recordings can lead to powerful discoveries of key insights. We use speech-to-text, NLP, and acoustic analysis to distill key insights from player conversations.
Architectures
Built on Alibaba Cloud with GPU backed parallel compute, Kuberenets container services and custom AI modules working as a whole.
