While still in an early development stage, the results are compelling enough to suggest there could be a real future in this kind of system for home sports viewing -- and it could come sooner rather than later.
Called, simply, "Soccer on your Tabletop," the practicality of the system lies in the fact that it doesn't rely on multi-angle, 3D cameras on the field. Instead, the research team figured out how to map 3D characters onto 2D broadcasts of games.
That's no easy feat, but the research team is now demonstrating a working prototype just in time for the World Cup.
"There are numerous challenges in monocular reconstruction of a soccer game," according to a paper presented at last week's CVPR, an annual computer vision conference. "We must estimate the camera pose relative to the field, detect and track each of the players, reconstruct their body shapes and poses, and render the combined reconstruction."
"Our approach is trained on meshes extracted from FIFA video games. Based on this data, a neural network estimates per pixel depth values of any new soccer player, comparing favorably to other state-of-the-art body depth and pose estimation techniques."
There are still some kinks. Players tend to skip around, and the ball is largely invisible in the demo videos. The paper notes that the researchers plan to get to that in future development.
But the core concept and demonstrable results are impressive. Unlike most newfangled 3D entertainment, this is something I can actually envision using myself.
(I'm already cringing at the thought of watching my soccer-obsessed wife try to flick an under-performing Messi off our coffee table.)
Any AR 3D viewer, such as MicroSoft's HoloLens, can be used to view games from multiple angles, according to the researchers. Right now, 3D games are rendered from 4K YouTube videos.
For the technology to really become practical, rendering times will have to be increased to the point that 3D games based off 2D broadcasts can be viewed in near-real-time.