{"id":6585,"date":"2015-11-05T08:19:48","date_gmt":"2015-11-05T08:19:48","guid":{"rendered":"http:\/\/revoscience.com\/en\/?p=6585"},"modified":"2015-11-05T08:19:48","modified_gmt":"2015-11-05T08:19:48","slug":"system-automatically-converts-2-d-video-to-3-d","status":"publish","type":"post","link":"https:\/\/www.revoscience.com\/en\/system-automatically-converts-2-d-video-to-3-d\/","title":{"rendered":"System automatically converts 2-D video to 3-D"},"content":{"rendered":"

Exploiting video game software yields broadcast-quality 3-D video of soccer games in real time.<\/strong><\/em><\/span><\/p>\n

\"Illustration:<\/a>
Illustration: Christine Daniloff\/MIT<\/figcaption><\/figure>\n

CAMBRIDGE, Mass.<\/strong> —\u00a0By exploiting the graphics-rendering software that powers sports video games, researchers at MIT and the Qatar Computing Research Institute (QCRI) have developed a system that automatically converts 2-D video of soccer games into 3-D.<\/span><\/p>\n

The converted video can be played back over any 3-D device \u2014 a commercial 3-D TV, Google\u2019s new Cardboard system, which turns smartphones into 3-D displays, or special-purpose displays such as Oculus Rift.<\/span><\/p>\n

[pullquote]The researchers conducted a user study in which the majority of subjects gave the 3-D effect a rating of 5 (\u201cexcellent\u201d) on a five-point (\u201cbad\u201d to \u201cexcellent\u201d) scale; the average score was between 4 (\u201cgood\u201d) and 5.[\/pullquote]<\/p>\n

The researchers presented the new system last week at the Association for Computing Machinery\u2019s Multimedia conference.<\/span><\/p>\n

\u201cAny TV these days is capable of 3-D,\u201d says Wojciech Matusik, an associate professor of electrical engineering and computer science at MIT and one of the system\u2019s co-developers. \u201cThere\u2019s just no content. So we see that the production of high-quality content is the main thing that should happen. But sports is very hard. With movies, you have artists who paint the depth map. Here, there is no luxury of hiring 100 artists to do the conversion. This has to happen in real-time.\u201d<\/span><\/p>\n

The system is one result of a collaboration between QCRI and MIT\u2019s Computer Science and Artificial Intelligence Laboratory. Joining Matusik on the conference paper are Kiana Calagari, a research associate at QCRI and first author; Alexandre Kaspar, an MIT graduate student in electrical engineering and computer science; Piotr Didyk, who was a postdoc in Matusik\u2019s group and is now a researcher at the Max Planck Institute for Informatics; Mohamed Hefeeda, a principal scientist at QCRI; and Mohamed Elgharib, a QCRI postdoc. QCRI also helped fund the project.<\/span><\/p>\n

Zeroing in<\/strong><\/span><\/p>\n

In the past, researchers have tried to develop general-purpose systems for converting 2-D video to 3-D, but they haven\u2019t worked very well and have tended to produce odd visual artifacts that detract from the viewing experience.<\/span><\/p>\n

\u201cOur advantage is that we can develop it for a very specific problem domain,\u201d Matusik says. \u201cWe are developing a conversion pipeline for a specific sport. We would like to do it at broadcast quality, and we would like to do it in real-time. What we have noticed is that we can leverage video games.<\/span><\/p>\n

Today\u2019s video games generally store very detailed 3-D maps of the virtual environment that the player is navigating. When the player initiates a move, the game adjusts the map accordingly and, on the fly, generates a 2-D projection of the 3-D scene that corresponds to a particular viewing angle.<\/span><\/p>\n

The MIT and QCRI researchers essentially ran this process in reverse. They set the very realistic Microsoft soccer game \u201cFIFA13\u201d to play over and over again, and used Microsoft\u2019s video-game analysis tool PIX to continuously store screen shots of the action. For each screen shot, they also extracted the corresponding 3-D map.<\/span><\/p>\n

Using a standard algorithm for gauging the difference between two images, they winnowed out most of the screen shots, keeping just those that best captured the range of possible viewing angles and player configurations that the game presented; the total number of screen shots still ran to the tens of thousands. Then they stored each screen shot and the associated 3-D map in a database.<\/span><\/p>\n

Jigsaw puzzle<\/strong><\/span><\/p>\n

For every frame of 2-D video of an actual soccer game, the system looks for the 10 or so screen shots in the database that best correspond to it. Then it decomposes all those images, looking for the best matches between smaller regions of the video feed and smaller regions of the screen shots. Once it\u2019s found those matches, it superimposes the depth information from the screen shots on the corresponding sections of the video feed. Finally, it stitches the pieces back together.<\/span><\/p>\n

The result is a very convincing 3-D effect, with no visual artifacts. The researchers conducted a user study in which the majority of subjects gave the 3-D effect a rating of 5 (\u201cexcellent\u201d) on a five-point (\u201cbad\u201d to \u201cexcellent\u201d) scale; the average score was between 4 (\u201cgood\u201d) and 5.<\/span><\/p>\n

Currently, the researchers say, the system takes about a third of a second to process a frame of video. But successive frames could all be processed in parallel, so that the third-of-a-second delay needs to be incurred only once. A broadcast delay of a second or two would probably provide an adequate buffer to permit conversion on the fly. Even so, the researchers are working to bring the conversion time down still further.<\/span><\/p>\n