Scientists from Skoltech’s ADASE (Advanced Data Analytics in Science and Engineering) laboratory have developed a way to increase the resolution of depth maps, which in turn will help make virtual reality and computer graphics more realistic.
Work results presented at the prestigious International Conference on Computer Vision 2019 in Korea.
When taking a photo, we capture visual information about the surrounding objects: the pixels of the photo contain the colors of the corresponding parts. Depth maps are photos that capture spatial information — their pixels contain the distance from the camera to the corresponding points in space.
In applications such as computer graphics, augmented or virtual reality, this spatial information is used to restore the shape of the three-dimensional surface of objects and their subsequent display, for example, on a computer screen.
One of the problems with sensors capturing depth cameras is that the resolutions of these captured maps (i.e. the spatial frequency of distance measurements) are insufficient for high-quality restoration of the shape of objects. As a result, their virtual reconstructions look unrealistic. Researchers are faced with the task of learning how to obtain the same high-resolution maps from low-resolution depth maps.
To this end, scientists from the Skoltech ADASE laboratory have proposed a new method for assessing the quality of the results, close to human perception. Learning an artificial neural network using a quality measurement method allows you to get a super-resolution method of depth maps, the visual quality of the results of which significantly exceeds the visual quality of the results obtained using existing methods.
“With the super-resolution of depth maps, it is necessary to evaluate the quality of the result. This assessment is used, firstly, to compare different methods among themselves, and secondly, during the development of the method, as feedback for its improvement. The simplest way to obtain such an assessment is to measure the similarity of the result to some standard.
In the vast majority of works devoted to the problem of super-resolution of depth maps, a quantitative measure of quality is the average deviation of the distance in the super-resolved depth map from its reference value. This method absolutely does not reflect the visual quality of the three-dimensional reconstruction obtained from the super-resolved depth map, ”says Oleg Voinov, the first author of the study.
“We offer a fundamentally different way of assessment, based on the human perception of the difference between visualizations of three-dimensional reconstructions built on super-resolved and reference depth maps. When you use it, the graphics are really realistic. We hope that our method will be widely used, ”says one of the developers, Alexei Artemov.
Skolkovo Institute of Science and Technology is a non-state technological university located in the Skolkovo innovation center. The institute was established in 2011 with the support of the Massachusetts Institute of Technology. The institute model provides for the close integration of technological education, research and entrepreneurial skills. The institute conducts training in master’s and PhD programs, the working language is English.