3D Shape Estimation of Multiview RGB Images from Deep Convolutional Network

In this paper, we describe point cloud generation method from multiple RGB images based on convolutional neural network. The proposed method is motivated from human’s 3d understanding from orthographic sketches, which infer spatial relationship from front, top, side view of the 3d object. The network of the proposed method employed generative model, to make it predict point clouds of atypical objects. Auto encoder network is utilized to encode three RGB images into latent vectors, and generate point clouds. Loss functions are defined which measure reconstruction performance and uniformity of the point clouds to make the network generate point clouds similar to the original and distribute uniformly along entire region. From this structure, we expected the network to learn spatial relation of the original 3d model from multiple RGB images. The result shows that the proposed method can predict overall shape of the objects, and it is hard to express detailed geometry. As a further work, network structure improvement to generate detailed shape of the object, predict occluded region will be performed.