A SfM-based 3D face reconstruction method robust to self-occlusion by using a shape conversion matrix

Sung Joo Lee, Kang Ryoung Park, Jaihie Kim

Research output: Contribution to journalArticlepeer-review

32 Scopus citations

Abstract

This paper presents a 3D face reconstruction method using multiple 2D face images. Structure from motion (SfM) methods, which have been widely used to reconstruct 3D faces, are vulnerable to point correspondence errors caused by self-occlusion. In order to solve this problem, we propose a shape conversion matrix (SCM) which estimates the ground-truth 2D facial feature points (FFPs) from the observed 2D FFPs corrupted by self-occlusion errors. To make the SCM, the training observed 2D FFPs and ground-truth 2D FFPs are collected by using 3D face scans. An observed shape model and a ground-truth shape model are then built to represent the observed 2D FFPs and the ground-truth 2D FFPs, respectively. Finally, the observed shape model parameter is converted to the ground truth shape model parameter via the SCM. By using the SCM, the true locations of the self-occluded FFPs are estimated exactly with simple matrix multiplications. As a result, SfM-based 3D face reconstruction methods combined with the proposed SCM become more robust against point correspondence errors caused by self-occlusion, and the computational cost is significantly reduced. In experiments, the reconstructed 3D facial shape is quantitatively compared with the 3D facial shape obtained from a 3D scanner, and the results show that SfM-based 3D face reconstruction methods with the proposed SCM show a higher accuracy and a faster processing time than SfM-based 3D face reconstruction methods without the SCM.

Original languageEnglish
Pages (from-to)1470-1486
Number of pages17
JournalPattern Recognition
Volume44
Issue number7
DOIs
StatePublished - Jul 2011

Keywords

  • 3D face reconstruction
  • Self-occlusion
  • Shape conversion matrix
  • Structure from motion (SfM)

Fingerprint

Dive into the research topics of 'A SfM-based 3D face reconstruction method robust to self-occlusion by using a shape conversion matrix'. Together they form a unique fingerprint.

Cite this