The proposed method for estimation of face direction overcomes the challenges encountered by traditional two-dimensional image-based methods
Conventional techniques for estimating face orientations rely upon the movements of eyes, nose, and mouth. This limits accurate detection if the person is wearing a mask, or is facing sideways. To address this problem, researchers recently collected point cloud data from various face orientations by a depth–gyro sensor and used them to train a deep learning-based classification model. The proposed method demonstrates high accuracy for estimating face orientation, even with a small dataset.
Image Title: Face orientation estimation using depth–gyro sensor and deep learning model
Image Caption: A depth–gyro sensor collects 3D point cloud data of a person’s face along with face orientation information. It is used to train a deep learning classification model, which then accurately estimates the face orientation.
Image Credit: Chinthaka Premachandra from SIT, Japan
Image Source Link: https://ieeexplore.ieee.org/document/10190334
License Type: CC BY-NC-ND 4.0
Usage restrictions: Under a Creative Commons license CC BY-NC-ND 4.0. You are free to share the material. Attribution is required. You may not use the material for commercial purposes. You cannot distribute any derivative works or adaptations of the original work.
In recent years, artificial intelligence has demonstrated tremendous potential for the development and advancement of a wide variety of technologies. A good example is facial direction estimation, which finds applications in driver assistance systems that prevent distracted driving, methods to prevent cheating in examinations, and software for creating three-dimensional (3D) virtual avatars.
Traditional facial orientation estimation techniques recognize the characteristic parts of the face, including the nose, eyes, and mouth, and detect their movements. However, such two-dimensional (2D) image-based methods raise privacy concerns and fail when features of the face are hidden due to a mask, or if the face is turned sideways. The solution may lie in optimizing facial detection using point cloud data (data obtained from a discrete set of data points) and a depth sensor. In fact, some previous studies have employed an estimation model based on the deep learning of 3D point cloud data in five face directions: frontal, diagonal frontal, right, left, and horizontal. However, considering the level of accuracy required for driver assistance systems that crucially verify the driver’s status, this five-class (k = 5) classification is insufficient for satisfactorily detecting the face direction.
To address this limitation, scientists from Shibaura Institute of Technology, led by Professor Chinthaka Premachandra of the Graduate School of Engineering and Science, have developed a more precise, horizontal wide-range angle detection approach (with k > 5). They accurately measured the horizontal angle of the face during the acquisition of the training data using gyroscopic sensors. Their paper was made available online in the IEEE Sensors Journal on 21 July, 2023.
In this study, the scientists gathered point cloud data from various orientations using a depth sensor, which was integrated with a gyro sensor during data collection. This data was employed to train a deep learning-based classification model, which was utilized for face orientation estimation. The scientists changed the horizontal angle of the face relative to the camera from +90 degrees to -90 degrees, using step sizes of 30, 22.5, 18, and 15 degrees between them. As a result, the classification of face direction was represented by more than seven classes (k = 7, 9, 11, 13).
“Precise training data for each orientation was obtained from the integration of the depth and gyro sensors, which reduce the number of point cloud samples required for constructing the classification model. Furthermore, applying a weight reduction process to reduce the weight of point cloud data enhanced training efficiency and resulted in fast face orientation estimation,” explains Prof. Premachandra.
The proposed classification method, designed for more than seven classes, achieves remarkable performance in face direction detection through deep learning. For example,
Overall, this study opens new doors to a wide range of practical applications where accurate face orientation detection is required. As Prof. Premachandra explains,
Let us hope that this novel face direction detection technology enables accurate face orientation detection soon!
This study was partially supported by the Branding Research Fund of the Shibaura Institute of Technology.