Abstract
One of the most common forms of three-dimensional data is point clouds. In addition to its high flexibility in storing three-dimensional space, this type of data is the closest type of data to the output of three-dimensional sensors. Semantic segmentation of point clouds is a fundamental operation on this type of data, with applications in robotics, self-driving cars, virtual reality, remote sensing, and other fields that work with this type of data. Since deep learning models require abundant data for training, this type of data is not an exception to this rule with these models.
However, the problem is that collecting and labeling this type of data is more difficult and costly compared to other types of data. Therefore, in extensive research in this area, attempts have been made to improve this problem using methods such as semi-supervised learning, self-supervised learning, few-shot learning, and other types of limited-supervise learning approach. In these methods, less attention has been paid to the fact that there are abundant capacities in the field of two-dimensional images, and we can use these capacities to improve the performance of point cloud models.
Since each image is a kind of mapping of three-dimensional points onto a two-dimensional plane, and it is also common to have several cameras recording two-dimensional images of space alongside point cloud sensors, it is not unexpected to assume that we usually have two-dimensional images of these point clouds. Therefore, in this study, a framework has been proposed to implicitly incorporate the knowledge that exists in two-dimensional images and models into the process of training a semantic segmentation model of point clouds so that the model can improve its accuracy and efficiency without requiring more data.
The proposed framework is designed based on the knowledge distillation and teacher-student methods and can be easily applied to any semantic segmentation model of point clouds regardless of its type. The results of the proposed approach on the well-known S3DIS dataset show a 7% increase in the MIoU metric on a base and common model for point cloud semantic segmentation without increasing the number of weights and training data for this model.
Complete Pipeline Overview:
Detailed Architecture of the Teacher Model (Zoom in for clarity):
