Learning Complex 3D Human Self-Contact


  • Mihai Fieraru Institute of Mathematics of the Romanian Academy
  • Mihai Zanfir Institute of Mathematics of the Romanian Academy
  • Elisabeta Oneata Institute of Mathematics of the Romanian Academy
  • Alin-Ionut Popa Institute of Mathematics of the Romanian Academy
  • Vlad Olaru Institute of Mathematics of the Romanian Academy
  • Cristian Sminchisescu Lund University Institute of Mathematics of the Romanian Academy




3D Computer Vision, AI Responses to the COVID-19 Pandemic (Covid19)


Monocular estimation of three dimensional human self-contact is fundamental for detailed scene analysis including body language understanding and behaviour modeling. Existing 3d reconstruction methods do not focus on body regions in self-contact and consequently recover configurations that are either far from each other or self-intersecting, when they should just touch. This leads to perceptually incorrect estimates and limits impact in those very fine-grained analysis domains where detailed 3d models are expected to play an important role. To address such challenges we detect self-contact and design 3d losses to explicitly enforce it. Specifically, we develop a model for Self-Contact Prediction (SCP), that estimates the body surface signature of self-contact, leveraging the localization of self-contact in the image, during both training and inference. We collect two large datasets to support learning and evaluation: (1) HumanSC3D, an accurate 3d motion capture repository containing 1,032 sequences with 5,058 contact events and 1,246,487 ground truth 3d poses synchronized with images collected from multiple views, and (2) FlickrSC3D, a repository of 3,969 images, containing 25,297 surface-to-surface correspondences with annotated image spatial support. We also illustrate how more expressive 3d reconstructions can be recovered under self-contact signature constraints and present monocular detection of face-touch as one of the multiple applications made possible by more accurate self-contact models.




How to Cite

Fieraru, M., Zanfir, M., Oneata, E., Popa, A.-I., Olaru, V., & Sminchisescu, C. (2021). Learning Complex 3D Human Self-Contact. Proceedings of the AAAI Conference on Artificial Intelligence, 35(2), 1343-1351. https://doi.org/10.1609/aaai.v35i2.16223



AAAI Technical Track on Computer Vision I