PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos

Authors

  • Tianyu Luan Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China
  • Yali Wang Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China
  • Junhao Zhang Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China
  • Zhe Wang University of California, Irvine
  • Zhipeng Zhou Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China
  • Yu Qiao Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China SIAT Branch, Shenzhen Institute of Artificial Intelligence and Robotics for Society

Keywords:

3D Computer Vision, Biometrics, Face, Gesture & Pose

Abstract

The end-to-end Human Mesh Recovery (HMR) approach has been successfully used for 3D body reconstruction. However, most HMR-based frameworks reconstruct human body by directly learning mesh parameters from images or videos, while lacking explicit guidance of 3D human pose in visual data. As a result, the generated mesh often exhibits incorrect pose for complex activities. To tackle this problem, we propose to exploit 3D pose to calibrate human mesh. Specifically, we develop two novel Pose Calibration frameworks, i.e., Serial PC-HMR and Parallel PC-HMR. By coupling advanced 3D pose estimators and HMR in a serial or parallel manner, these two frameworks can effectively correct human mesh with guidance of a concise pose calibration module. Furthermore, since the calibration module is designed via non-rigid pose transformation, our PC-HMR frameworks can flexibly tackle bone length variations to alleviate misplacement in the calibrated mesh. Finally, our frameworks are based on generic and complementary integration of data-driven learning and geometrical modeling. Via plug-and-play modules, they can be efficiently adapted for both image/video-based human mesh recovery. Additionally, they have no requirement of extra 3D pose annotations in the testing phase, which releases inference difficulties in practice. We perform extensive experiments on the popular benchmarks, i.e., Human3.6M, 3DPW and SURREAL, where our PC-HMR frameworks achieve the SOTA results.

Downloads

Published

2021-05-18

How to Cite

Luan, T., Wang, Y., Zhang, J., Wang, Z., Zhou, Z., & Qiao, Y. (2021). PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos. Proceedings of the AAAI Conference on Artificial Intelligence, 35(3), 2269-2276. Retrieved from https://ojs.aaai.org/index.php/AAAI/article/view/16326

Issue

Section

AAAI Technical Track on Computer Vision II