Pose Estimation

Pose estimation is the computer vision task of localising keypoints — typically anatomical landmarks such as joints — on a body, hand, or object, and inferring a structured representation of its pose. Outputs may be two-dimensional pixel coordinates, three-dimensional positions in camera or world coordinates, or higher-level skeletons and meshes. The task underlies applications from sports analytics to extended reality avatars and physiotherapy.

Problem variants

Pose estimation systems are categorised along several axes. Two-dimensional pose estimation predicts pixel-space keypoint locations from a single image, while three-dimensional pose estimation lifts these to 3D coordinates, typically with the help of camera calibration, multi-view input, or learned priors. Single-person systems assume one subject in the frame, while multi-person systems handle crowded scenes through top-down (detect-then-pose) or bottom-up (predict-all-keypoints-then-group) pipelines. Specialised variants include hand pose estimation, facial landmark detection, animal pose estimation, and six-degree-of-freedom object pose estimation for robotics. Dense models such as DensePose produce per-pixel correspondences to a canonical body surface, supporting full-body reconstruction.

Architectures

Modern pose estimators are built on convolutional and transformer backbones. Influential systems include OpenPose, which introduced part affinity fields for multi-person bottom-up pose estimation; HRNet, which maintains high-resolution feature maps throughout the network for precise keypoint localisation; PoseNet and MoveNet, optimised for mobile and edge deployment; AlphaPose, a top-down system with high accuracy; and YOLOv8-Pose, which extends real-time object detection to keypoint regression. Google's MediaPipe provides production-grade pose, hand, and face solutions tuned for on-device inference. Transformer-based estimators such as ViTPose match or surpass convolutional baselines on COCO Keypoints.

Datasets and metrics

COCO Keypoints is the dominant benchmark for 2D multi-person pose estimation, with 17 annotated joints per person. MPII Human Pose and CrowdPose stress-test models on cluttered scenes. Human3.6M, 3DPW, and AMASS support 3D pose research. Average Precision based on Object Keypoint Similarity is the standard metric for COCO; for 3D pose, Mean Per-Joint Position Error is widely reported. Procrustes-aligned Mean Per-Joint Position Error removes global rotation and translation to isolate intrinsic skeletal accuracy.

Common challenges

Occlusion, motion blur, unusual viewpoints, scale variation, clothing variability, and crowded scenes remain difficult. Reliable 3D pose estimation from a single monocular view is fundamentally ambiguous and benefits from multi-view or temporal models. Generalisation across body types, age groups, and ethnicities is a recognised fairness concern, with several datasets and audits flagging performance gaps for under-represented demographics.

Applications

Pose estimation powers fitness coaching applications, physical therapy and rehabilitation tools, sports performance analytics, motion capture for animation and gaming, augmented reality avatars and try-on, surveillance and crowd analytics, ergonomic assessment in workplaces, gesture-based interfaces, sign language recognition, and human-robot interaction. Industrial uses include pose-based safety monitoring in manufacturing and construction. Pose estimation has been deployed across Malaysian sports, healthcare, and industrial settings. The National Sports Institute of Malaysia (Institut Sukan Negara) uses pose estimation systems for technique analysis in disciplines including badminton, diving, and gymnastics, leveraging both commercial motion capture suites and computer vision tools built on OpenPose and HRNet. The Badminton Association of Malaysia has piloted pose-based shot analytics for elite athletes. Hospitals under the Ministry of Health, including selected Hospital Kuala Lumpur clinics, and private chains such as Sunway Medical Centre, IHH Healthcare's Pantai Hospital, and KPJ Healthcare, have piloted pose estimation for physiotherapy and post-stroke rehabilitation tracking. Compliance with the Personal Data Protection Act 2010 requires explicit patient consent, on-device or in-clinic processing where possible, and clear retention controls. Industrial users include Petronas, which uses pose-based safety monitoring at upstream facilities, and Sime Darby Plantation, which has explored ergonomic assessment of harvesting workers. Manufacturing tenants in the Penang Free Industrial Zone deploy pose estimation in line-side worker assistance systems. Public universities including Universiti Teknologi Malaysia, Universiti Sains Malaysia, and Multimedia University contribute pose estimation research, with MDEC supporting startup activity through the Malaysia Digital programme.

1. Cao, Z. et al. (2017). Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. CVPR.
2. Sun, K. et al. (2019). Deep High-Resolution Representation Learning for Human Pose Estimation. CVPR.
3. Lin, T.-Y. et al. (2014). Microsoft COCO: Common Objects in Context. ECCV.
4. Google. (2023). MediaPipe Pose Landmarker Guide. developers.google.com/mediapipe.