Style-based controllable motion generation

Panini Bhamidipati, Shubh Maheshwari, Rose Yu
Department of Computer Science
UC San Diego
Paper Supplementary Poster Code Data

A novel constraint composition for motion diffusion to recommend clinically optimal body poses.

Summary

Recommending clinically optimal body motion can significantly improve an athlete's performance. The goal of our work is to suggest optimal body poses an athlete should perform. To distinguish between optimal and sub-optimal pose we use the Movement Competency Screen (MCS) to guide the generative model to produce motion with high fidelity and diversity.

  • We prosose a novel constraint composition formulation to incorporate multiple constraints like MCS Score and text prompt.

  • Furthermore, we release a new dataset (OpenCap Dataset) encompassing more than 3000 high-quality human motion samples from 13 different primitive movements like Squats, Pull-ups, and Push-ups with their corresponding MCS Score labeled by experts.

  • Our experiments show qualitative and quantitative improvement over baselines by incorporating constrain composition.

Dataset

The datset comprises of over 3000 high-quality motion samples and 13 action labels. Each video is accompanied a "MCS Score," ranging from 1 to 5. This score serves as a qualitative measure, with 1 indicating suboptimal movement and 5 representing an exemplary execution of the specific action.

Temporal Segmentation

For the temporal segmentation, that is to find the start and the end frame for each exercise iteration, we found out that the angular velocity of the joints is a good indicator.

The attached video shows an activity being performed. The color of the mesh changes for every iteraction. The top right plot shows the DTW metric similarity between different selected segments. The middle plot in the image shows the angular velocity for each joint ( in deg/s ) for each timestep. The red peaks in the bottom plot showcase the timestep at which an iteration ends.

The uniqueness of the OpenCap dataset lies in its focus on capturing a diverse range of human motions, with an emphasis on activities commonly associated with physical fitness. To ensure the dataset’s richness and authenticity, the motion samples were meticulously recorded by observing athletes from the University of California, San Diego (UCSD) performing various sports movements.

Generating Style Tags

  • The style tags encompass both the exercise category (such as push-ups and squats) and the MCS scores assigned by experts.

  • We train 2 classifiers to predict the category and the MCS score.

Extracted Curve Skeleton

The above table shows the accuracy between classifiers with different window size based.

Comparisions

Constraint composition

Interpolation end reference image.

Stylized generatation

Interpolation end reference image.

Compared to motion skeleton extracted from other methods, by incorporating structural cues, ours is more effective at embedding skeleton from incomplete mesh sequence.

BibTeX

@article{,
  author    = {Bhamidipati, Panini and Maheshwari, Shubh and Yu, Rose},
  title     = {Style-based controllable motion generation},
  journal   = {},
  year      = {2024},
}