Are your moves any good?

Hi there! 'Dance Sync', originally called 'Pose' is a program that allows you to compare your dancing to your model dancer's.

When learning a dance, it can be hard to pinpoint where you're going wrong by just looking in the mirror. Instead, record a video of your dancing and plug it into 'Dance Sync' - it will analyse your movement and compare it to the video of the orginial dancer.

You will get a comparison video, with alerts where you are not in sync and a final score at the end.

Quickstart

Go to the Executing section

Happy dancing!

How does it work?

1. How do we compare the dancers?

We compare the angles of the dancers' arms, legs, torso etc...

1. Use a Pose Estimation Model to find the postions of a dancer's joints
   

2. For the limbs we want to compare, find the corresponding joint postions
   

3. Compute angle using (x,y) coordinates of joints and some trig
   

4. Find difference between the limb's gradient (i.e. angle) of dancer1 and dancer2. Repeat and average for ALL limbs in once frame. Repeat for all frames
   

2. What happens if we input videos of different lengths?

There may be a case where the dance song starts a little bit later in one video. This is not a problem as the program automatically syncs the two dancers so comparisons only happen when they are dancing to the same part of the song!

But how do we do this?
We sync the dancers using the music! Both dancers are dancing to the same music so we can use this to find which when they are doing the same choreo.

1. Find the offset between the audio in the two videos by...
   

• Computing the cross-correlation of the sound waves (function to tell us how similar the waves are,) for different time lags
  (Good videos explaining cross-correlations here: https://youtu.be/_r_fDlM0Dx0 | https://youtu.be/ngEC3sXeUb4 )
  
• Find the time lag (a.k.a. phase difference) where the cross-correlation is a maximum
  (This is also known as TIME DELAY ANALYSIS)

2. Trim the video in which the dancing starts later
   

• We cut the video with same duration as the time lag so...
  
• both dancers start the same choreography at the beginning of the video!
  

Want to try it out?

I am working on making this accessible for everyone: non-programmers included! Unfortunately, I'm not quite there yet so you will need to install a bunch of packages (sorry!) and run the code on macOS 12.5.1.

Dependencies

Have these installed these before running:

• OpenCV2 https://pypi.org/project/opencv-python/
• MediaPipe https://google.github.io/mediapipe/getting_started/install.html#installing-on-macos
• Numpy https://numpy.org/install/
• Praat https://www.fon.hum.uva.nl/praat/download_mac.html
• Ffmpeg https://ffmpeg.org/download.html
  

Executing

• Download the folder
  
• Launch with these arguments: python dance.py video/chuu.mov video/cyves.mov
  
• Make sure the videos are in .mov format and are the same dimension i.e. 1280x720 px
  

Acknowledgments

• Inspiration: https://www.youtube.com/watch?v=zxhXj72WClE
• Useful documentation of how pose estimation works: https://google.github.io/mediapipe/solutions/pose.html
• (Dr Spyros Kousidis') Tutorial on syncing videos using reference audio: http://www.dsg-bielefeld.de/dsg_wp/wp-content/uploads/2014/10/video_syncing_fun.pdf