Articulated Human pose evaluation/benchmark

Benchmark your method against several other methods on the popular FLIC and LSP datasets.

How to use

To benchmark the algorithms simply run the scripts files benchmark_flic.m and benchmark_lsp.m to evaluate the algorithms on the FLIC and LSP datasets, respectively.

Adding a new algorithm to the list

Adding predictions of a new algorithm is fairly simple:

1. Create a folder with the name of the algorithm in algorithms/.
2. Create a file called algorithm.txt and assign a label/alias name for the algorithm to be used to identify the algorithms name in the plot's legend.
3. Add the predictions files with the keypoints/sticks coordinates with the names pred_keypoints_lsp_oc.mat, pred_keypoints_lsp_pc.mat, pred_sticks_lsp_oc.mat and pred_sticks_lsp_pc.mat.

Note: The scripts will skip the missing files when benchmarking a method.

Options

Several options are available for configuration. These, however, require the user to change the file manually.

Plot options:

• list: specify which algorithms to plot. If empty, plots all algorithms.
• bSave: save plot images to plots/ folder (if set to true).
• printLegend: prints a legend in every plot (if set to true).
• pcp_threshold: PCP evaluation threshold.
• pck_threshold: PCK evaluation threshold.

Available datasets

For now, the available datasets for PCK and PCP evaluation are the FLIC and LSP. Other datasets may be introduced if it is justifiable for inclusion.

@inproceedings{modec13,
    title={MODEC: Multimodal Decomposable Models for Human Pose Estimation},
    author={Sapp, Benjamin and Taskar, Ben},
    booktitle={In Proc. CVPR},
    year={2013},
}

@inproceedings{Johnson11,
	title = {Learning Effective Human Pose Estimation from Inaccurate Annotation},
	author = {Johnson, Sam and Everingham, Mark},
	year = {2011},
	booktitle = {IEEE Proc. CVPR}
}

Benchmark results

All available methods for the LSP benchmark were downloaded from MPII's website.

FLIC methods were gather from some authors's predictions available online.

Results of the algorithms are shown bellow.

Frames Labeled In Cinema (FLIC)

PCK(0.2) - Observer Centric

Method	Elbow	Wrist
Sapp et al., CVPR'13	72.5	54.5
Yang et al., CVPR'16	91.6	88.8
Chen et al., NIPS'14	89.8	86.8
Wei et al., CVPR'16	92.5	90.0
Newell et al., arXiv'16	98.0	95.5

Leeds Sport Pose (LSP)

PCP(0.5) - Person Centric

Method	Torso	Upper leg	Lower leg	Upper arm	Forearm	Head	PCP
Wang et al., CVPR'13	87.5	56.0	55.8	43.1	32.1	79.1	54.1
Pishchulin et al., ICCV' 13	88.7	63.6	58.4	46.0	35.2	85.1	58.0
Tompson et al., NIPS'14	90.3	70.4	61.1	63.0	51.2	83.7	66.6
Fan et al., CVPR'15	95.4	77.7	69.8	62.8	49.1	86.6	70.1
Chen et al., NIPS'14	96.0	77.2	72.2	69.7	58.1	85.6	73.6
Yang et al., CVPR'16	95.6	78.5	71.8	72.2	61.8	83.9	74.8
Rafi et al., BMVC'16	97.6	87.3	80.2	76.8	66.2	93.3	81.2
Belagiannis et al., arXiv'16	96.0	86.7	82.2	79.4	69.4	89.4	82.1
Lifshitz et al., ECCV'16	97.3	88.8	84.4	80.6	71.4	94.8	84.3
Pishchulin et al., CVPR'16	97.0	88.8	82.0	82.4	71.8	95.8	84.3
Yu et al., ECCV'16	98.0	93.1	88.1	82.9	72.6	83.0	85.4
Insafutdinov et al., ECCV'16	97.0	90.6	86.9	86.1	79.5	95.4	87.8
Wei et al., CVPR'16	98.0	92.2	89.1	85.8	77.9	95.0	88.3
Bulat et al., ECCV'16	97.7	92.4	89.3	86.7	79.7	95.2	88.9

PCK(0.2) - Person Centric

Method	Head	Shoulder	Elbow	Wrist	Hip	Knee	Ankle	Total
Wang et al., CVPR'13	84.7	57.1	43.7	36.7	56.7	52.4	50.8	54.6
Pishchulin et al., ICCV' 13	87.2	56.7	46.7	38.0	61.0	57.5	52.7	57.1
Tompson et al., NIPS'14	90.6	79.2	67.9	63.4	69.5	71.0	64.2	72.3
Fan et al., CVPR'15	92.4	75.2	65.3	64.0	75.7	68.3	70.4	73.0
Chen et al., NIPS'14	91.8	78.2	71.8	65.5	73.3	70.2	63.4	73.4
Yang et al., CVPR'16	90.6	78.1	73.8	68.8	74.8	69.9	58.9	73.6
Rafi et al., BMVC'16	95.8	86.2	79.3	75.0	86.6	83.8	79.8	83.8
Yu et al., ECCV'16	87.2	88.2	82.4	76.3	91.4	85.8	78.7	84.3
Belagiannis et al., arXiv'16	95.2	89.0	81.5	77.0	83.7	87.0	82.8	85.2
Lifshitz et al., ECCV'16	96.8	89.0	82.7	79.1	90.9	86.0	82.5	86.7
Pishchulin et al., CVPR'16	97.0	91.0	83.8	78.1	91.0	86.7	82.0	87.1
Insafutdinov et al., ECCV'16	97.4	92.7	87.5	84.4	91.5	89.9	87.2	90.1
Wei et al., CVPR'16	97.8	92.5	87.0	83.9	91.5	90.8	89.9	90.5
Bulat et al., ECCV'16	97.2	92.1	88.1	85.2	92.2	91.4	88.7	90.7

Acknowledgements

This code is a modified version of the original code made available by MPII.

License

The available code is released under the MIT license.