Update scores and paper references

2024/papers/index.md

@@ -18,6 +18,12 @@ __For this year's Challenge, please cite the Challenge description paper:__
 
 The above paper is a draft. We will update it after CinC to include information about the conclusion of the Challenge.
 
+__For this year's Challenge data, please cite the Challenge data paper:__
+
+[Reyna MA, Deepanshi, Weigle J, Koscova Z, Campbell K, Shivashankara KK, et al. ECG-Image-Database: A dataset of ECG images with real-world imaging and scanning artifacts; a foundation for computerized ECG image digitization and analysis, 2024. URL https://arxiv.org/abs/2409.16612.](https://arxiv.org/abs/2409.16612)
+
+The above paper is a draft. We will update it after CinC as well.
+
 __For the ECG digitization code, please cite the ECG-Image-Kit publications:__
 
 [Shivashankara KK, Deepanshi, Shervedani AM, Reyna MA, Clifford GD, Sameni R. ECG-Image-Kit: a synthetic image generation toolbox to facilitate deep learning-based electrocardiogram digitization. Physiological Measurement 2024; 45:055019.](https://iopscience.iop.org/article/10.1088/1361-6579/ad4954)
@@ -38,4 +44,50 @@ Please *read* and follow the templates. Each year, we need to provide feedback f
 
 ## Conference Papers
 
-The conference papers for [Computing in Cardiology 2024](https://www.cinc2024.org/) will be available on the [CinC](https://www.cinc.org/cinc-papers-on-line/) website.
+The conference papers for [Computing in Cardiology 2024](https://www.cinc2024.org/) will be available on the [CinC](https://www.cinc.org/cinc-papers-on-line/) website.
+
+## BibTeX Citations
+
+You can cite the above papers with the following BibTeX references:
+
+    @article{2024Challenge,
+        author = {Reyna, Matthew A. and Deepanshi and Weigle, James and Koscova, Zuzana and Elola, Andoni and Seyedi, Salman and Campbell, Kiersten and Clifford, Gari D. and Sameni, Reza},
+        title = {{D}igitization and {C}lassification of {ECG} {I}mages: {T}he {G}eorge {B}.\ {M}oody {P}hysio{N}et {C}hallenge 2024},
+        journal={Computing in Cardiology},
+        volume={51},
+        pages={1--4},
+        year={2024},
+        publisher={IEEE}
+        }
+
+    @misc{ECG-Image-Database,
+        title={{ECG}-{I}mage-{D}atabase: A Dataset of {ECG} Images with Real-World Imaging and Scanning Artifacts; A Foundation for Computerized {ECG} Image Digitization and Analysis}, 
+        author={Matthew A. Reyna and Deepanshi and James Weigle and Zuzana Koscova and Kiersten Campbell and Kshama Kodthalu Shivashankara and Soheil Saghafi and Sepideh Nikookar and Mohsen Motie-Shirazi and Yashar Kiarashi and Salman Seyedi and Gari D. Clifford and Reza Sameni},
+        year={2024},
+        eprint={2409.16612},
+        archivePrefix={arXiv},
+        primaryClass={q-bio.QM},
+        url={https://arxiv.org/abs/2409.16612}, 
+        }
+
+    @article{ECGImageKit2024-PMEA,
+        author = {Kshama Kodthalu Shivashankara and Deepanshi and Afagh Mehri Shervedani and Matthew A. Reyna and Gari D. Clifford and Reza Sameni},
+        title = {{ECG}-{I}mage-{K}it: a synthetic image generation toolbox to facilitate deep learning-based electrocardiogram digitization},
+        journal = {Physiological Measurement},
+        publisher = {IOP Publishing},
+        year = {2024},
+        volume = {45},
+        pages = {055019},
+        doi = {10.1088/1361-6579/ad4954}
+        }
+
+    @article{Goldberger2000,
+        title={{P}hysio{B}ank, {P}hysio{T}oolkit, and {P}hysio{N}et: Components of a New Research Resource for Complex Physiologic Signals},
+        author={Goldberger, Ary L and Amaral, Luis AN and Glass, Leon and Hausdorff, Jeffrey M and Ivanov, Plamen Ch and Mark, Roger G and Mietus, Joseph E and Moody, George B and Peng, Chung-Kang and Stanley, H Eugene},
+        journal={Circulation},
+        volume={101},
+        number={23},
+        pages={e215--e220},
+        year={2000},
+        publisher={American Heart Association}
+        }

2024/results/hackathon_scores_classification.tsv

@@ -0,0 +1,2 @@
+Team name	Leaderboard: F-measure	Color scans of clean papers: F-measure	Black-and-white scans of clean papers: F-measure	Mobile phone photos of clean papers: F-measure	Mobile phone photos of stained papers: F-measure	Mobile phone photos of deteriorated papers: F-measure	Color scans of deteriorated papers: F-measure	Black-and-white scans of deteriorated papers: F-measure	Screenshots of computer monitor: F-measure
+The Easy Geese	0.082	0.007	0.112	0.232	0.262	0.341	0.501	0.101	0.362

2024/results/hackathon_scores_digitization.tsv

@@ -0,0 +1,3 @@
+Team name	Leaderboard: SNR	Color scans of clean papers: SNR	Black-and-white scans of clean papers: SNR	Mobile phone photos of clean papers: SNR	Mobile phone photos of stained papers: SNR	Mobile phone photos of deteriorated papers: SNR	Color scans of deteriorated papers: SNR	Black-and-white scans of deteriorated papers: SNR	Screenshots of computer monitor: SNR
+ECGénie	0.057	0.054	0.060	-3.017	-3.080	-1.771	-0.193	-0.189	-3.787
+The Easy Geese	-5.272	-0.071	-10.474	-2.330	-1.755	-1.346	0.061	-3.713	-1.739

2024/results/index.md

@@ -15,6 +15,7 @@ Teams that satisfied all of the [Challenge rules](../#rules) were eligible for r
 - [Team summary table](team_summary_table.tsv): summary of the teams that participated in the 2024 Challenge. This table includes ranking and prize eligibility information.
 - Scores and rankings (sorted by rank) for __official entries, i.e., eligible for rankings__: [digitization](official_scores_digitization.tsv) and [classification](official_scores_classification.tsv) scores
 - Scores (sorted alphabetically by team name) for __unofficial entries, i.e., not eligible for rankings__: [digitization](unofficial_scores_digitization.tsv) and [classification](unofficial_scores_classification.tsv) scores
+- Scores (sorted by rank) for __hackathon entries__: [digitization](hackathon_scores_digitization.tsv) and [classification](hackathon_scores_classification.tsv) scores
 
 The score tables include signal-to-noise ratios (SNR) scores for the digitization task and macro _F_-measure scores for the classification tasks. All scores are on the hidden data. The leaderboard scores are scores reported during the official phase on color and black-and-white scans of paper ECGs from the PTB-XL data. The other scores are scores for models retrained on the full test set and evaluated on variants of the paper ECGs from the PTB-XL data, including scans and photographs of clean, stained, and deteriorated ECG papers and photographs of computer monitors showing ECGs.
 

2024/results/official_scores_classification.tsv

@@ -5,9 +5,13 @@ Rank	Team name	Leaderboard: F-measure	Color scans of clean papers: F-measure	Bla
 4	CeZIS	0.645	0.691	0.586	0.648	0.659	0.700	0.700	0.576	0.691
 5	ECG_UVA	0.516	0.580	0.481	0.070	0.068	0.076	0.595	0.468	0.086
 6	DSAIL	0.429	0.502	0.118	0.072	0.080	0.078	0.463	0.093	0.104
-7	USST_Med	0.393	0.056	0.056	0.056	0.056	0.056	0.056	0.056	0.056
-8	Revenger	0.330	0.332	0.327	0.306	0.316	0.306	0.331	0.319	0.288
-9	PulsePlex	0.317	0.222	0.222	0.222	0.222	0.222	0.222	0.222	0.222
-10	DSAIL_Nyeri	0.227	0.290	0.104	0.055	0.058	0.053	0.271	0.077	0.058
-11	Leicester Fox	0.068	0.044	0.078	0.068	0.058	0.067	0.056	0.074	0.060
-12	Ahus AI Lab	0.056	0.056	0.056	0.056	0.056	0.056	0.056	0.056	0.056
+7	PKU_NIHDS	0.421								
+8	USST_Med	0.393	0.056	0.056	0.056	0.056	0.056	0.056	0.056	0.056
+9	Revenger	0.330	0.332	0.327	0.306	0.316	0.306	0.331	0.319	0.288
+10	PulsePlex	0.317	0.222	0.222	0.222	0.222	0.222	0.222	0.222	0.222
+11	ISIBrno-AIMT	0.306								
+12	DSAIL_Nyeri	0.227	0.290	0.104	0.055	0.058	0.053	0.271	0.077	0.058
+13	TimeBeater	0.183								
+14	DeTECRohr	0.153								
+15	Leicester Fox	0.068	0.044	0.078	0.068	0.058	0.067	0.056	0.074	0.060
+16	Ahus AI Lab	0.056	0.056	0.056	0.056	0.056	0.056	0.056	0.056	0.056

2024/results/official_scores_digitization.tsv

@@ -1,13 +1,17 @@
 Rank	Team name	Leaderboard: SNR	Color scans of clean papers: SNR	Black-and-white scans of clean papers: SNR	Mobile phone photos of clean papers: SNR	Mobile phone photos of stained papers: SNR	Mobile phone photos of deteriorated papers: SNR	Color scans of deteriorated papers: SNR	Black-and-white scans of deteriorated papers: SNR	Screenshots of computer monitor: SNR
 1	SignalSavants	12.151	4.930	3.479	-1.071	-0.723	-1.304	0.506	0.514	-1.759
 2	BAPORLab	5.493	6.220	4.735	-0.261	-1.116	0.807	0.905	0.358	0.123
-3	Ahus AI Lab	3.047	3.318	2.777	0.138	-0.345	0.792	0.670	-0.320	-0.512
-4	USST_Med	2.202	-0.058	-0.058	-0.375	-0.375	-0.375	-0.375	-0.375	-0.375
-5	Inventec AIC	0.397	4.101	-2.569	-4.574	-5.106	-3.782	-2.157	-2.802	-2.450
-6	TimeBeater	-0.056	-0.058	-0.056	-0.375	-0.375	-0.375	-0.375	-0.375	-0.375
-7	VinUniTeam	-0.066	-0.069	-0.062	-0.027	-0.027	-0.029	-0.030	-0.026	-0.047
-8	PulsePlex	-0.081	-0.082	-0.080	-0.303	-0.301	-0.302	-0.303	-0.303	-0.303
-9	Revenger	-0.733	-0.148	-1.267	-9.019	-8.545	-6.398	-1.636	-3.559	-6.532
-10	PapyrusECG	-5.192	1.211	-11.781	-11.960	-12.635	-8.888	-1.549	-10.926	-11.648
-11	Leicester Fox	-12.032	-12.032	-12.032	-12.813	-12.813	-12.813	-12.813	-12.813	-12.813
-12	DeRC_ECG	-31.302	-31.302	-31.302	-31.122	-31.122	-31.122	-31.122	-31.122	-31.122
+3	wavie_ABI	5.469								
+4	Ahus AI Lab	3.047	3.318	2.777	0.138	-0.345	0.792	0.670	-0.320	-0.512
+5	USST_Med	2.202	-0.058	-0.058	-0.375	-0.375	-0.375	-0.375	-0.375	-0.375
+6	mins-eth	0.893								
+7	Inventec AIC	0.397	4.101	-2.569	-4.574	-5.106	-3.782	-2.157	-2.802	-2.450
+8	TimeBeater	-0.056	-0.058	-0.056	-0.375	-0.375	-0.375	-0.375	-0.375	-0.375
+9	VinUniTeam	-0.066	-0.069	-0.062	-0.027	-0.027	-0.029	-0.030	-0.026	-0.047
+10	RoadRuNNers	-0.071								
+11	PulsePlex	-0.081	-0.082	-0.080	-0.303	-0.301	-0.302	-0.303	-0.303	-0.303
+12	ISIBrno-AIMT	-0.675								
+13	Revenger	-0.733	-0.148	-1.267	-9.019	-8.545	-6.398	-1.636	-3.559	-6.532
+14	PKU_NIHDS	-1.103								
+15	PapyrusECG	-5.192	1.211	-11.781	-11.960	-12.635	-8.888	-1.549	-10.926	-11.648
+16	Leicester Fox	-12.032	-12.032	-12.032	-12.813	-12.813	-12.813	-12.813	-12.813	-12.813