Where do we get the result file for all evaluation

[DhavalRepo18]

Assuming we do not want to run the benchmark, but want to run a new method and get the comparison done. What is the process?

[qhliu26]

Hi! You can now access the evaluation results on the benchmark here https://github.com/TheDatumOrg/TSB-AD/tree/main/benchmark_exp/benchmark_eval_results. Let us know if any help is needed.

[DhavalRepo18]

Thanks.

[DhavalRepo18]

@qhliu26 thanks for uploading the (subset) of results. I have one question what if results are not reproducible? Do we need further investigation? or all care has been taken to ensure results are reproducible?

[qhliu26]

Hi! Based on my experience, various factors such as hyperparameter settings, normalization techniques, random seeds, or even different Python versions can impact the performance and evaluation results to some extent. Some of these variations can be mitigated by running the algorithms multiple times and reporting the results with error bars. Additionally, comparing performance across the entire benchmark can provide a more reliable and robust evaluation. Specifically the evaluation results currently available on github are produced under numpy random seed 2024 and python 3.9. If there are specific algorithms for which you have observed significant deviations from our reported results, please let us know. We will be happy to investigate further.