Cancelled submissions use up submissions count quota

[curious-broccoli]

(Not urgent)

I'm running Codabench 1.17. locally. One submission got stuck in the queue so I as the submitter cancelled it, it was in the state "Submitting".

Is it intended that it still counts to the submission limit? Users also can't hard-remove it on their own.

[ihsaan-ullah]

This has to be discussed. It is possible to exploit the quota if you cancel a submission while it is running. For example you print something important in the logs, then cancel the submission and prepare another submission with the logged info.

Edit

Maybe it is not that important because people can do the same trick to fail a submission intentionally

[curious-broccoli]

I guess that's true, but maybe it can be made so that submissions in "Submitting" state, don't count.