Run Uploading

Required Files

When uploading a run, the following file must be present:

• Run description, xml

Several other files are optional, e.g.,

• predictions, arff
• trace, arff
• model serialized (rather not, disk quota's)
• model readable (rather not, disk quota's)

Run Description

The Run Description is an XML file, following this XSD Schema. The following fields are available. Note that some fields from the XSD when uploading, because they are supposed to be set by the server:

• task_id; The id of the task that is solved in this run. Note that if you give a wrong task ID, this will most likely be detected during evaluation.
• flow_id; A id reference to the flow (algorithm, workflow,...) used to solve the task. This id is returned by the API when you first upload the implementation, or given in the implementation description when you download an existing implementation. Please note: If you give an invalid flow id, this can only be detected when people rerun the flow with your hyperparameter settings. Please use with utmost care!
• error_message; Whenever an error occurs during the run, this can be reported here for further inspection and on line logging purposes. This will be deprecated and not uploading erroneous runs is preferred.
• run_details; Any other information useful for reproducing the run, e.g. exact package versions, hardware info. -->
• parameter_setting; for this please see the complex type parameter_setting. Same as with flow id, please handle with utmost care.
• tag; Tags, describing the run
• output_data; all data that was the output of this run, i.e., predictions, evaluation scores. Most of this will be determined server side, but users can also provide evaluation scores for their own evaluation measures.

Parameter Setting

Information to be added

Output Data

This field allows to add client side outputs to the run (apart from the predictions). For example, workbenches like Weka and scikit-learn can add their calculated Accuracy measure to the upload, to be verified by the server. Also runtime measurements can be added here. These follow the complex type evaluation. See Evaluation Measures for a list of supported evaluation measures.

The evaluation has the following important fields:

• name; A reference to the math_function that was used
• value; scalar value that was the result of this evaluation measure
• stdev; the standard deviation if it was performed over multiple folds
• array_data; in case it is a class-specific measure (e.g., precision, recall) the per class scores And the following attributes:
• repeat; the cross-validation repeat
• fold; the cross-validation fold
• sample; for learning curve tasks, will be deprecated
• interval_start; for data stream tasks, will be deprecated
• interval_end; for data stream tasks, will be deprecated

Note that it is important to set the correct attributes. The following combinations of attributes are supported:

• None; the evaluation measure calculated over the full dataset (in case of run time, the time to build the model over the full dataset; in case of other measures, the average over the repeats/folds)
• repeat / fold; the evaluation measure calculated in this given repeat / fold
• repeat / fold / sample; learning curve related, might be deprecated soon
• interval_start / interval_end; data stream related, might be deprecated soon

Trace

When doing hyperparameter optimization, an additional arff file with the trace can be uploaded. This contains the following fields:

• repeat: cross-validation repeat
• fold: cross-validation fold
• iteration: the index order within this repeat/fold combination
• evaluation (float): the evaluation score that was attached based on the validation set
• selected {True, False}: Whether in this repeat/run combination this was the selected hyperparameter configuration (exactly one should be tagged with True)
• Per optimized parameter a column that has the name of the parameter and the prefix "parameter_"