Competition Management

Using the CLI a new competition is easily created:

DRES> competition create -n NAME -d DESCRIPTION

Creates a (uniquely) named competition NAME, described by DESCRIPTION.

Alternatively, a similar result can be achieved from the frontend UI.

Then, using the frontend UI, a competition might be expaned by adding teams (with users), task types, task groups and tasks.

Task Types

A task type is basically an organisational or administrative template for a task (group). It defines which types of media items the task accepts, how its hints (aka task description) is built and its default duration. Additionally, scoring and other settings are performed on a task type level. For the large international evaluation campagns, VBS and LSC, DRES provides task type templates.

The settings are:

Target:

* JUDGEMENT - Human judges decide about the submissions
* MULTIPLE_MEDIA_ITEMS - Multiple media items are accepted (i.e. multiple image items are targeted)
* SINGLE_MEDIA_ITEM - A single media item is the target
* SINGLE_MEDIA_SEGMENT - In case of video items, a sub segment of it is targeted

Task Components:
 
* EXTERNAL_IMAGE - Path to external image
* EXTERNAL_VIDEO - Path to external video
* IMAGE_ITEM - Image items from media collections
* TEXT - Textual query hints
* VIDEO_ITEM_SEGEMNT - Visual hints in form of a sub-segement of a video item from media collections.

Scoring:

* AVS - Scoring like in Ad-hoc Video Search form VBS
* KIS - Scoring like in Known Item Search from VBS

Submission Filter

* NO_DUPLICATES - Does not allow duplicate submissions
* ONCE_CORRECT_PER_TEAM - Only one correct submission per team allowed - any further are ignored
* TEMPORAL_SUBMISSION - The temporal aspect realtive to the task time is important

Options

* HIDDEN_RESULTS - Whether the submissions (wrong or correct) are shown in the viewer for everybody
* MAP_TO_SEGMENT - Whether the submissions should internally be mapped to the known segments (see media collection for further info). This is usally desired.

Task Group

A task group is a collection of tasks, all with the same task type. One can for example create a task type and create two groups for this type: One for testing and one for the actual competition

Task

A task has a name, a target media collection and a duration. One or more targets (based on task type settings) and one or more query descriptions (aka task hints). Start and end times are in seconds and automatically computed: For three hints, the first from the beginnin visible, the second after thirty seconds and the last after sixty seconds are set up as follows: first has neither start nor end set. Second has start set to 30 and last has start set to 60.