Skip to content

Latest commit

 

History

History
90 lines (62 loc) · 4.43 KB

README.md

File metadata and controls

90 lines (62 loc) · 4.43 KB
Raw

License

DSTC10 Track 2 - Task 1: Dialogue State Tracking

This repository contains the data, scripts and baseline code for DSTC10 Track 2 - Task 1. This challenge task aims to develop robust dialogue state trackers on spoken conversations.

Organizers: Seokhwan Kim, Yang Liu, Di Jin, Alexandros Papangelis, Behnam Hedayatnia, Karthik Gopalakrishnan, Dilek Hakkani-Tur

Important Links

Task

Participants will develop dialogue state trackers using any existing public data sets. In the test phase, participants will be given a set of unlabeled test instances and submit up to 5 system outputs.

Evaluation

Each submission will be evaluated in the following metrics:

  • Joint metric
    • Joint goal accuracy
  • Slot-level metrics
    • Slot-level accuracy
    • Value prediction: Precision/Recall/F-measure
    • None prediction: Precision/Recall/F-measure

Please find more details at scores.py.

Data

In this challenge task, participants are allowed to use any public data (including the validation data) to develop their trackers. While most existing data sets include written conversations, the DSTC10 validation dataset contains the n-best ASR outputs for spoken conversations.

In the test phase, participants will be evaluated on the results generated by their models for the unlabeled test set also with the n-best ASR outputs for spoken conversations. The test set will be on the same domains, entities and locales as the validation set.

Data and system output format details can be found from data/README.md.

Participation

Each participating team will submit up to 5 system outputs for the test instances in logs.json.

The system outputs must follow the same format as labels.json for the validation set. Before making your submission, please double check if every file is valid with no error from the following script:

$ python scripts/check_results.py --dataset test --dataroot data/ --outfile [YOUR_SYSTEM_OUTPUT_FILE]
Found no errors, output file is valid

Any invalid submission will be excluded from the official evaluation.

Once you're ready, please make your submission by completing the Submission Form by 11:59PM UTC-12 (anywhere on Earth), September 21, 2021.

Timeline

  • Validation data released: Jun 14, 2021
  • Test data released: Sep 13, 2021
  • Entry submission deadline: Sep 21, 2021
  • Objective evaluation completed: Sep 28, 2021

Rules

  • Participation is welcome from any team (academic, corporate, non profit, government).
  • Each team can participate in either or both sub-tracks by submitting up to 5 entries for each track.
  • The identity of participants will NOT be published or made public. In written results, teams will be identified as team IDs (e.g. team1, team2, etc). The organizers will verbally indicate the identities of all teams at the workshop chosen for communicating results.
  • Participants may identify their own team label (e.g. team5), in publications or presentations, if they desire, but may not identify the identities of other teams.
  • Participants are allowed to use any external datasets, resources or pre-trained models.
  • Participants are NOT allowed to do any manual examination or modification of the test data.
  • All the submitted system outputs with the evaluation results will be released to public after the evaluation period.

Contact

Join the DSTC mailing list to get the latest updates about DSTC10

For specific enquiries about DSTC10 Track2

Please feel free to contact: seokhwk (at) amazon (dot) com