MoviE Text Audio QA (MetaQA)[1] contains more than 400K questions for both single and multi-hop reasoning, and provides more realistic text and audio versions. MetaQA serves as a comprehensive extension of WikiMovies., in English language. Containing 400,000+ in Text, MP3 file format.
According to the original paper, there are three versions of this dataset: Vanilla, NTM and Audio. Vanilla is a composed of original WikiMovies as 1-hop dataset, 21 types of 2-hop questions and 15 types of 3-hop questions. NTM is generated by translating the Vanilla set using machine translation models. Audio subset is generated by audio datasets with the help of text-to-speech (TTS) system.
If not specified, dataset below refers to Vanilla set of MetaQA.
This dataset can be downloaded via the link. Note that this dataset is divided into 1-hop, 2-hop and 3-hop in the original paper: Variational reasoning for question answering with knowledge graph.
| Model / System | Year | Hits@1 | F1 | Exact Match | Language | Reported by |
|---|---|---|---|---|---|---|
| SSKGQA | 2022 | 99.1 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| NRQA | 2022 | 98.1 | - | - | EN | Guo et al. |
| DCRN | 2022 | 97.5 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| VRN | 2017 | 97.5 | - | - | EN | Zhang et al. |
| EmbedKGQA | 2020 | 97.5 | - | - | EN | Saxena et al. |
| QAGCN | 2022 | 97.3 | - | - | EN | Wang et al. |
| NSM+p | 2021 | 97.3 | - | - | EN | He et al. |
| QNRKGQA | 2022 | 97.3 | - | - | EN | Ma et al. |
| NSM+h | 2021 | 97.2 | - | - | EN | He et al. |
| Edge-aware GNN | 2022 | 97.2 | 98.5 | - | EN | Zhang et al. |
| GlobalGraph | 2022 | 99.0 | 97.6 | - | EN | Zhang et al. |
| 2HR-DR | 2022 | 98.8 | 97.3 | - | EN | Zhang et al. |
| SGReader | 2022 | 96.7 | 96.0 | - | EN | Zhang et al. |
| GRAFT-Net | 2022 | 97.4 | 91.0 | - | EN | Zhang et al. |
| ARN_DistMult | 2023 | 97.12 | - | - | EN | Cui et al. |
| ARN_TuckER | 2023 | 97.11 | - | - | EN | Cui et al. |
| NSM | 2021 | 97.1 | - | - | EN | He et al. |
| ARN_ComplEx | 2023 | 97.09 | - | - | EN | Cui et al. |
| SRN | 2020 | 97.0 | - | - | EN | Qiu et al. |
| GraftNet | 2018 | 97.0 | - | - | EN | Sun et al. |
| PullNet | 2019 | 97.0 | - | - | EN | Sun et al. |
| ARN_ConvE | 2023 | 96.70 | - | - | EN | Cui et al. |
| MINERVA | 2022 | 96.3 | - | - | EN | Wang et al. |
| ReifKB | 2020 | 96.2 | - | - | EN | Cohen et al. |
| KV-MemNN | 2022 | 96.2 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| TransferNet | 2022 | 96.0 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| Borders et al. 's QA system | 2017 | 95.7 | - | - | EN | Zhang et al. |
| KV-MemNN | 2017 | 95.8 | - | - | EN | Zhang et al. |
| AlAgha, 2022 | 2022 | 95.4 | - | - | EN | AlAgha, 2022 |
| KGQA Based on Query Path Generation | 2022 | 93.9 | - | - | EN | Yang et al. |
| IRN | 2022 | 85.9 | - | - | EN | Wang et al. |
| Supervised embedding | 2017 | 54.4 | - | - | EN | Zhang et al. |
| T5+KG | 2022 | - | - | 71.47 | EN | Moiseev et al. |
| T5 | 2022 | - | - | 24.5 | EN | Moiseev et al. |
| T5+C4 | 2022 | - | - | 23.53 | EN | Moiseev et al. |
| Model / System | Year | Hits@1 | F1 | Exact Match | Language | Reported by |
|---|---|---|---|---|---|---|
| KGQA Based on Query Path Generation | 2022 | 99.9 | - | - | EN | Yang et al. |
| PullNet | 2019 | 99.9 | - | - | EN | Sun et al. |
| DCRN | 2022 | 99.9 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| NSM | 2021 | 99.9 | - | - | EN | He et al. |
| NSM+p | 2021 | 99.9 | - | - | EN | He et al. |
| NSM+h | 2021 | 99.9 | - | - | EN | He et al. |
| QAGCN | 2022 | 99.9 | - | - | EN | Wang et al. |
| QNRKGQA | 2022 | 99.9 | - | - | EN | Ma et al. |
| QNRKGQA+h | 2022 | 99.9 | - | - | EN | Ma et al. |
| SSKGQA | 2022 | 99.7 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| EmbedKGQA | 2020 | 98.8 | - | - | EN | Saxena et al. |
| TransferNet | 2022 | 98.5 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| NRQA | 2022 | 97.5 | - | - | EN | Guo et al. |
| Edge-aware GNN | 2022 | 96.8 | 93.7 | - | EN | Zhang et al. |
| GlobalGraph | 2022 | 95.5 | 83.0 | - | EN | Zhang et al. |
| 2HR-DR | 2022 | 93.7 | 81.4 | - | EN | Zhang et al. |
| SGReader | 2022 | 80.7 | 79.8 | - | EN | Zhang et al. |
| GraftNet | 2022 | 94.8 | 72.7 | - | EN | Zhang et al. |
| ReifKB+mask | 2020 | 95.4 | - | - | EN | Cohen et al. |
| SRN | 2020 | 95.1 | - | - | EN | Qiu et al. |
| ARN_ComplEx | 2023 | 94.92 | - | - | EN | Cui et al. |
| GraftNet | 2018 | 94.8 | - | - | EN | Sun et al. |
| AlAgha, 2022 | 2022 | 94.1 | - | - | EN | AlAgha, 2022 |
| ARN_ConvE | 2023 | 93.59 | - | - | EN | Cui et al. |
| ARN_TuckER | 2023 | 93.20 | - | - | EN | Cui et al. |
| MINERVA | 2022 | 92.9 | - | - | EN | Wang et al. |
| ARN_DistMult | 2023 | 92.54 | - | - | EN | Cui et al. |
| VRN | 2017 | 89.9 | - | - | EN | Zhang et al. |
| VRN | 2022 | 89.2 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| KV-MemNN | 2022 | 82.7 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| Borders et al. 's QA system | 2017 | 81.8 | - | - | EN | Zhang et al. |
| ReifKB | 2020 | 81.1 | - | - | EN | Cohen et al. |
| KV-MemNN | 2022 | 76.0 | - | - | EN | Zhang et al. |
| IRN | 2022 | 71.3 | - | - | EN | Wang et al. |
| Supervised embedding | 2017 | 29.1 | - | - | EN | Zhang et al. |
| KV-MemNN | 2017 | 25.1 | - | - | EN | Zhang et al. |
| T5+KG | 2022 | - | - | 33.57 | EN | Moiseev et al. |
| T5+C4 | 2022 | - | - | 32.78 | EN | Moiseev et al. |
| T5 | 2022 | - | - | 32.65 | EN | Moiseev et al. |
| Model / System | Year | Hits@1 | F1 | Exact Match | Language | Reported by |
|---|---|---|---|---|---|---|
| SSKGQA | 2022 | 99.6 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| DCRN | 2022 | 99.3 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| NSM | 2021 | 98.9 | - | - | EN | He et al. |
| NSM+p | 2021 | 98.9 | - | - | EN | He et al. |
| NSM+h | 2021 | 98.9 | - | - | EN | He et al. |
| QNRKGQA | 2022 | 98.9 | - | - | EN | Ma et al. |
| QNRKGQA+h | 2022 | 98.9 | - | - | EN | Ma et al. |
| KGQA Based on Query Path Generation | 2022 | 98.5 | - | - | EN | Yang et al. |
| QAGCN | 2022 | 97.6 | - | - | EN | Wang et al. |
| ARN_ConvE | 2023 | 97.06 | - | - | EN | Cui et al. |
| ARN_ComplEx | 2023 | 96.59 | - | - | EN | Cui et al. |
| Edge-aware GNN | 2022 | 96.3 | 91.0 | - | EN | Zhang et al. |
| NRQA | 2022 | 96.1 | - | - | EN | Guo et al. |
| ARN_TuckER | 2023 | 95.97 | - | - | EN | Cui et al. |
| EmbedKGQA | 2020 | 94.8 | - | - | EN | Saxena et al. |
| TransferNet | 2022 | 94.7 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| AlAgha, 2022 | 2022 | 93.4 | - | - | EN | AlAgha, 2022 |
| ARN_DistMult | 2023 | 93.13 | - | - | EN | Cui et al. |
| PullNet | 2022 | 91.4 | - | - | EN | Sun et al. |
| GlobalGraph | 2022 | 81.4 | 62.4 | - | EN | Zhang et al. |
| ReifKB+mask | 2020 | 79.7 | - | - | EN | Cohen et al. |
| GraftNet | 2022 | 77.8 | 56.1 | - | EN | Zhang et al. |
| GraftNet | 2022 | 77.7 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| GraftNet | 2018 | 77.2 | - | - | EN | Sun et al. |
| SRN | 2020 | 75.2 | - | - | EN | Qiu et al. |
| ReifKB | 2020 | 72.3 | - | - | EN | Cohen et al. |
| VRN | 2017 | 62.5 | - | - | EN | Zhang et al. |
| SGReader | 2022 | 61.0 | 58.0 | - | EN | Zhang et al. |
| MINERVA | 2022 | 55.2 | - | - | EN | Wang et al. |
| KV-MemNN | 2022 | 48.9 | - | - | EN | Mingchen Li and Jonathan Shihao Ji |
| IRN | 2022 | 35.6 | - | - | EN | Wang et al. |
| Supervised embedding | 2017 | 28.9 | - | - | EN | Zhang et al. |
| Borders et al. 's QA system | 2017 | 28.4 | - | - | EN | Zhang et al. |
| KV-MemNN | 2017 | 10.1 | - | - | EN | Zhang et al. |
| T5+KG | 2022 | - | - | 43.41 | EN | Moiseev et al. |
| T5 | 2022 | - | - | 42.31 | EN | Moiseev et al. |
| T5+C4 | 2022 | - | - | 39.66 | EN | Moiseev et al. |
[1] Zhang, Yuyu, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. Variational reasoning for question answering with knowledge graph. In Thirty-Second AAAI Conference on Artificial Intelligence. 2018.