diff --git a/dbpedia/LC-QuAD v2.md b/dbpedia/LC-QuAD v2.md index f3ba7bb6..7b7a9192 100644 --- a/dbpedia/LC-QuAD v2.md +++ b/dbpedia/LC-QuAD v2.md @@ -3,33 +3,34 @@ name: LC-QuAD v2 datasetUrl: https://figshare.com/projects/LCQuAD_2_0/62270 --- -| Model / System | Year | Precision | Recall | F1 | Language | Reported by | Gold Entity | -| :-----------------------: | :--: | :-------: | :----: | :---: | :------: | :----------------------------------------------------------------------------------: | :---------: | -| T5-Small | 2022 | - | - | 92 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| T5-Base | 2022 | - | - | 91 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| PGN-BERT-BERT | 2022 | - | - | 86 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| SGPT_Q,K [1] | 2022 | - | - | 89.04 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ | -| PGN-BERT | 2022 | - | - | 77 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| NSpM [2] | 2022 | - | - | 66.47 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ | -| BART | 2022 | - | - | 64 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| Zou et al. + Bert | 2021 | - | - | 59.30 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | -| CLC | 2021 | - | - | 59 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| Multi-hop QGG | 2020 | - | - | 53 | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | -| Zou et al. + Tencent Word | 2021 | - | - | 52.90 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | -| Multi-hop QGG | 2021 | - | - | 52.60 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | -| AQG-net | 2021 | - | - | 44.90 | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | -| SGPT_Q [3] | 2022 | - | - | 83.45 | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ❌ | -| ChatGPT | 2023 | - | - | 42.76 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| GPT-3.5v3 | 2023 | - | - | 39.04 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| GPT-3.5v2 | 2023 | - | - | 33.77 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| GPT-3 | 2023 | - | - | 33.04 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| FLAN-T5 | 2023 | - | - | 30.14 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | -| UNIQORN | 2021 | 33.1 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| QAnswer | 2020 | 30.80 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| GraftNet | 2018 | 19.7 | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ | -| ElNeuQA-ConvS2S [1] | 2021 | 26.90 | 27 | 26.90 | EN | [Diomedi, Hogan](https://arxiv.org/pdf/2107.02865.pdf) | ❌ | -| GRAFT-Net + Clocq [2] | 2022 | 19.70 | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ | -| Platypus | 2018 | 3.6 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| Pullnet | 2019 | 1.1 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| UNIK-QA | 2020 | 0.5 | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | -| GETT-QA [4] | 2023 | 40.3 | - | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2303.13284.pdf) | ❌ | +| Model / System | Year | Precision | Recall | F1 | Hits@1 | Language | Reported by | Gold Entity | +| :-----------------------: | :--: | :-------: | :----: | :---: | :----: | :------: | :----------------------------------------------------------------------------------: | :---------: | +| T5-Small | 2022 | - | - | 92 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| T5-Base | 2022 | - | - | 91 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| PGN-BERT-BERT | 2022 | - | - | 86 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| SGPT_Q,K [1] | 2022 | - | - | 89.04 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ | +| PGN-BERT | 2022 | - | - | 77 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| NSpM [2] | 2022 | - | - | 66.47 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ✅ | +| BART | 2022 | - | - | 64 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| Zou et al. + Bert | 2021 | - | - | 59.30 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | +| CLC | 2021 | - | - | 59 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| Multi-hop QGG | 2020 | - | - | 53 | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2204.12793.pdf) | ✅ | +| Zou et al. + Tencent Word | 2021 | - | - | 52.90 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | +| Multi-hop QGG | 2021 | - | - | 52.60 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | +| AQG-net | 2021 | - | - | 44.90 | - | EN | [Zou et al.](https://arxiv.org/pdf/2111.06086.pdf) | ✅ | +| SGPT_Q [3] | 2022 | - | - | 83.45 | - | EN | [Al Hasan Rony et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9815253) | ❌ | +| W. Han et al. | 2023 | - | - | - | 42.6 | EN | [Han et al.](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | ❌ | +| ChatGPT | 2023 | - | - | 42.76 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| GPT-3.5v3 | 2023 | - | - | 39.04 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| GPT-3.5v2 | 2023 | - | - | 33.77 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| GPT-3 | 2023 | - | - | 33.04 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| FLAN-T5 | 2023 | - | - | 30.14 | - | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) | ❌ | +| UNIQORN | 2021 | 33.1 | - | - | 25.2 | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| QAnswer | 2020 | 30.80 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| GraftNet | 2018 | 19.7 | - | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ | +| ElNeuQA-ConvS2S [1] | 2021 | 26.90 | 27 | 26.90 | - | EN | [Diomedi, Hogan](https://arxiv.org/pdf/2107.02865.pdf) | ❌ | +| GRAFT-Net + Clocq [2] | 2022 | 19.70 | - | - | - | EN | [Christmann P. et al](https://arxiv.org/pdf/2108.08597.pdf) | ❌ | +| Platypus | 2018 | 3.6 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| Pullnet | 2019 | 1.1 | - | - | 11.9 | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| UNIK-QA | 2020 | 0.5 | - | - | - | EN | [Pramanik et al.](https://arxiv.org/abs/2108.08614) | ❌ | +| GETT-QA [4] | 2023 | 40.3 | - | - | - | EN | [Banerjee et al.](https://arxiv.org/pdf/2303.13284.pdf) | ❌ | diff --git a/freebase/WebQuestionsSP.md b/freebase/WebQuestionsSP.md index 62f3b19f..5bce8051 100644 --- a/freebase/WebQuestionsSP.md +++ b/freebase/WebQuestionsSP.md @@ -47,6 +47,7 @@ | KGQA-CL(XLnet) | 2023 | - | - | 61.46 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | | KGQA-CL(DistilRoberta) | 2023 | - | - | 61.05 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | | KGQA-CL(GPT2) | 2023 | - | - | 60.49 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) | +| W. Han et al. | 2023 | - | 75.2 | - | EN | [Han et al.](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | | NSM | 2021 | - | 74.30 | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) | | Rigel | 2022 | - | 73.3 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) | | SGM | 2022 | 72.36 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) | diff --git a/systems.md b/systems.md index 39eed033..5e7446af 100644 --- a/systems.md +++ b/systems.md @@ -139,3 +139,4 @@ | TIARA | Shu et al. | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | yes | [Link](https://github.com/microsoft/KC/tree/main/papers/TIARA) | [Link](https://aclanthology.org/2022.emnlp-main.555.pdf) | In this paper, we present a new KBQA model, TIARA, which addresses those issues by applying multi-grained retrieval to help the PLM focus on the most relevant KB contexts, viz., entities, exemplary logical forms, and schema items. Moreover, constrained decoding is used to control the output space and reduce generation errors. | Shu et al. | | MACRE | Xu et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_40) | MACRE is a novel approach for multi-hop question answering over KGs via contrastive relation embedding (MACRE) powered by contrastive relation embedding and context-aware relation ranking. | Xu et al. | | KGQAcl/rr | Hu et al. | [Link](https://arxiv.org/pdf/2303.10368.pdf) | yes | [Link](https://github.com/HuuuNan/PLMs-in-Practical-KBQA) | [Link](https://arxiv.org/pdf/2303.10368.pdf) | KGQA-CL and KGQA-RR are tow frameworks proposed to evaluate the PLM's performance in comparison to their efficiency. Both architectures are composed of mention detection, entity disambiguiation, relation detection and answer query building. The difference lies on the relation detection module. KGQA-CL aims to map question intent to KG relations. While KGQA-RR ranks the related relations to retrieve the answer entity. Both frameworks are tested on common PLM, distilled PLMs and knowledge-enhanced PLMs and achieve high performance on three benchmarks. | Hu et al. | +| W. Han et al. | Han et al. | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | no | - | [Link](https://link.springer.com/chapter/10.1007/978-3-031-30672-3_39) | This model is based on machine reading comprehension. To transform a subgraph of the KG centered on the topic entity into text, the subgraph is sketched through a carefully designed schema tree, which facilitates the retrieval of multiple semantically-equivalent answer entities. Then, the promising paragraphs containing answers are picked by a contrastive learning module. Finally, the answer entities are delivered based on the answer span that is detected by the machine reading comprehension module. | Han et al. | \ No newline at end of file