Skip to content

Commit

Permalink
Merge pull request #266 from DavidBlavid/v2.0
Browse files Browse the repository at this point in the history 
Solve issues #236, #239, #240, #241 and #242
  • Loading branch information
@xixi019
xixi019 authored Jun 24, 2024
2 parents 6f15a71 + d73107c commit 8ddf409
Show file tree
Hide file tree
Showing 9 changed files with 79 additions and 1 deletion.
1 change: 1 addition & 0 deletions freebase/ComplexQuestions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

| Model / System | Year | Average F1 | Reported by |
|:-------------------:|:---------:|:----------:|:----------------------------------------------------------------------------:|
| TFS-KBQA | 2024 | 44.0 | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf)|
| Lan and Jiang [1] | 2020 | 43.3 | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) |
| Reranking | 2022 | 42.9 | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) |
| Luo et al. [2] | 2020 | 42.8 | [Yonghui Jia and Wenliang Chen](https://arxiv.org/pdf/2204.12808.pdf) |
Expand Down
12 changes: 12 additions & 0 deletions freebase/ComplexWebQuestions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,30 @@
| GPT-3 | 2023 | - | - | 51.77 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) |
| FLAN-T5 | 2023 | - | - | 46.69 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) |
| BART-large | 2022 | 68.2 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) |
| FiDeLiS + gpt-4-turbo | 2024 | 64.32| 71.47 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| TFS-KBQA | 2024 | 63.6 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) |
| DECAF (BM25 + FiD-3B) | 2022 | - | 70.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| CBR-KBQA | 2022 | - | 70.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| DECAF (BM25 + FiD-large) | 2022 | - | 68.1 ± 0.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| DECAF (DPR + FiD-large) | 2022 | - | 67.0 ± 0.4 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| FiDeLiS + gpt-3.5-turbo | 2024 | 61.78| 63.12 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| ToG + gpt-4-turbo | 2024 | 60.2 | 68.51 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| HGNet | 2022 | - | 58.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| ProgramTransfer | 2022 | 58.7 | 58.1 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) |
| ToG + gpt-3.5-turbo | 2024 | 56.96| 57.59 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| RoG | 2024 | 56.17| 61.39 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| ProgramTransfer-o | 2022 | 55.8 | 54.7 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) |
| ProgramTransfer-pa | 2022 | 54.5 | 54.3 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) |
| KD-CoT | 2024 | - | 53.92 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| NSM+h | 2022 | - | 53.9 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| FC-KBQA | 2024 | 54.1 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) |
| REAREV | 2022 | - | 52.9 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) |
| QNRKGQA+h | 2022 | - | 51.5 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
| SR+NSM | 2022 | - | 50.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| QNRKGQA | 2022 | - | 50.5 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
| RCAANet | 2024 | - | 50.5 | - | EN | [Lv et al.](https://www.researchsquare.com/article/rs-4292912/v1) |
| RE-KBQA | 2024 | - | 50.3 | - | EN | [Lv et al.](https://www.researchsquare.com/article/rs-4292912/v1) |
| DGEN | 2024 | - | 49.5 | - | EN | [Lv et al.](https://www.researchsquare.com/article/rs-4292912/v1) |
| NSM-distill | 2022 | - | 48.8 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) |
| Rigel | 2022 | - | 48.7 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) |
| TransferNet | 2022 | - | 48.6 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) |
Expand All @@ -36,6 +47,7 @@
| TERP | 2022 | - | 49.2 | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) |
| TeacherNet | 2022 | 44.0 | 48.8 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) |
| NSM | 2022 | 44.0 | 48.8 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) |
| RnG-KBQA | 2024 | 42.3 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) |
| NSM+h | 2022 | - | 48.8 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
| TransferNet | 2022 | - | 48.6 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) |
| NSM+p | 2022 | - | 48.3 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
Expand Down
13 changes: 13 additions & 0 deletions freebase/WebQuestionsSP.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,25 @@
| Model / System | Year | F1 | Hits@1 | Accuracy | Language | Reported by |
| :---------------------------------: | :--: | :--------: | :--------: | :------: | :------: | :-----------------------------------------------------------------------------------: |
| chatGPT | 2023 | - | - | 83.70 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) |
| TFS-KBQA | 2024 | 79.9 | 79.8 | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) |
| Pangu | 2024 | 79.6 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) |
| TIARA | 2022 | 78.9 | 75.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) |
| DECAF (DPR + FiD-3B) | 2022 | 78.8 | 82.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| FiDeLiS + gpt-4-turbo | 2024 | 78.32 | 84.39 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| GPT-3.5v3 | 2023 | - | - | 79.60 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) |
| DECAF (DPR + FiD-large) | 2022 | 77.1 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| FC-KBQA | 2024 | 76.9 | - | - | EN | [Shouhui Wang and Biao Qin](https://aclanthology.org/2024.lrec-main.1074.pdf) |
| FiDeLiS + gpt-3.5-turbo | 2024 | 76.78 | 79.32 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| UniK-QA | 2022 | - | 79.1 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| Sun et al. | 2024 | - | 77.8 | - | EN | [Sun et al.](https://aclanthology.org/2024.lrec-main.496.pdf) |
| TERP | 2022 | - | 76.8 | - | EN | [Qiao et al.](https://aclanthology.org/2022.coling-1.156.pdf) |
| SQALER+GNN | 2022 | - | 76.1 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) |
| EmQL | 2020 | - | 75.5 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| GMT-KBQA | 2022 | 76.6 | - | 73.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) |
| GPT-3.5v2 | 2023 | - | - | 72.34 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) |
| Program Transfer | 2022 | 76.5 | 74.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| RnG-KBQA (T5-large) | 2022 | 76.2 ± 0.2 | 80.7 ± 0.2 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| ToG + gpt-4-turbo | 2024 | 75.97 | 81.84 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| RnG-KBQA | 2022 | 75.6 | - | 71.1 | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) |
| ArcaneQA | 2022 | 75.3 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| QNRKGQA+h | 2022 | - | 75.7 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
Expand All @@ -26,6 +33,7 @@
| QNRKGQA | 2022 | - | 74.9 | - | EN | [Ma et al.](https://link.springer.com/chapter/10.1007/978-3-031-10983-6_11) |
| ReTrack | 2022 | 74.7 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) |
| ReTrack | 2021 | 74.6 | 74.7 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| DGEN | 2024 | - | 74.5 | - | EN | [Lv et al.](https://www.researchsquare.com/article/rs-4292912/v1) |
| BART-large | 2022 | 74.6 | - | - | EN | [Hu et al.](https://aclanthology.org/2022.coling-1.145.pdf) |
| Subgraph Retrieval | 2022 | 74.5 | 83.2 | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) |
| QGG | 2022 | 74.0 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
Expand All @@ -52,6 +60,8 @@
| NSM | 2021 | - | 74.30 | - | EN | [He et al.](https://arxiv.org/pdf/2101.03737.pdf) |
| Rigel | 2022 | - | 73.3 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) |
| SGM | 2022 | 72.36 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) |
| ToG + gpt-3.5-turbo | 2024 | 72.32 | 75.13 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| RCAANet | 2024 | 72.3 | 76.4 | - | EN | [Lv et al.](https://www.researchsquare.com/article/rs-4292912/v1) |
| CBR-SUBG | 2022 | 72.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| NPI | 2022 | - | 72.6 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) |
| TextRay | 2022 | - | 72.2 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) |
Expand All @@ -66,8 +76,10 @@
| HGNet | 2021 | 70.3 | 70.6 | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| GrailQA Ranking | 2021 | 70.0 | - | - | EN | [Shu et. al.](https://aclanthology.org/2022.emnlp-main.555.pdf) |
| SQALER | 2022 | - | 70.6 | - | EN | [Costas Mavromatis and George Karypis](https://arxiv.org/pdf/2210.13650.pdf) |
| RoG | 2024 | 69.81 | 83.15 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| STAGG | 2015 | 69.00 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) |
| UHop | 2019 | 68.5 | - | - | EN | [Ma L et al.](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9747229) |
| RE-KBQA | 2024 | 68.5 | 74.6 | - | EN | [Lv et al.](https://www.researchsquare.com/article/rs-4292912/v1) |
| KBIGER | 2022 | 68.4 | 75.3 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) |
| NSM | 2022 | - | 69.0 | - | EN | [Cao et al.](https://aclanthology.org/2022.acl-long.559.pdf) |
| GraftNet-EF+LF | 2018 | - | 68.7 | - | EN | [Sun et al.](https://aclanthology.org/D18-1455.pdf) |
Expand Down Expand Up @@ -96,6 +108,7 @@
| FLAN-T5 | 2023 | - | - | 59.87 | EN | [Tan et al.](https://arxiv.org/pdf/2303.07992.pdf) |
| KGT5 | 2022 | 56.1 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| FILM | 2022 | 54.7 | - | - | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| KD-CoT | 2024 | 50.2 | 73.7 | - | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| ReifKB | 2020 | - | 52.7 | - | EN | [Cohen et al.](https://arxiv.org/pdf/2002.06115.pdf) |
| KV-Mem | 2022 | 38.6 | 46.7 | - | EN | [Du et al.](https://arxiv.org/pdf/2209.03005.pdf) |
| KGQA-RR(GPT2) | 2023 | - | - | 18.11 | EN | [Hu et al.](https://arxiv.org/pdf/2303.10368.pdf) |
1 change: 1 addition & 0 deletions freebase/freebaseQA.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

| Model / System | Year | Exact Match | Accuracy | Hits@1 | Language | Reported by |
| :----------------------: | :--: | :---------: | :------: | :-------: | :------: | :----------------------------------------------------: |
| Sun et al. | 2024 | - | - | 84.3 | EN | [Sun et al.](https://aclanthology.org/2024.lrec-main.496.pdf) |
| DECAF (DPR + FiD-large) | 2022 | - | - | 79.0±0.6 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| DECAF (BM25 + FiD-large) | 2022 | - | - | 78.8± 0.5 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
| FILM | 2022 | - | - | 63.3 | EN | [Yu et al.](https://arxiv.org/pdf/2210.00063.pdf) |
Expand Down
1 change: 1 addition & 0 deletions frontend/static/entries.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
/datasets/other/
/datasets/other/COVID-QA
/datasets/other/CrunchQA
/datasets/other/CR-LT-KGQA
/datasets/other/DBLP-QuAD
/datasets/other/Event-QA
/datasets/other/MetaQA - 1 Hop
Expand Down
18 changes: 18 additions & 0 deletions other/CR-LT-KGQA.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
name: CR-LT-KGQA
datasetUrl: https://github.com/D3Mlab/cr-lt-kgqa
---

| Model / System | Year | Accuracy | Language | Reported by |
|:-----------------------------------:|:----:|:--------:|:--------:|:----------------------------------------------------------------------------------:|
| gpt-3.5-turbo (Zero-shot) | 2024 | 32.74 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-3.5-turbo (Few-shot) | 2024 | 36.61 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-3.5-turbo (Chain of Thought) | 2024 | 37.42 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-4-turbo (Zero-shot) | 2024 | 37.74 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-4-turbo (Few-shot) | 2024 | 43.61 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-4-turbo (Chain of Thought) | 2024 | 45.42 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| Reasoning on Graphs (RoG) | 2024 | 60.32 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-3.5-turbo + ToG | 2024 | 62.48 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-3.5-turbo + FiDeLiS | 2024 | 67.34 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-4-turbo + ToG | 2024 | 67.24 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
| gpt-4-turbo + FiDeLiS | 2024 | 72.12 | EN | [Sui et al.](https://arxiv.org/pdf/2405.13873) |
Loading

0 comments on commit 8ddf409

Please sign in to comment.