Expanded correct queries for 3 questions + minor prompt/typo fixes

[rishsriv]

The 3 questions fixed were:

What is the total number of citations received by each author?
Previously, this only relied on sum(publication.citation_num) for the correct answer. Now, COUNT(c.cited) is also accepted as a correct answer

Subtract 2 weeks from the most recent predicted graduation date and give the month.
Allow for more date formats

Which papers have the highest number of authors, ordered by the number of authors in descending order?
Allow answers that include both the paper id and paper title, instead of just one

[wendy-aw]

should we expand it to:
SELECT {author.name, author.aid}, COUNT(c.cited) AS total_citations FROM author a JOIN writes w ON a.aid = w.aid JOIN publication p ON w.pid = p.pid JOIN cite c ON p.pid = c.cited GROUP BY {} ORDER BY total_citations DESC; to include author.aid too?

[rishsriv]

will do, thank you!

[wendy-aw]

Thanks for consolidating. But I think should leave the writes.paperid SQL cos they produce different dataframes:

SELECT writes.paperid, COUNT(writes.authorid) AS num_authors FROM writes GROUP BY writes.paperid ORDER BY num_authors DESC NULLS LAST;
 paperid | num_authors 
---------+-------------
       4 |           3
       2 |           3
       1 |           3
       5 |           2
       3 |           1

SELECT paper.paperid, COUNT(DISTINCT writes.authorid) AS num_authors FROM paper JOIN writes ON paper.paperid = writes.paperid GROUP BY paper.paperid ORDER BY num_authors DESC;
 paperid | num_authors 
---------+-------------
       1 |           3
       2 |           3
       4 |           3
       5 |           2
       3 |           1

[rishsriv]

It's the same results – just ordered differently! :) The new ordering logic in sql-eval takes care of the ordering when doing the eval

[wendy-aw]

Oh we already fixed that? Ok my bad pls ignore! Then I think we might have a few other redundant queries lying around :P

[wendy-aw]

Thanks for the fixes! Just two small comments from me

[wongjingping]

LGTM, thank you for the detailed review and improvements to make the benchmark more robust and informative!

[wongjingping]

shall we also allow the COUNT(c.cited) to incorporate either permutations of a.name/a.aid? It would look something like:

SELECT {a.name, a.aid}, COUNT(c.cited) AS total_citations FROM author a JOIN writes w ON a.aid = w.aid JOIN publication p ON w.pid = p.pid JOIN cite c ON p.pid = c.c
ited GROUP BY {} ORDER BY total_citations DESC;

[rishsriv]

Yup agreed, adding a fix for this!

[wongjingping]

good catch!

[wongjingping]

As there is a tie here in the top 3 papers, I'll add some more data to defog-data to ensure that we break the ties via the data since we only need to verify the ordering by number of authors and not the paper id/name:

 paperid |                    title                     | num_authors 
---------+----------------------------------------------+-------------
       1 | A Study on Machine Learning Algorithms       |           3
       2 | The Effects of Climate Change on Agriculture |           3
       4 | COVID-19 Impact on Society                   |           3
       5 | Machine Learning in Tackling Climate Change  |           2
       3 | Social Media and Mental Health               |           1

[rishsriv]

Good point, though the eval logic now takes "tie-breaks" into account for order by questions

[rishsriv]

New logic here! I think we should be good to go without changes to defog-data

sql-eval/eval/eval.py

Line 42 in b8b7704

if order_by_clause: