Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kappa incorrect when changing number of reviewers #108

Open
esyriani opened this issue Mar 30, 2023 · 0 comments
Open

Kappa incorrect when changing number of reviewers #108

esyriani opened this issue Mar 30, 2023 · 0 comments
Labels

Comments

@esyriani
Copy link
Member

We are 6 reviewers. We imported 100 papers and screened them. Each paper is assigned all 6 reviewers. We got a value of Kappa. Then we resolved all conflicts.
Next, we imported 1816 papers and screened them. Each paper is assigned 2 reviewers. The Kappa is negative before resolving conflicts but we only have 8% of conflicts. See statistics figure at the end.

At the beginning, we had 2 screening phases: P1 and P2 (final). The first 100 papers were screened in P1. Then we deleted P2 and imported all remaining 1816 papers in P1. Perhaps this also had an effect on top of changing the number of reviewers per paper.

image

@esyriani esyriani added the bug label Mar 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Critical
Development

No branches or pull requests

1 participant