New duplicate algorithm to check for similar entries

[george-gca]

I added the option to check for duplicate entries based on the similarity of the title and abstract. Sometimes we can have a duplicate entry that is a fixed version of another entry, with a corrected typo or added comma, for example.

I decided to go with difflib.SequenceMatcher for this similarity, since it is a built-in solution. Added the options to use only the title for this check, set the similarity score, discard stopwords for a more strict check considering only the useful words, and also added a pretty diff print support thanks to the rich library:

[PeterLombaers]

Thanks for this contribution, it look really nice! It makes a lot of sense to me to want to deduplicate using some fuzzy matching. The code looks clean! I was testing out the features and it seems to be working well. Just a few comments, which I'll put in the comments below.

[PeterLombaers]

When I was testing with --verbose, I got confused when I did not see anything pretty printing. I think this happened because my duplicated were already dropped by the original dedup method. When I added rows that were dropped only by your method, I indeed so the nice diff. If I'm using --verbose I expect to get feedback no matter which method does the deduplicating.

[PeterLombaers]

But thanks again, it looks very nice! We'll also need to have a chat about how this relates to asreview2.0 @J535D165 !

[george-gca]

What do you think should be done when verbose is true for the current algorithm? I mean, what would be the expected output? Because the pretty print probably will print everything dimmed in most cases.

[Rensvandeschoot]

Great contribution!! I have recently used fuzzy matching for our FORAS project, where we obtained over 10K records to screen. I checked for duplicates within the dataset and between the titles obtained via the database search and the most likely title match in OpenAlex. I saved the matching score and went through the data, checking these scores from low to high, and I found many fuzzy duplicates of the following type:

• Titles containing extra or different punctuation.
• One title has a spelling mistake corrected in the other.
• Presence of HTML code in one title (e.g., PTSD), but not in the other.
• An abstract number at the beginning of one title (e.g., “T262.”), missing from the other.
• A subtitle in one record versus a single-title format in the other.

All such cases are precise duplicates and can be corrected without losing any information.

But I also found cases with different versions of the same work:

• re-prints of the same paper in a different journal,
• pre-print + journal version,
• conference abstract + journal version,
• book chapter + journal version,
• dissertation + journal version,
• version 1 + version 2 of the same paper

You might want to keep both records in these cases, but the labeling decisions will be the same.

So, my question is whether it is possible to store the matching score so that a user can manually check the records with lower matching scores?

and, hopefully, my comment helps with starting the discussion on what to do with fuzzy-duplicates in ASReview v2.0 :-)

[george-gca]

Currently you can choose to print the duplicates on the terminal instead of already removing them by avoiding the -o option. It will print the line of the entries and a pretty diff between them, but it will print this in order of finding, not based on the score. This could be added as an option, for instance, or added as a column to the dataset like you said. What option would be best? Also, adding this column to the dataset wouldn't somehow affect the usage of the dataset with ASReview, or does it simply ignore the extra columns?

This should be enough to match most of the cases you pointed here just by playing with the threshold param, since I do some cleaning before doing the actual match. I've also added an option to remove stopwords before checking, to create even more similar titles. The cases that could fall short would be:

• presence/abscence of HTML or latex code, if it is too long
• added subtitle to the title

The added subtitle might be a pitfall, since some papers build on top of others, including the title, being completely different papers with similar titles. And sometimes they like to follow a trend of titles, like the case for x is all you need.

It would be actually a great test for this code the project you mentioned.

[PeterLombaers]

What do you think should be done when verbose is true for the current algorithm? I mean, what would be the expected output? Because the pretty print probably will print everything dimmed in most cases.

True, I'm not sure what I would want to see in verbose mode for the other case. I would leave it as it is for now. We might to think in the future about wether we want to maintain this feature and how it would look then, also for the other deduplication. I do think it's very nice to have such verbose output when deduplicating, so that you can clearly see which ones get marked as duplicate and why.

[george-gca]

I agree with you. I had a quick look at the ASReviewData.drop_duplicates code, and it resets index by default, meaning that comparing dataframes before and after the deduplication would fail.

I changed the code a little bit to allow verbose when not using similar, also moved part of the code to dedup.py since it was starting to be too much code in the entrypoint for that.

[PeterLombaers]

I like the changes! I just have a few comments, but other than that I'm all for merging :)

[george-gca]

One question, is it really necessary to enforce the maximum line length to be 88? I mean, this line length is too small even for someone working on a laptop screen.

For instance, line 124 has 88 columns, and it doesn't even reach half of my laptop screen (it has 1920x1080 resolution). I feel like some print statements in this code would be more readable if this line length increased by a little, and it wouldn't hurt the coding experience.

[PeterLombaers]

I kinda like it, I can put two windows next to each other on my laptop.

[george-gca]

I usually use line wrap for this. Also, I believe using half monitor code is more of an exception rather than the regular use case. And at least for my case I have access to external bigger monitors, so it is kind of a waste of space. I am not saying that it should not have a limit, rather it could be a little bigger (like idk 119?).

[PeterLombaers]

This is my usual setup, which actually fits on my laptop, so that's why I like it. But feel free to file an issue on the asreview repo and see if you can convince @J535D165 😄.

[george-gca]

Even in your setup the lines trespass the 88 columns threshold for half monitor and you need to scroll right to see the rest of it, so a line wrap is more fitting I think. But that is just personal opinion.

[george-gca]

@PeterLombaers do you have any dataset with DOIs that I can use to test the new solution? The change is rather annoying if I am to consider pids for the similar case, like this loop has to change:

for j, t in s.iloc[i+1:][abs(s.str.len() - len(text)) < 5].items():

[george-gca]

I believe now it is finished.

[PeterLombaers]

Sorry, I feel like there was some miscommunication! In your last commit you implement deduplicating based on PID, but ASReviewData.drop_duplicates already does this. So having a second implementation will only confuse matters.

How about the following plan to keep things simple:

• Revert the last commit
• If --similar, we use your method
• Else we ASReviewData.drop_duplicates, so simply how it is in the current version of datatools
• If you want to use both, you have to call asreview data dedup twice, once with --similar and once without
• Any extra arguments like --verbose and --title_only, etc. will only apply in case of the --similar flag
• in the help text of the arguments it says whether the argument only applies for --similar
• We add a test to check that the different options give the correct deduplication

This means that the behaviour with --similar and without is not quite the same. For example, if you don't use --similar, you always use both title and abstract for deduplicating, but if you use --similar, you can select --title_only. But I'm fine with that difference for now, lets get you PR into shape to merge, then we can always look later at streamlining the datatools API as a whole.

If you do the first five points on that list, I can help by writing tests, adding the help texts and linting the code.

Do you agree with this plan?

[PeterLombaers]

BTW, When checking out your pull request, I made the following csv file to test the different settings:

abstract, title, doi, publication_year
abstract, <h2>this is a title of a paper<h2>, doi1, 2005
abstract, this is a title of a paper, doi1, 2001
abstract, this is a title of a paper&, doi3,
abstract, this is the title of a paper, doi4, 2003
abstract, this is a title%^ of a paper, doi5, 2004
abstract, this is a title of a pãper, doi6, 2000

Might be useful for running sanity checks that everything is working as expected.

[george-gca]

If you do the first five points on that list, I can help by writing tests, adding the help texts and linting the code.

Do you agree with this plan?

Sorry, I disagree. I agree with what you previously said (or at least that's what I understood), that it is best to have a streamlined functionality that looks like one single tool, not like two separate tools were glued together. Also one should not need to run once without --similar and once with --similar, since an exact copy is also a similar considering the string difference algorithm.

I have another proposition: make all output look like the version with --similar flag. The difference is that without the similar flag we will look for exact matches kinda like it is currently done, only making a few changes so I can find which one is the original and which is the duplicate, and keep it how it is for the --similar case. I won't suggest always using the same algorithm because without the --similar is way faster.

What I am thinking now for the case without --similar is:

• remove duplicates, compare sizes before and after removal
• size changed (meaning some duplicates were removed), check again for duplicates in the original data (now with keep=False so we flag all data with copies, not only the duplicates) and create a df with only the copies
• use the same algorithm used in the --similar case, but only for this subset of the data that have/are copies. This way we can find from which exactly row that duplicate is a duplicate of, and running in a potentially smaller subset will be faster
• use the same print as for the similar case

What do you think? Also, help for the tests and linting would be great.

[PeterLombaers]

The reason I suggest keeping it split, is that in the near future (a few months), ASReview version 2 will be released. The ASReviewData object will change and how we deal with duplicates inside ASReview might also change. The current deduplicate without --similar makes use of ASReviewData.drop_duplicates, so if we change that now we will have to do the same work again for ASReview v2.

If we keep the two methods (with and without --similar) separate for now, we can unify them once v2 lands. The --similar method can be completely inside asreview-datatools, and I don't mind making a new release for the updated dedup method for datatools. But I don't think we want any more releases for ASReview v1 that are not bugfixes.

[george-gca]

That makes sense. So what do you think I should do? Wait for V2?

[PeterLombaers]

I leave the choice up to you. We can add it to v1 with the plan I wrote above, or we leave this pull request open for now and then integrate your deduplication algorithm in the v2 deduplication. Then we'll be careful that you get the correct recognition as a contributer when we implement the deduplication in v2.

[george-gca]

I can keep them separate for now and get the PR accepted so some users (including me) might benefit from it. Then, when V2 launches I can get back to it and port the solution. What do you think?