Selecting tool for annotation exercise

[dennyabrain]

We have some experience building a custom annotation tool for annotating multimedia data. Given that we'll be scaling up to a lot more annotators this time and given a time crunch, we were wondering if there's some off the shelf tool that we could use. This discussion mainly took place on Slack, but i am porting it over here because I think this discussion would be useful to link in the future as well.

[dennyabrain]

@whymath said

Not sure if you meant for text or images, but I'm assuming text based on your post above; unfortunately text annotation tools are also generally less well-known that those for Computer Vision.
To start with, I have to mention that I haven't actually used any of the below, so Swair might be able to give much better feedback here; however, that said, here's what I came across:

1. doccano seems to be a somewhat popular free tool, though it does have limitations in what all types of annotations it can handle - only 1) text classification, 2) sequence labeling and 3) sequence-to-sequence.
2. Prodigy is an example of a commercial platform by the Spacy folks themselves. In fact, there's a tweet they've highlighted by a UK-based fact-checking org where they used Prodigy to manage annotations across 90 people, which seemed quite close to our own use case. Note that they themselves say Prodigy isn't yet optimized for teams and that they're working on an enhancement called Prodigy Teams, but for small teams I'm guessing that might not be a big deal.
3. Hivemind is another one that I came across that seemed interesting with what looks like a slick UI and some project-level management features, but not sure about any real-world reviews. However, one of the clients they mentioned on their website, Auquan, is a friend's company, so I might be able to check with them for feedback if required.
   In case y'all meant for images, then there's quite a few popular ones - VGG Image Annotator, Diffgram, and Intel's CVAT are FOSS examples of such libraries.

[dennyabrain]

@arnav on slack said

Off the shelf tools :
Prodigy is the least mental effort and best overall option but it is not cheap.
Doccano is an open source alternative I’ve used in the past. It’s a little more effort but is actively maintained and works decently well.
Snorkel is one with lots of additional features (though I’ve never used it), unnecessary for our use case but if they offer it for free (will need to reach out), I’m sure the annotation interface must be very easy as well.
Inception, Brat are ones that came up in a brief search and look okay to me.
ipyannotate - simple jupyter widget
Most of these are full fledged annotation tools. If we want something just to slice the rows and keep track of what’s assigned to whom, I don’t know of scripts that already do that. But I can’t imagine it being very hard to write a small script for it for our custom use case. Shouldn’t take more than a couple of hours. I’ll be a bit busy this week but if this can wait, happy to do it sometime late next week.

[dennyabrain]

I made a checklist for the annotation task requirement to vet every tool against.

MUST HAVES

• can add multiple annotators
• can support data of both text and image type
• can define a custom annotation form/task

NICE TO HAVES

• can self-host
• can assign to users automatically satisfying certain parameters

[dennyabrain]

One way in which our requirement are slightly diff from what some of these tools allow is that for every post, we are seeking 4,5 annotations (they can be binary fields or multiple choice selections). so for example under every post, annotators will see questions like
is this Online Gender Based Violence - Yes, No
Is this Explicit - Yes, No
and there is a possibility of having multiple choice questions in this too.

What i am seeing in docanno is that one can only do certain standard annotation tasks like - Text Classification, Sequence Labelling, Sequence to Sequence, Image Classification, Speech to text. It does not allow customization of the annotation task itself. so for that reason i am striking that out. lmk if i am wrong

[dennyabrain]

i see two contradictory things about prodigy personal licence.

on this page - https://prodi.gy/buy, it says personal licence has "unlimited annotators"
and on this page - https://prodi.gy/docs, it says "Prodigy doesn’t provide a lot of annotation-adjacent functionality, such as user accounts or project management. We designed Prodigy to be the best tool possible for small teams working on rapidly evolving projects. We’re also working on a second product, Prodigy Teams, which will help organizations keep all their annotation work in one place, and make it easier to collaborate on larger-scale projects."

even on the interface or docs i don't see how one can add an annotator. when one launches prodigy, it becomes available on the web browser and there's no login flow. Then i read this blog post of Full Fact collaborating with Prodigy for their needs - https://fullfact.org/blog/2018/feb/how-we-customised-prodigy-ai/ in there it says "At the moment, we have 90 volunteers live with the corresponding 90 Prodigy instances which altogether adds up to 5.5 GB of memory."
basically running on instance of prodigy per annotator!!! I wanted to run this by you to see if i am missing something obvious.
4:53
so far the prodigy checklist looks like this

• can add multiple annotators
• can support data of both text and image type (haven't tried this but theoretically should be possible)
• can define a custom annotation form/task
• can self-host
• can assign to users automatically satisfying certain parameters