Handle completed assignments that may not have submitted properly

[ejolly]

Often times trying to access any options from the Assignments-Completed menu (i.e. Refresh Assignment States, Approve All Assignments or Pay All Bonuses) do not work and return:
Error: GetAssignment: This operation can be called with a status of: Reviewable,Approved,Rejected (1448302123263) [500]

This happens despite other Mturk API queries made by TurkServer working just fine (e.g. checking account balance, creating new HITs, adding assignments, etc). Restarting the meteor app+Turkserver, or the hosting server doesn't seem to work. Functionality only seems to return randomly after some time has passed not using TurkServer, or if the mongodb is wiped and the entire bundled app redeployed to a server with a clean database.

[mizzao]

I'm pretty sure this error is caused by a single assignment that is in a weird state. You will see the list of assignments update down the page as you click any of those buttons and the update will stop on the problem one.

I think this happens because you try to access assignments that were either returned (and not submitted), or submitted on TurkServer but not on MTurk (due to some technical error that can usually be fixed by the user reloading the page). What it means by 'Reviewable, Approved, Rejected' means the assignment has to be recorded as complete on MTurk.

Do you notice that someone e-mails you about the HIT not submitting correctly? Or perhaps someone returned the HIT after submitting it on TurkServer but the submission didn't go through? Can you locate find the assignment that is causing this and paste in what the database records for it?

Thanks,
Andrew

[ejolly]

Ah I see. Yup I just found one that Turkserver believes was submitted but is not visible on Amazon. The user did not email me about this though:

{
    "_id" : "nmAY9ThG6Jigy4AEr",
    "batchId" : "Yw3D2aitXs5atPGqg",
    "hitId" : "3T6SSHJUZF4G4ICTAQBD64MJ254IIV",
    "assignmentId" : "3KGTPGBS6YFJP8RL56K0RBZXYMN2U5",
    "workerId" : "AJMXUF6G3KDLP",
    "acceptTime" : ISODate("2015-11-20T22:14:55.302Z"),
    "status" : "completed",
    "ipAddr" : [
        "127.0.0.1"
    ],
    "userAgent" : [
        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0"
    ],
    "submitTime" : ISODate("2015-11-20T22:39:34.285Z"),
    "exitdata" : {
        "Feedback" : "AJMXUF6G3KDLP  mturk ID"
    }
}

So do you suggest removing or changing the status of this record and then trying to use the admin interface again?

[mizzao]

You could e-mail the user and have them make sure to submit the HIT. The code should automatically submit the HIT for them when they reload the page, since it is completed but somehow the submission process broke at the end.

If you don't want to do that or the assignment has already expired, you could also set status: "assigned" on this document which would allow things to proceed as normal. But then the user may not be able to submit when they come back.

You could also change the code in TurkServer to not give up when an assignment of this state is recorded. In general you should feel free to fork the code and modify it; I'd be happy to review a PR.

[ejolly]

Given the that the assignment already expired and the user isn't coming back changing the status within the database worked great.

I'll also look into modifying TurkServer for this.

[mizzao]

@josephcc ran into this issue as well. One approach would be to handle the Error: GetAssignment: This operation can be called with a status of: Reviewable,Approved,Rejected more gracefully, and continue trying to refresh statuses when it is not available.

#81 should help with reducing the occurrence of this as well.

[josephcc]

I'm thinking at least skipping the assignments with this issue so we can still pay the rest of the assignments in the webmin. Any thoughts on where in the codebase I should look at?

[mizzao]

Yes, you should add a new condition inside this catch block here:

https://github.com/TurkServer/turkserver-meteor/blob/master/server/assignment.js#L326

Basically, if you get that message, just return instead of throwing that 500 error, which will skip trying to refresh the assignment.

[josephcc]

hmm, but then the front-end wouldn't be notified that some assignments went wrong. what about catching the 500 where this was called here?

turkserver-meteor/admin/admin.coffee

Line 432 in 586ec64

asst.refreshStatus()

[mizzao]

Yeah, that would be even better!

I think it would be reasonable to throw the 500 to the client here (

turkserver-meteor/admin/admin.coffee

Line 436 in 586ec64

"ts-admin-refresh-assignment": (asstId) ->

), but catch it and keep going in the function you linked.

I think you would only want to catch the 500 error related to the HIT not being submitted yet. There are many other reasons why that API call might throw, and you probably don't want to treat them all the same. Unfortunately we can't seem to be able to get error codes directly using https://github.com/JoseBarrios/mturk-api, so pattern matching on strings seems best for now.

[josephcc]

Partial solution here: josephcc@49551e6

[mizzao]

By the way, #28 is the root cause of this issue; if we could figure out on which browser or under which conditions this is happening, it would go a long way toward reducing this problem.

[josephcc]

Yeah, this is just a quick fix, because I didn't want them to have to wait for payment over the weekend. (and there are a lot of them :p

[mizzao]

By the way, do you put your personal (or MS) e-mail on the HIT, so that people can contact you if they have problems? Or do you have access to the communal e-mail for the account?

You can insert a message right at the end of your HIT that if someone has a problem submitting, to reload the page (it should auto-submit); as well as replying this to anyone who e-mails you with the issue.

[josephcc]

I wasn't sure if I should, the mturk account doesnt say MS anywhere and has a very generic account name XD

[mizzao]

Well, you can create a throwaway e-mail for this purpose if you want, but you should definitely give workers some way to contact you. You'll find out about bugs and issues that you would otherwise never have heard of, and that will probably reduce the number of these unsubmitted HITs and waiting problems, among other things.