Many analyses not completing

[gubuntu]

problem

Many analyses are not completing

proposed solution

troubleshoot and fix...

[lucernae]

Hi @gubuntu , downloaded the layer and tested it in my local env. I can replicate this consistently if I'm using a huge bounds. I think that's the cause (analysis extent is too large).
I'm suggesting we should change the focus to informatively tell the user why an analysis is failing. At the moment we record the stack trace, but it only contains information that the anlysis is crashed (probably huge memory usage is the cause), so it's not really informative for the user. We don't know a way to flag analysis as failing too. But periodic task check might solve that.

So, in summary, I had to implement more robust task tracker (the issue is in the backlog #383). It's not going to fix the issue, but provides more information on why a task is failing. Then we can check individual cause for each task. But this is going to be a huge feature to implement.

[gubuntu]

@NyakudyaA can you please emulate a few of the stauck processes on the desktop and see how long they take or if there are any other issues?

[NyakudyaA]

@NyakudyaA can you please emulate a few of the stauck processes on the desktop and see how long they take or if there are any other issues?

@gubuntu Will attend to this and let them run. But my machine is a little beast. It might outperform the server

[lucernae]

guys, @timlinux @NyakudyaA , this issue is related with #518
#383

But, honestly it is such a big block of features if we want to implement the whole diagnostics in GeoSAFE. I was hoping that maybe we could at least show some relevant information for the user, at least to provide some feedback so user knows what is going on first. Now GeoSAFE hides celery complexity so it only shows that a task is pending, but doesn't tell the progress (because we don't have a way to get it in the first place). What I proposes is to have an Analysis Detail page again, but fill it with some useful information on how to replicate the analysis on Desktop, so user like @NyakudyaA can replicate it.

We have Analysis Detail page currently, but we only put it for developers like me to debug analysis error. It might be good to add some several details like:

• Link to exposure layer or download link
• Link to hazard layer or download link
• Link to aggregation layer or download link
• Link to analysis page
• Show Analysis extent in the map
• Show Analysis extent in text form
• Show the stage of the analysis (is it received by headless, currently calculating, finished calculating, generating the report, until done)
• Some Action button to cancel or rerun analysis
• Link to analysis product if any (analysis layer, report)
• Show the last stack trace and exception type if task has failed

I am suggesting this because I already tried rerunning the analysis again and again and it really took a long time to complete and I set up a cron schedule to always restart InaSAFE Headless daily, to make sure celery worker is always fresh (no memory leak). This mean what really happens is those pending analysis took more than a day to complete and got restarted every day, so it never completes.