Dashboard: Request to Elasticsearch failed

[davidediruscio]

After having to stop / start the docker images to solve the task stuck and then for the workers not showing up, the dashboard doesn't work anymore: no list of projects, no data, excepted for the two default projects: microprofile and MeasurePlatform.

[davidediruscio]

@borisbaldassari could you share all the error by clicking in more info?

[davidediruscio]

@borisbaldassari can you share the CROSSMINER instance url, thanks.

[davidediruscio]

Hi @creat89 , @valeriocos ,

Sorry but I can't click now -- I had to clear, update, and restart the instance to get the latest fixes and make some progress on our analysis tasks.. If that happens again I'll do it before anything else, I swear. ;-)

As for the instance it's all happening at the ci4 instance:
http://ci4.castalia.camp:5600/

I'll let you know if/when that occurs again. Thanks for your interest!

[davidediruscio]

Just happened again:

Visualize: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards …
more

Error: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
KbnError@http://ci4.castalia.camp:5600/bundles/commons.bundle.js:1:7780
RequestFailure@http://ci4.castalia.camp:5600/bundles/commons.bundle.js:1:8525
callResponseHandlers/<@http://ci4.castalia.camp:5600/bundles/commons.bundle.js:1:752855
Promise.try@http://ci4.castalia.camp:5600/bundles/commons.bundle.js:1:517092
Promise.map/<@http://ci4.castalia.camp:5600/bundles/commons.bundle.js:1:516472
Promise.map@http://ci4.castalia.camp:5600/bundles/commons.bundle.js:1:516437
callResponseHandlers@http://ci4.castalia.camp:5600/bundles/commons.bundle.js:1:752425
fetchSearchResults/<@http://ci4.castalia.camp:5600/bundles/commons.bundle.js:1:741460
processQueue@http://ci4.castalia.camp:5600/bundles/vendors.bundle.js:63:134252
scheduleProcessQueue/<@http://ci4.castalia.camp:5600/bundles/vendors.bundle.js:63:135201
$digest@http://ci4.castalia.camp:5600/bundles/vendors.bundle.js:63:146077
$apply@http://ci4.castalia.camp:5600/bundles/vendors.bundle.js:63:148845
done@http://ci4.castalia.camp:5600/bundles/vendors.bundle.js:63:101113
completeRequest@http://ci4.castalia.camp:5600/bundles/vendors.bundle.js:63:106024
createHttpBackend/</xhr.onload@http://ci4.castalia.camp:5600/bundles/vendors.bundle.js:63:106783

[davidediruscio]

thank you for the pointers @borisbaldassari, I'll have a look today

[davidediruscio]

It seems some shards are not in good shape: https://ci4.castalia.camp:9200/_cluster/health/?level=shards. This affects also the indexes: https://ci4.castalia.camp:9200/_cat/indices (check the indexes with the red label)

Can you try to give more resources to elasticsearch by increasing the values at:https://github.com/crossminer/scava-deployment/blob/dev/docker-compose.yml#L162 ? thanks

[davidediruscio]

The PR crossminer/scava#377 should improve the performance of moving data between the API and ES

Please @borisbaldassari you could update the dashimporter container and report if the error is solved on your side?

[davidediruscio]

Hi @valeriocos thanks for the update. I did as recommended, stopped the importer container, rebuilt the dashboard image, changed the java args values and restarted the whole thing (docker-compose start).

I guess that the other containers have not been stopped doing that (the instance is doing some intensive computation on the set of projects after I re-initialised the platform yesterday) and I hope that stopping and restarting the importer did not mess up things.. We'll see, thanks again!

[davidediruscio]

(sob ;-)

Should I have done it differently??

[davidediruscio]

Sorry for the late reply @borisbaldassari

We can try the following:

• If you have some intense calculation going on (maybe related to the task analysis), you can comment the sections kibiter, prosoul and dash-importer in the docker-compose, then do a docker-compose stop and docker-compose up. When the calculation is over (or all the projects you want have been analysed), you can uncomment the sections kibiter, prosoul and dash-importer, then stop and start the docker-compose. In this way, you should circumvent some memory issues.
  EDIT: I see that you have 3 workers active on your instance, they are probably using most of the memory, causing ES/kibiter to fail

• You can try to delete the indexes in elasticsearch (curl -XDELETE https://<user>:<pwd>@ci4.castalia.camp:9200/sc* -k and curl -XDELETE https://<user>:<pwd>@ci4.castalia.camp:9200/.k* -k). In the next dash-importer execution they will be recreated.

• A complementary action could be to free up disk space not needed in your machine. For instance, you could remove dangling images (docker rmi $(docker images -f "dangling=true" -q)

• I'm trying to import the projects already analysed* in your instance to crossminer.bitergia.net, and check if the problem you report appears also there.
  *projects completed or almost (99%): 'modelingsirius', 'modelinggendoc','modelingsphinx', 'technologyapp4mc', 'technologymdmbl','technologyepf', 'technologyscout'

[davidediruscio]

Hi @valeriocos ,

The dashboards now are empty, see screenshot.

To answer some of your questions:

• Waiting for the projects analysis to be completed will take days. Well, a bit less since most projects are now in error (see other issues) but it will still take days.
• Yes, there are many workers involved, but I've no choice if I want to have computations achieved in a reasonable amount of time. BTW the host is quite powerful (64GB of RAM) and we're "only" consuming 2/3 of it, leaving 20GB available.
• We still have 192GB disk available on the host, so I don't think it's an issue.
• How do you know which projects have been completed? I can't manage to get this information myself..

Thanks, cheers! :-)

[davidediruscio]

Thank you for the clarifications @borisbaldassari

How do you know which projects have been completed? I can't manage to get this information myself..

I enter to the project page and check the status or the progress bar (http://ci4.castalia.camp:5601/#/project/configure/technologyscout)

After some attempts, it seems that settings the bulk_size to 50 works fine (see log below). I'm uploading the data to crossminer.bitergia.net. I leave the process running and keep you informed.

2019-09-28 15:44:51,640 Importing items from http://ci4.castalia.camp:8182 to https://admin:[email protected]:9200/<built-in method index of list object at 0x7f973b684a48>
2019-09-28 15:44:53,057 Uploading Scava data to Elasticsearch
2019-09-28 15:44:53,057 Looking for projects at url 'http://ci4.castalia.camp:8182'
2019-09-28 15:44:53,591 Looking for 'modelingsirius' project metrics at url 'http://ci4.castalia.camp:8182'

[davidediruscio]

The process is still ongoing, but no errors appeared:

2019-09-28 15:44:51,640 Importing items from http://ci4.castalia.camp:8182 to https://admin:[email protected]:9200/<built-in method index of list object at 0x7f973b684a48>
2019-09-28 15:44:53,057 Uploading Scava data to Elasticsearch
2019-09-28 15:44:53,057 Looking for projects at url 'http://ci4.castalia.camp:8182'
2019-09-28 15:44:53,591 Looking for 'modelingsirius' project metrics at url 'http://ci4.castalia.camp:8182'
2019-09-28 23:19:07,505 Total number of items: 53 (metric)
2019-09-28 23:19:07,505 Metric enrichment summary (metrics in input) - processed: 53, empty: 5
2019-09-28 23:19:07,505 Metric enrichment summary (enriched metrics in output) - total: 126101, enriched: 126101, failed: 0, zero: 64761
2019-09-28 23:19:07,507 project is missing lastExecuted info, the current time will be set: {'_type': 'org.eclipse.scava.repository.model.eclipse.EclipseProject', 'parent': {}, 'executionInformation': {'_id': '7a7521d2-2900-4bcb-bcb3-684ad7a65d0a', '_type': 'org.eclipse.scava.repository.model.ProjectExecutionInformation', 'storage': {'_id': '4d1328d4-18ed-414b-9028-003c082a2aea', '_type': 'org.eclipse.scava.repository.model.LocalStorage', 'path': '/root/scava/modelingemfcompare'}, 'metricProviderData': [], 'inErrorState': True}, 'vcsRepositories': [{'_id': 'f9d364f8-faae-48df-8f92-88d9a4ffae9e', '_type': 'org.eclipse.scava.repository.model.vcs.git.GitRepository', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.vcs.git.VcsRepository'], 'url': 'git://git.eclipse.org/gitroot/emfcompare/org.eclipse.emf.compare.git', 'name': 'emfcompare/org.eclipse.emf.compare'}], 'communicationChannels': [{'_id': 'f6711785-3141-4377-8b09-2a62a7e9c900', '_type': 'org.eclipse.scava.repository.model.eclipse.Documentation', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.eclipse.CommunicationChannel'], 'url': 'http://wiki.eclipse.org/EMF_Compare', '_ossmeterId': ':EclipseDocumentation', 'nonProcessable': True}, {'_id': '4f4e1740-a139-465a-82ae-a35d5a092fd8', '_type': 'org.eclipse.scava.repository.model.cc.wiki.Wiki', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.cc.wiki.CommunicationChannel'], 'url': 'http://wiki.eclipse.org/EMF_Compare', '_ossmeterId': 'null:Wiki', 'nonProcessable': True}, {'_id': 'f1e7baa2-d931-4aa6-b1ae-46d941309190', '_type': 'org.eclipse.scava.repository.model.cc.forum.Forum', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.cc.forum.CommunicationChannel'], 'name': 'EMF Compare forum / newsgroup', '_ossmeterId': 'EMF Compare forum / newsgroup:OssmeterForum', 'url': 'http://eclipse.org/forums/eclipse.emfcompare', 'nonProcessable': True, 'description': 'EMF Compare forum / newsgroup (accessible through NNTP http://www.eclipse.org/newsgroups/register.php)'}, {'_id': 'f11b5159-370f-4cd5-8c00-1c2702f576bf', '_type': 'org.eclipse.scava.repository.model.cc.forum.Forum', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.cc.forum.CommunicationChannel'], 'name': 'EMF forum / newsgroup', '_ossmeterId': 'EMF forum / newsgroup:OssmeterForum', 'url': 'http://www.eclipse.org/forums/eclipse.tools.emf', 'nonProcessable': True, 'description': 'EMF forum / newsgroup (accessible through NNTP http://www.eclipse.org/newsgroups/register.php)'}, {'_id': '8b1201c2-4373-4dc6-8f1e-2b0b4b6c5dc0', '_type': 'org.eclipse.scava.repository.model.cc.nntp.NntpNewsGroup', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.cc.nntp.CommunicationChannel'], 'username': 'exquisitus', '_ossmeterId': 'news.eclipse.org/:NNTPNewsgroup', 'password': 'flinder1f7', 'port': 119, 'newsGroupName': 'EMF Compare forum / newsgroup', 'url': 'news.eclipse.org/', 'authenticationRequired': True, 'nonProcessable': False}, {'_id': '9a2c2e4e-6eb4-4b4c-9dda-d6dd885c59d2', '_type': 'org.eclipse.scava.repository.model.cc.nntp.NntpNewsGroup', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.cc.nntp.CommunicationChannel'], 'username': 'exquisitus', '_ossmeterId': 'news.eclipse.org/:NNTPNewsgroup', 'password': 'flinder1f7', 'port': 119, 'newsGroupName': 'EMF forum / newsgroup', 'url': 'news.eclipse.org/', 'authenticationRequired': True, 'nonProcessable': False}], 'bugTrackingSystems': [{'_id': 'fce2e651-94b1-4ddd-ba1c-f52e4dc46c9a', '_type': 'org.eclipse.scava.repository.model.bts.bugzilla.Bugzilla', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.bts.bugzilla.BugTrackingSystem'], 'component': 'Core', '_ossmeterId': 'https://bugs.eclipse.org/bugs/xmlrpc.cgi:EMFCompare:Core:bugzilla', 'cgiQueryProgram': 'https://bugs.eclipse.org/bugs/buglist.cgi?product=EMFCompare&component=Core', 'url': 'https://bugs.eclipse.org/bugs/xmlrpc.cgi', 'product': 'EMFCompare'}], 'companies': [{'_id': 'f3ef4b7e-0ad7-4b11-bc57-203d04adb2f9', '_type': 'org.eclipse.scava.repository.model.Company', '_superTypes': ['org.eclipse.scava.repository.model.NamedElement'], 'name': 'http://eclipse.org/membership/showMember.php?member_id=863', 'url': 'http://eclipse.org/membership/showMember.php?member_id=863'}], 'platforms': [], 'reviews': [], 'articles': [], 'releases': [], 'name': 'Eclipse EMF Compare', 'shortName': 'modelingemfcompare', 'paragraphUrl': 'http://www.eclipse.org/modeling.emfcompare/project-info/project-page-paragraph.html', 'description': '<p><a href="http://wiki.eclipse.org/index.php/EMF_Compare">Eclipse EMF Compare</a> brings model comparison to the EMF framework, this tool provides generic support for any kind of metamodel in order to compare and merge models. The objectives of this component are to provide a stable and efficient generic implementation of model comparison and to provide an extensible framework for specific needs.</p>\r\n', 'downloadsUrl': 'http://www.eclipse.org/emf/compare/downloads/', 'homePage': 'http://www.eclipse.org/emf/compare/', 'projectplanUrl': 'http://www.eclipse.org/emf/compare/project-info/plan-kepler.xml', 'updatesiteUrl': 'http://download.eclipse.org/modeling/emf/compare/updates/releases', 'state': 'Regular'}
2019-09-28 23:19:07,508 project is missing lastExecuted info, the current time will be set: {'_type': 'org.eclipse.scava.repository.model.eclipse.EclipseProject', 'parent': {}, 'executionInformation': {'_id': '7ab6b853-656e-4368-bce6-29786e07e4b2', '_type': 'org.eclipse.scava.repository.model.ProjectExecutionInformation', 'storage': {'_id': '419c495f-355d-47fd-bd41-bed88dbad10e', '_type': 'org.eclipse.scava.repository.model.LocalStorage'}, 'metricProviderData': []}, 'vcsRepositories': [{'_id': '8321ee7c-3af9-45ae-8bee-255cdbee41b9', '_type': 'org.eclipse.scava.repository.model.vcs.git.GitRepository', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.vcs.git.VcsRepository'], 'url': 'git://git.eclipse.org/gitroot/emf-parsley/org.eclipse.emf-parsley.git', 'name': 'emf-parsley/org.eclipse.emf-parsley'}, {'_id': 'c4417346-037b-4abe-b141-5d2f6890da42', '_type': 'org.eclipse.scava.repository.model.vcs.git.GitRepository', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.vcs.git.VcsRepository'], 'url': 'git://git.eclipse.org/gitroot/emf-parsley/org.eclipse.emf-parsley.core.git', 'name': 'emf-parsley/org.eclipse.emf-parsley.core'}], 'communicationChannels': [{'_id': 'c18405e0-9075-4d48-a51e-d40a1f3a4185', '_type': 'org.eclipse.scava.repository.model.eclipse.Documentation', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.eclipse.CommunicationChannel'], 'url': 'https://www.eclipse.org/emf-parsley/documentation.html', '_ossmeterId': ':EclipseDocumentation', 'nonProcessable': True}, {'_id': '6483c355-4dfb-45ca-918b-db7498093a56', '_type': 'org.eclipse.scava.repository.model.cc.forum.Forum', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.cc.forum.CommunicationChannel'], 'name': 'Emf Parsley Forum', '_ossmeterId': 'Emf Parsley Forum:OssmeterForum', 'url': 'http://www.eclipse.org/forums/eclipse.parsley', 'nonProcessable': True, 'description': 'Use this forum for any question and suggest'}, {'_id': '7eb6f8c1-1242-431d-b2c8-aad43bf766aa', '_type': 'org.eclipse.scava.repository.model.cc.nntp.NntpNewsGroup', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.cc.nntp.CommunicationChannel'], 'username': 'exquisitus', '_ossmeterId': 'news.eclipse.org/:NNTPNewsgroup', 'password': 'flinder1f7', 'port': 119, 'newsGroupName': 'Emf Parsley Forum', 'url': 'news.eclipse.org/', 'authenticationRequired': True, 'nonProcessable': False}], 'bugTrackingSystems': [{'_id': 'f4ffa739-c0db-4552-ae81-ca51ddb644bd', '_type': 'org.eclipse.scava.repository.model.bts.bugzilla.Bugzilla', 'persons': [], '_superTypes': ['org.eclipse.scava.repository.model.bts.bugzilla.BugTrackingSystem'], 'component': '', '_ossmeterId': 'https://bugs.eclipse.org/bugs/xmlrpc.cgi:EMF.Parsley::bugzilla', 'cgiQueryProgram': 'https://bugs.eclipse.org/bugs/buglist.cgi?product=EMF.Parsley', 'url': 'https://bugs.eclipse.org/bugs/xmlrpc.cgi', 'product': 'EMF.Parsley'}], 'companies': [], 'platforms': [], 'reviews': [], 'articles': [], 'releases': [], 'name': 'Eclipse EMF Parsley', 'shortName': 'modelingemfparsley', 'paragraphUrl': 'http://www.eclipse.org/modeling.emf.parsley/project-info/project-page-paragraph.html', 'description': '<p style="text-align: justify;"><strong><span style="color: rgb(0, 128, 0);">Eclipse EMF Parsley</span></strong><img alt="parsley1-1" class="wp-image-6998 alignnone" height="16" src="http://www.rcp-vision.com/wp-content/uploads/2013/05/parsley1-1.png" width="15" />&nbsp;is an Eclipse project that provides a set of reusable UI components based on EMF (e.g., trees, tables, forms, views and editors) that can be used in your EMF-based plugin out-of-the-box.&nbsp;These parts are highly customizable via a&nbsp;DSL that let change a lot of details in a very compact form.&nbsp;</p>\r\n<p style="text-align: justify;">This project has been designed starting form EMF UI generated code, adding extensibility mechanism that make it very modular: all components&nbsp;developed are highly reusable and customizable via Dependency Injection (in particular, Google Guice).</p>\r\n<p style="text-align: justify;">As in cooking,<span style="color: #008000;"> &nbsp;<strong>EMF Parsley</strong><span style="color: #008000;"><img alt="parsley1-1" height="16" src="http://www.rcp-vision.com/wp-content/uploads/2013/05/parsley1-1.png" width="15" /></span></span>&nbsp;aims to be used in several EMF based parts. Its little components can be embedded whereever there is an EMF model and fully customized.</p>\r\n<p style="text-align: justify;">&nbsp;</p>\r\n<p class="rtecenter">&nbsp;</p>\r\n<p><iframe allowfullscreen="" frameborder="0" height="315" src="//www.youtube.com/embed/-S8mh5p-ChE" width="420"></iframe></p>\r\n', 'downloadsUrl': 'https://www.eclipse.org/emf-parsley/download.html', 'homePage': 'http://www.eclipse.org/emf-parsley', 'state': 'Regular'}
2019-09-28 23:19:08,691 Looking for 'modelinggendoc' project metrics at url 'http://ci4.castalia.camp:8182'
2019-09-28 23:52:03,447 Total number of items: 53 (metric)
2019-09-28 23:52:03,448 Metric enrichment summary (metrics in input) - processed: 53, empty: 5
2019-09-28 23:52:03,448 Metric enrichment summary (enriched metrics in output) - total: 63457, enriched: 63457, failed: 0, zero: 46987
2019-09-28 23:52:03,595 Looking for 'modelingsphinx' project metrics at url 'http://ci4.castalia.camp:8182'
2019-09-29 00:19:41,096 Total number of items: 53 (metric)
2019-09-29 00:19:41,096 Metric enrichment summary (metrics in input) - processed: 53, empty: 6
2019-09-29 00:19:41,096 Metric enrichment summary (enriched metrics in output) - total: 61394, enriched: 61394, failed: 0, zero: 51011
2019-09-29 00:19:41,289 Looking for 'technologyapp4mc' project metrics at url 'http://ci4.castalia.camp:8182'
2019-09-29 00:44:54,297 Total number of items: 53 (metric)
2019-09-29 00:44:54,297 Metric enrichment summary (metrics in input) - processed: 53, empty: 5
2019-09-29 00:44:54,298 Metric enrichment summary (enriched metrics in output) - total: 63452, enriched: 63452, failed: 0, zero: 46990
2019-09-29 00:44:54,466 Looking for 'technologymdmbl' project metrics at url 'http://ci4.castalia.camp:8182'

[davidediruscio]

http://ci4.castalia.camp:8182 is not available, the importer stopped at technologymdmbl

[davidediruscio]

@valeriocos Yeah again: no more projects, no more workers, ...
Now I need to focus on my own tasks (use case + deliverable) so I'll just let it go and hope that it automagically works again by itself. If not, I'll update it when I have some time. ;-)

[davidediruscio]

ok! good luck!

[davidediruscio]

* We still have 192GB disk available on the host, so I don't think it's an issue.

Unfortunately, it might be the issue, cf.
crossminer/scava#390 (comment). It seems that the container's volume size is set to 10GB, and I can't find a way to specify another value in the docker-compose/Dockerfile.