Installing plugin takes a long time and sometimes errors out when installing the aws dependencies #293
Comments
|
What kind of resources does your CI system have? On my laptop It takes a little more than a minute even with compilation: https://gist.github.com/majormoses/871cd3dbad9639b66813d5427b4d00a0 My laptop has 16GB of RAM and a |
|
I attempted to replicate again restricting the docker container to a single CPU and |
|
During the CI build is there any indicator which resources are potentially a bottleneck? I would check:
|
|
It seems there's a problem with the latest versions of the |
|
Hmm I will have to look into it more this weekend and try to reproduce this. |
|
Installing on t3.mediums in AWS took ~15 mins on average. Tested on 3 different Centos 7 instances |
|
Interesting I will try it on an aws instance some time next week at work to see if its different but again locally I can not reproduce any install failures or installs that take more than 2 minutes, I tested this in and outside of docker. I read through the upstream issues and rubygems has no intention of solving it at this time and the aws gem maintainers suggest something that is not exactly trivial or convenient for this use case. The recommendation is to only require the components you need rather than @robert-put in the case of aws |
|
Was finally able to reproduce it in vagrant: https://gist.github.com/majormoses/871cd3dbad9639b66813d5427b4d00a0#file-timed-logs-install-aws-plugin-vagrant-txt using 1 GB and 1 core, I will try to see if above some threshold it becomes a quick install as I see locally on my machine. Apologies in advance as I will be traveling for work this week and may not respond in a timely manner. |
|
I doubt it will make it much better but we did finally remove the |
|
Is there any update on this? Installation is still very slow using latest version. Thanks for assisting. |
|
I have so far only been able to replicate this on servers with very few resources and I only install this plugin on a handful of servers so I have not had a personal need to work on this. Essentially what we need to do is rather than pull in all of |
|
Similar to the compiling of nokogiri and unf_ext, is there anything that
can be done to speed up the aws-sdk?
We’ve seen instances like t2.medium do puppet installs of sensu and plugins
including this one and from time to time get OOM errors.
Building native extensions. This could take a while...
ERROR: Error installing sensu-plugins-aws:
ERROR: Failed to build gem native extension.
current directory:
/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/nokogiri-1.8.5/ext/nokogiri
/opt/sensu/embedded/bin/ruby -r ./siteconf20181119-26141-tabkmz.rb
extconf.rb
Cannot allocate memory - /opt/sensu/embedded/bin/ruby -r
./siteconf20181119-26141-tabkmz.rb extconf.rb 2>&1
…On Wed, Nov 21, 2018 at 1:20 AM Ben Abrams ***@***.***> wrote:
I have so far only been able to replicate this on servers with very few
resources and I only install this plugin on a handful of servers so I have
not had a personal need to work on this. Essentially what we need to do is
rather than pull in all of aws-sdk we need to pull in specific portions
of it to reduce the number of dependencies during install. This will be
something that will need a lot of help testing as it has a very high risk
of accidentally leaving something out. We do have some test coverage but I
am not confident that we have enough to where we can rely solely on
automated testing. If someone wants to start taking a crack at it I am all
for it I just don't have the bandwidth myself to undertake this.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#293 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAr84V5h3OFKKFyM15twbzOc01U80HMiks5uxPCwgaJpZM4WfOZ_>
.
|
|
The problem I think is less with compiling and more an issue of dependency explosion by simply including |
majormoses
changed the title
|
Also seeing this issue today on a slow machine 1 core (Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz ) and 2GB RAM. Is there a fix or workaround for this? I will attempt to only install the needed gems. Do you have an example on how to install just the aws rds sensu plugin checks? |
|
@zeninfinity if you can pull the code from #342 and run |
|
Ok, so I get the following input: `# /opt/sensu/embedded/bin/gem install ./sensu-plugins-aws-18.0.0.gem ... GemWrappers: Can not wrap missing file: metrics-netstat-tcp.rb |
|
Hmm I will take a closer look when I have some time. Does it actually fail or are those just warnings? |
|
Was A LOT quicker...like 3 minutes vs. hanging for more than 25 minutes. Though still not 100% sure what to do next. @majormoses - Just warnings. Seems to have installed the gems. |
excellent
Do you use any of the existing resources/checks? If so just try running your checks as you would and report back if any of them start breaking. If not you can start defining some checks and report back which ones you tested and their status.
cool, we can look into the warnings, want to focus on verification of existing functionality and unblocking people, we can then work through the warnings. |
|
Actually I was really trying to run |
|
|
gotcha ya those wrapped things are not warnings then. I will take a closer look when I have the time. Can you please comment to this on the PR so the author of the PR knows that what is currently proposed does not work. |
Yep, can do. |
|
its gonna be more like
its gonna be more like |
|
Yep. Nothing in the /opt/sensu/embedded/bin directory with rds in the name. |
|
So is there a version of sensu-plugins-aws that is a working version? |
|
Ok...just an FYI so people don't get down the wrong rabbit hole. It looks like it is installing these in my rvm ruby location. Pre-installation I get: which is the repo I pulled down. Post installation I get: So my ruby isn't setup to install them in /opt/sensu/embedded/bin/ ... though 109 checks/metrics are in that location. |
The latest released version should be working but it does require some non trivial resources to install at the moment in a timely manner. |
|
Ok, that makes sense. When I put I'm battling an rvm vs embedded ruby issue. /etc/default/sensu is: Is there another way I can install this to have sensu be able to run this metrics-rds.rb? |
|
Digging deeper I was able to install sensu-plugins-aws via |
|
@majormoses This plugin uses aws-sdk v3 for only check-s3-bucket visibility (https://github.com/sensu-plugins/sensu-plugins-aws/pull/271/files). |
|
Sorry for the missed message, I thought quite a few of them were using v3 but I think I was remembering this effort and realizing this was go get it to v2: #240. I don't want to drop features and I am fairly certain you can't just drop it to v2 or I would not have likely (still possible) bumped it to a v3 in that PR. We should be moving forward on sdk versions rather than backwards and the outlined problem have only ever been able to reproduce with low resource environments. I agree there is an issue but overall this is fairly easily mitigated for orgs until someone wants to bring up the rest of the code to use v3 and cut a lot of the gems being pulled in. |
@majormoses So I've been trying to install this plugin during acceptance testing and even pipeline builds and it always causes a HUGE bottleneck for the install.
So in taking your advice (https://github.com/sensu-plugins/community/blob/master/best_practices/production_deployments/plugins/WITHOUT_COMPILERS.md) and (sensu/sensu-puppet#908) I compiled unf_ext and nokogiri, uploaded to artifactory and installed prior to installing this gem and it's not helping (Maybe the document or this plugins README should be updated to handle the compiled dependencies in order to avoid needing compiler to be installed on the resulting client machine.
I took centos:7 image and preinstalled sensu to save me a step and here is the resulting output. As you can see the time it took to install this plugin took 6 minutes alone even after the dependency gems were installed. In our CI systems they don't have much memory or cpu and I've seen them take > 50 minutes for this step for some reason... (< 7m for full build if I remove just this plugin).
There has to be a better way to speed up this install.
The text was updated successfully, but these errors were encountered: