What to do when the 'Inject nginx image' step fails?

[Baitanik]

metal.log

I have attached the log..
now how to get what is the actual problem it is facing ..
I have executed with --verbose while running bootstrap.sh ...

Could anyone please help..

[TeddyAndrieux]

Hello @Baitanik,

First, thank you for your report.

I think from the output you provided the minion id you put in the bootstrap.yaml file in ca:minion does not match the minion id of the minion running on the bootstrap node (you can get this one with salt-call --local grains.get id).
Note that before starting the bootstrap script you can force a specific salt minion id just by setting this one in /etc/salt/minion_id file, (for example if you want your bootstrap node to be named "bootstrap" you just have to do a mkdir /etc/salt/ && echo 'bootstrap' > /etc/salt/minion_id before starting the bootstrap script and that it)

I know the error is not clear at all and we should really work on that.

Just for your information, we start using GitHub discussions so for that kind of question (or any other question you have) do not hesitate to open one. https://github.com/scality/metalk8s/discussions

[NicolasT]

As part of bootstrapping the whole system, we need to inject a container image to run nginx in containerd, which is later used to run a (fake) container registry from which all other images are pulled. This step seems to fail:

          ID: Inject nginx image
    Function: containerd.image_managed
        Name: docker.io/library/nginx:1.15.8
      Result: False
     Comment: Failed to import archive: time="2020-12-17T16:09:12Z" level=debug msg="unpacking 1 images"
     Started: 16:09:09.688320
    Duration: 2790.134 ms
     Changes:

Could you check the containerd logs, using journalctl -f -u containerd while the bootstrap.sh script is running?

From the log, it appears you ran the bootstrap.sh script before as well, right? Or is this a 'clean' machine? Re-running bootstrap.sh should never harm, the step (Inject nginx image) could have failed due to a temporary issue on the system.

[NicolasT]

Ping @Baitanik in case you didn't notice this was moved into Discussions 😃

[Baitanik]

Thanks for all the reply ..
we found the issue the image it was not able to import actually ...
wanted to know what is the best way to debug metal k8s installation issues.. earlier i had faced issue.. when docker was installed..
and it keep on failing .. for runc installation... it was very difficult to get the actual issue...

sometimes of one of rpm package is already installed in node... and again metalk8s-salt tries to install it ... it fails...
but the errors are somewhat confusing in the log..

[NicolasT]

What is the best way to debug metal k8s installation issues
That's hard to say, during deployment a lot of things happen in various parts of the system, and there's no single way to look into all of them. The 'easiest' approach is to look at which Salt step failed to execute (which was fairly clear from your logs), then look at what the step does, or logging related to the subsystem it interacts with.

We're also looking into ways to provide more direct feedback about what's going on while bootstrap is running, instead of not giving any information for minutes in a row until a step succeeds (or fails), then dumping a lot of information all at once. See for example #2207.

On docker being installed and this being incompatible, indeed this is an issue (cfr. #2989 (reply in thread)) that's hard to troubleshoot unless you know what to look for. We'll improve on it!

Meanwhile, please continue providing feedback, this really helps for us to know how/where to improve this project.

[Baitanik]

maybe we could have some checklist of something... when we run bootstrap... in the perquisite step it will check the existing system and tell if some packages is installed.. please uninstall it or something like that ?

[NicolasT]

#2992

[Baitanik]

Thanks @NicolasT .... :)

[Baitanik]

as we have created jira .. FYI one time we had seen following issue also..

installed:
.. ...
python2-msgpack.x86_64 0:0.6.2-2.el7
python2-psutil.x86_64 0:5.6.7-1.el7
salt.noarch 0:3000.3-1.el7
systemd-python.x86_64 0:219-78.el7_9.2
zeromq.x86_64 0:4.1.4-7.el7
systemd-sysv.x86_64 0:219-78.el7_9.2

Failed:
python-urllib3.noarch 0:1.10.2-7.el7

Complete!
<< END >>

This script will now exit

---

but it was easier to guess.. uninstall urllib3 which was earlier installed by pip....
we uninstall urllib3 .. ....

then it gave following error

 File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 867, in resolve
  raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1' distribution was not found and is required by requests
<< END >>

then we uninstall request package...

then it went through...

[NicolasT]

That's useful, thanks! I'll link this thread in the ticket as well.

[NicolasT]

Now also: #3050