AWX 13 broken due upstream issues

[ikke-t]

Permission issues:

• Permission denied for launch_awx.sh and launch_awx_task.sh ansible/awx#7545
• Missing module python-memcached in built-in venv awx ansible/awx#7544

So let's label awx_pod with 13.0 once those are fixed. This module is broken due those ATM at master branch. If you want to use a working module, select earlier version.

[Klaas-]

memcached was removed from awx in 12, I'd guess we need to change this role to reflect the upstream changes in kubernetes role; CCing @cfelder ansible/awx#7240

[Klaas-]

https://github.com/ikke-t/awx_pod/blob/master/templates/settings.py.j2
https://github.com/ansible/awx/blob/devel/installer/roles/kubernetes/templates/configmap.yml.j2#L128-L202

and

https://github.com/ansible/awx/blob/devel/installer/roles/kubernetes/templates/deployment.yml.j2
https://github.com/ikke-t/awx_pod/blob/master/templates/awx.yml.j2

don't have my test setup ready at the moment so I won't get around to that soon :D

[ikke-t]

I suppose the order of getting this back to work would be to submit PRs to awx about those two starter script execute bits. Then waiting to get the upstream containers being rebuilt to work with PR changes. Once having those in place, there is place to make and test the above changes to drop memcached and update settings.

Creating the PR is small job, I wonder if @cfelder already has the changes in his repo?

I have a bit quieter time at work than usual, I could also do some of this one day.

[cfelder]

I am already on it (removing memcache).

I'll also post an example playbook here to run with a derived container image which has already fixed executable bits.

If I'll find the correct spot in the awx repo I can make a PR as well for fixing those Dockerfiles.

[cfelder]

The following example uses a derived container with fixed acls and removed preloading data into awx as well.

- name: run AWX on host
  hosts: all
  become: yes
  vars:
    awx_pod_name: awx
  tasks:
    - include_role:
        name: awx_pod
      vars:
        awx_admin_user: admin
        awx_admin_password: foobar
        awx_podman_dir: /tmp
        awx_host_port: 8052
        awx_pod_label: awx
        awx_awx_image: quay.io/cfelder/awx:nopreload-{{ awx_awx_version }}

Dockerfile

FROM docker.io/ansible/awx:13.0.0
LABEL maintainer="Christian Felder"

USER root

RUN dnf update -y && \
  dnf clean all -y
RUN sed -i '/.*awx-manage.*create_preload_data.*/d' \
  /usr/bin/launch_awx_task.sh \
  && \
 chmod 755 /usr/bin/launch_awx*.sh

USER 1000

needs: #19

[Klaas-]

If I recall right awx assumes it runs priviledged, so you won't need to change the runscripts but add priviledged to the web container. With 11 it seems this was optional for the web container, now that they are not seperated anymore it seems mandatory.

[Klaas-]

and for the no preload, I am guessing you could just configmap a launch_awx_task.sh into the image, then you don't need to rebuild it. But that would make a fine upstream change, create a env variable that you can set to skip it :)

[ikke-t]

please review if these changes fix the issues, I could just press the PR button: ansible/awx@6c107db

[cfelder]

If I recall right awx assumes it runs priviledged

RunAsUser and RunAsGroup are not honoured in podman versions shipped with EL7 and EL8 you would need at least podman 1.7

[Klaas-]

@cfelder https://github.com/ikke-t/awx_pod/blob/master/templates/awx.yml.j2#L216-L217 that also for the web container, that should help :)

[cfelder]

@cfelder https://github.com/ikke-t/awx_pod/blob/master/templates/awx.yml.j2#L216-L217 that also for the web container, that should help :)

I am still getting

$ podman logs -f awx_awxtask
/bin/sh: /usr/bin/launch_awx_task.sh: Permission denied

$ podman logs -f awx_awxweb
/bin/sh: /usr/bin/launch_awx.sh: Permission denied

although I added securityContext...

Looks like privileged: True is not honored in kube play

when using

$ podman run --privileged=true --rm -it docker.io/ansible/awx:13.0.0 /usr/bin/launch_awx.sh

the shell script is executed correctly.

Nevertheless shouldn't running unprivileged containers be preferred?

[Klaas-]

It should be honoured, otherwise the tasks container should have never been working :)

but upstream also changed the security context to unpriviledged for all containers:
ansible/awx@b13a175

so maybe there is something else in the deployment that has changed (or the deployment really was never working for upstream 13)

[Koleon]

Hello guys,
I'm very interested in the latest AWX running on podman (RHEL8). Are there any options how to help with it?

[brianmillett]

Not sure if this is progress, or chasing a red herring, but,
looking at the generated /etc/containers/pods/awx.yaml, I added the following securityContext to the awx_awxweb container and the awx_awxtask container.

securityContext:
  allowPrivilegeEscalation: true
  capabilities: {}
  privileged: true
  readOnlyRootFilesystem: false
  runAsGroup: 0
  runAsUser: 0
  seLinuxOptions: {}

Running the podman play command from the systemd service, I get the awx to start, but the logs:
podman logs -f awx-awx_awxweb & podman logs -f awx-awx_awxtask return the same errors: ModuleNotFoundError: No module named 'memcache'

Traceback (most recent call last):
File "/usr/bin/awx-manage", line 8, in
sys.exit(manage())
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py", line 154, in manage
execute_from_command_line(sys.argv)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 361, in execute
self.check()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 390, in check
include_deployment_checks=include_deployment_checks,
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/management/base.py", line 377, in _run_checks
return checks.run_checks(**kwargs)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/checks/registry.py", line 72, in run_checks
new_errors = check(app_configs=app_configs)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/checks/urls.py", line 40, in check_url_namespaces_unique
all_namespaces = _load_all_namespaces(resolver)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/checks/urls.py", line 57, in _load_all_namespaces
url_patterns = getattr(resolver, 'url_patterns', [])
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/utils/functional.py", line 80, in get
res = instance.dict[self.name] = self.func(instance)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/urls/resolvers.py", line 584, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/utils/functional.py", line 80, in get
res = instance.dict[self.name] = self.func(instance)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/urls/resolvers.py", line 577, in urlconf_module
return import_module(self.urlconf_name)
File "/var/lib/awx/venv/awx/lib64/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/urls.py", line 18, in
url(r'^api/', include('awx.api.urls', namespace='api')),
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/urls/conf.py", line 34, in include
urlconf_module = import_module(urlconf_module)
File "/var/lib/awx/venv/awx/lib64/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/api/urls/init.py", line 5, in
from .urls import urlpatterns
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/api/urls/urls.py", line 74, in
from .oauth2_root import urls as oauth2_root_urls
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/api/urls/oauth2_root.py", line 10, in
from oauth2_provider import views
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/oauth2_provider/views/init.py", line 2, in
from .base import AuthorizationView, TokenView, RevokeTokenView
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/oauth2_provider/views/base.py", line 62, in
class AuthorizationView(BaseAuthorizationView, FormView):
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/oauth2_provider/views/base.py", line 85, in AuthorizationView
server_class = oauth2_settings.OAUTH2_SERVER_CLASS
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/init.py", line 85, in oauth2_getattribute
val = settings.OAUTH2_PROVIDER.get(attr)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 500, in getattr_without_cache
return getattr(self._wrapped, name)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 413, in getattr
value = self._get_local(name)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 329, in _get_local
self._preload_cache()
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 283, in _preload_cache
if self.cache.get('_awx_conf_preload_expires', default=empty) is not empty:
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/conf/settings.py", line 137, in get
value = self.cache.get(key, **kwargs)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/cache/init.py", line 98, in getattr
return getattr(caches[DEFAULT_CACHE_ALIAS], name)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/cache/init.py", line 79, in getitem
cache = _create_cache(alias)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/cache/init.py", line 54, in _create_cache
return backend_cls(location, params)
File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/core/cache/backends/memcached.py", line 151, in init
import memcache
ModuleNotFoundError: No module named 'memcache'

[cfelder]

See #19 which removes memcache

[brianmillett]

See #19 which removes memcache

Thanks

I applied those changes, as well as the securityContext I mentioned earlier, and now I have a working podman awx 13.0.0

thanks.

[ikke-t]

Why does the web container need to be privileged? Sounds like there are some permission fixes to be done, normally any web server should not need any special privileges.

[ikke-t]

Or was that just to go around the fact that launchers were missing execute bits for the groups and others? That fix is btw now merged in to awx.

[brianmillett]

Or was that just to go around the fact that launchers were missing execute bits for the groups and others? That fix is btw now merged in to awx.

Yes, since only root could execute:

bash-4.4# ls -l /usr/bin/launch_awx.sh
-rwx------. 1 root root 795 Jun 23 18:59 /usr/bin/launch_awx.sh

Good to know that it has been fixed.