Build successful but no effect inside containers

[Nexlo]

ISSUE TYPE

• Documentation Report

container.yml

version: "2"
settings:
  conductor:
    base: ubuntu:14.04
    save: yes
    roles_path:
      - /mnt/d/Projects/ansible-playbook/roles/
    # volumes:      # Provide a list of volumes to mount
    # environment:  # List or mapping of environment variables
  project_name: ansible-playbook

services:
  web:
    from: ubuntu:14.04
    roles:
      - role: apache2-server
        gather_facts: no
    command: ["tail", "-f", "/dev/null"]
    ports:
      - "8080:80"

registries: {}

OS / ENVIRONMENT

Ansible Container, version 0.9.2
Linux, Nexlo-PC, 4.4.0-43-Microsoft, #1-Microsoft Wed Dec 31 14:42:53 PST 2014, x86_64
2.7.13 (default, Nov 24 2017, 17:33:09) 
[GCC 6.3.0 20170516] /usr/bin/python
{
  "ContainersPaused": 0, 
  "Labels": [], 
  "CgroupDriver": "cgroupfs", 
  "ContainersRunning": 0, 
  "ContainerdCommit": {
    "Expected": "773c489c9c1b21a6d78b5c538cd395416ec50f88", 
    "ID": "773c489c9c1b21a6d78b5c538cd395416ec50f88"
  }, 
  "InitBinary": "docker-init", 
  "NGoroutines": 36, 
  "Swarm": {
    "ControlAvailable": false, 
    "NodeID": "", 
    "Error": "", 
    "RemoteManagers": null, 
    "LocalNodeState": "inactive", 
    "NodeAddr": ""
  }, 
  "LoggingDriver": "json-file", 
  "OSType": "linux", 
  "HttpProxy": "", 
  "Runtimes": {
    "runc": {
      "path": "docker-runc"
    }
  }, 
  "DriverStatus": [
    [
      "Backing Filesystem", 
      "extfs"
    ], 
    [
      "Supports d_type", 
      "true"
    ], 
    [
      "Native Overlay Diff", 
      "true"
    ]
  ], 
  "OperatingSystem": "Docker for Windows", 
  "Containers": 2, 
  "HttpsProxy": "", 
  "BridgeNfIp6tables": true, 
  "MemTotal": 2076430336, 
  "SecurityOptions": [
    "name=seccomp,profile=default"
  ], 
  "Driver": "overlay2", 
  "IndexServerAddress": "https://index.docker.io/v1/", 
  "ClusterStore": "", 
  "InitCommit": {
    "Expected": "949e6fa", 
    "ID": "949e6fa"
  }, 
  "GenericResources": null, 
  "Isolation": "", 
  "SystemStatus": null, 
  "OomKillDisable": true, 
  "ClusterAdvertise": "", 
  "SystemTime": "2018-05-03T09:27:04.9817685Z", 
  "Name": "linuxkit-00155d65c109", 
  "CPUSet": true, 
  "RegistryConfig": {
    "AllowNondistributableArtifactsCIDRs": [], 
    "Mirrors": [], 
    "IndexConfigs": {
      "docker.io": {
        "Official": true, 
        "Name": "docker.io", 
        "Secure": true, 
        "Mirrors": []
      }, 
      "0.0.0.0:2375": {
        "Official": false, 
        "Name": "0.0.0.0:2375", 
        "Secure": false, 
        "Mirrors": []
      }
    }, 
    "AllowNondistributableArtifactsHostnames": [], 
    "InsecureRegistryCIDRs": [
      "127.0.0.0/8"
    ]
  }, 
  "DefaultRuntime": "runc", 
  "ContainersStopped": 2, 
  "NCPU": 2, 
  "NFd": 19, 
  "Architecture": "x86_64", 
  "KernelMemory": true, 
  "CpuCfsQuota": true, 
  "Debug": true, 
  "ID": "WYS6:73GN:YYWW:S7V4:FJSM:LFJY:E3E6:JC2A:YBRD:VFQM:UN7C:MAGR", 
  "IPv4Forwarding": true, 
  "KernelVersion": "4.9.87-linuxkit-aufs", 
  "BridgeNfIptables": true, 
  "NoProxy": "", 
  "LiveRestoreEnabled": false, 
  "ServerVersion": "18.03.1-ce", 
  "CpuCfsPeriod": true, 
  "ExperimentalBuild": false, 
  "MemoryLimit": true, 
  "SwapLimit": true, 
  "Plugins": {
    "Volume": [
      "local"
    ], 
    "Network": [
      "bridge", 
      "host", 
      "macvlan", 
      "null", 
      "overlay"
    ], 
    "Authorization": null, 
    "Log": [
      "awslogs", 
      "fluentd", 
      "gcplogs", 
      "gelf", 
      "journald", 
      "json-file", 
      "logentries", 
      "splunk", 
      "syslog"
    ]
  }, 
  "Images": 259, 
  "DockerRootDir": "/var/lib/docker", 
  "NEventsListener": 1, 
  "CPUShares": true, 
  "RuncCommit": {
    "Expected": "4fc53a81fb7c994640722ac585fa9ca548971871", 
    "ID": "4fc53a81fb7c994640722ac585fa9ca548971871"
  }
}
{
  "KernelVersion": "4.9.87-linuxkit-aufs", 
  "Components": [
    {
      "Version": "18.03.1-ce", 
      "Name": "Engine", 
      "Details": {
        "KernelVersion": "4.9.87-linuxkit-aufs", 
        "Os": "linux", 
        "BuildTime": "2018-04-26T07:22:38.000000000+00:00", 
        "ApiVersion": "1.37", 
        "MinAPIVersion": "1.12", 
        "GitCommit": "9ee9f40", 
        "Arch": "amd64", 
        "Experimental": "false", 
        "GoVersion": "go1.9.5"
      }
    }
  ], 
  "Arch": "amd64", 
  "BuildTime": "2018-04-26T07:22:38.000000000+00:00", 
  "ApiVersion": "1.37", 
  "Platform": {
    "Name": ""
  }, 
  "Version": "18.03.1-ce", 
  "MinAPIVersion": "1.12", 
  "GitCommit": "9ee9f40", 
  "Os": "linux", 
  "GoVersion": "go1.9.5"
}

OS / ENVIRONMENT (additional)

I am running Windows Subsystem for Linux on Windows 10 with connected docker client from my linux to my windows host (docker-for-windows). Works fine so far. I dont think/hope its related to this here.

SUMMARY

I am new with ansible-container ;) ..have a little doubt that I use something wrong. Thats why I've opend a documentation.

After successful ansible-container build I expect the tasks defined in my role which is assigned to the container have been executed inside the container.

Executing ansible-container --debug build says:
Applied role to service
Committed new layer as image and
All images successfully built

I've initialized the role via ansible-galaxy init apache2-server and added afterwards in roles/apache2-server/tasks/main.yml a task to install vim.

EXPECTED RESULTS

The packages are installed inside the container.

ACTUAL RESULTS

Packages are not installed inside the container.

OUTPUT build (end partial)

Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed /tmp/tmpUxGCsz/hosts inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/local/lib/python2.7/dist-packages/ansible/plugins/callback/__init__.pyc

PLAYBOOK: playbook.yml *********************************************************
1 plays in /tmp/tmpUxGCsz/playbook.yml

PLAY [web] *********************************************************************
META: ran handlers
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************

2018-05-03T09:32:14.703758 Playbook run finished.         [container.core] caller_file=/_ansible/container/core.py caller_func=conductorcmd_build caller_line=811 exit_code=0
2018-05-03T09:32:14.706906 Applied role to service        [container.core] caller_file=/_ansible/container/core.py caller_func=conductorcmd_build caller_line=814 role={"role": "apache2-server", "which_container": "web"} service=u'web'
2018-05-03T09:32:15.486004 Call: Engine.commit_role_as_layer [container.docker.engine] args=(u'9e4641b4131b1bdbd9832b6f1bccddcc4efafc63fabad5fb6925cd208e3457a9', u'web', '617182ac2f08d2a18e319b04064bdec142548304ebdff3c7634021dbd9664c2f', ordereddict([(u'command', ['tail', '-f', '/dev/null']), (u'from', u'ubuntu:14.04'), (u'ports', ['8080:80']), (u'roles', [ordereddict([('role', 'apache2-server'), ('which_container', 'web')])]), ('defaults', ordereddict([(u'gather_facts', u'no'), (u'which_container', u'web')]))])) caller_file=/_ansible/container/docker/engine.py caller_func=Engine.commit_role_as_layer caller_line=14 kwargs={'with_name': True}
2018-05-03T09:32:15.492248 Committing new layer           [container.docker.engine] caller_file=/_ansible/container/docker/engine.py caller_func=commit_role_as_layer caller_line=681 params={'message': 'Built with Ansible Container (https://github.com/ansible/ansible-container)', 'tag': '20180503093215', 'changes': u'', 'conf': {'Hostname': '', 'Domainname': '', 'Cmd': ['tail', '-f', '/dev/null'], 'WorkingDir': '', 'Labels': {'com.ansible.container.fingerprint': '617182ac2f08d2a18e319b04064bdec142548304ebdff3c7634021dbd9664c2f'}, 'Entrypoint': None, 'User': '', 'Env': [], 'ExposedPorts': {'80': {}}, 'OnBuild': []}, 'repository': u'ansible-playbook-web'}
2018-05-03T09:32:15.670654 Committed layer as image       [container.core] caller_file=/_ansible/container/core.py caller_func=conductorcmd_build caller_line=828 image=u'sha256:f9631436e69265a119c9bca771ba67d8d9a13aee26404c3ad82adf231f41ac26' service=u'web'
2018-05-03T09:32:15.708848 Build complete.                [container.core] caller_file=/_ansible/container/core.py caller_func=conductorcmd_build caller_line=833 service=u'web'
2018-05-03T09:32:15.711771 All images successfully built. [container.core] caller_file=/_ansible/container/core.py caller_func=conductorcmd_build caller_line=836
2018-05-03T11:32:16.422662 Conductor terminated. Preserving as requested. [container.docker.engine] caller_file=/usr/local/lib/python2.7/dist-packages/container/docker/engine.py caller_func=await_conductor_command caller_line=462 command_rc=0 conductor_id=u'644b0933b37031ff1c7553321bcc9c976b2fa73be0b3c7599c494c3f67fe471a' save_container=yes
nexlo@Nexlo-PC /mnt/d/Projects/ansible-playbook
 (ansible-container *$*)$  

QUESTIONS

Do I have to select/use this new create image manually?

Are my tasks really executed? (see output)

Thanks in advance for any help! :)

[Nexlo]

I've created a small example to provide some more information:
https://github.com/Nexlo/ansible-test

Includes:

• ansible-container init
• ansible-galaxy init roles/my-new-role
• customized container.yml
• customized roles/my-new-role/tasks/main.yml

After checkout you can run ansible-container build and ansible-container run, both successful but still without installed package inside the web-container.

What I am doing wrong? 😢

[Voronenko]

I have tried your example ,

at least command ansible-container --debug build installs packages as expected in your demo repo.
Please give it a try , if it appears to work - please support https://github.com/ansible/ansible-container/pull/938/files PR to get merged by @j00bar @gregdek or other maintainer.

Related issue:
#937

[Nexlo]

Hey Voronenko,

thank you very much for your time and feedback.
You are the 2nd reference that I've received the past 2 days :)

...and yes, the example is really working! Im kind of doomed because it seems like my issue is somewhere else between WSL (Windows-Subsystem for Linux), docker and ansible-container - I have no clue atm.
Also, to have it mentioned more detailed, I am useing 'npiperelay tool', see here. Thats how my ansible-container is talking out of WSL (debian) to my Docker installed on Windows.

At least I know now that I really have an issue and its not about me^^ I'll try to investigate this further and will give feedback.

I've tested for your mentioned issue #937 but I am not effected by that.

Thanks again :)

[Voronenko]

Offtopic, I am also using docker on windows with 64G ram to offload builds from my linux notebook, but I just turned on setting for windows docker daemon to listen on tcp port 2375.

For WSL - just works w/o additional tools;
For external access - I need to use port forwarding, as it listens to localhost by default.

[Voronenko]

@Nexlo Can you update on your progress? If it is still issue - we need to transform it into reproducible scenario.

[Nexlo]

Hey Voronenko,

I've tested ansible-container & docker without the docker-relay to my windows.
Unfourtunatly I dont even get docker running in my WSL (debian). I end up on the error:

failed to register layer: Error processing tar file(exit status 1): invalid argument

during execution of a simple docker pull ubuntu:18.04. This seems to be related to mounts & filesystems (#34817). No clue..

With the relay active and using my docker on my windows machine everything is working fine - beside the fact, that no roles are applied during execution of ansible-container build, even if it says so.

Still after further research and investigation I have no clue what this is exactly about.

@Voronenko which informations are missing for reproduction?

Thanks for any help! :)

Further system information:

SumUp:

• Windows10 (docker 18.03.1-ce-win65)
• WSL (debian) // Windows SubSystem for Linux
• python 2.7.13 & pip 9.0.1
• docker client 18.03.1-ce
• docker server 18.03.1-ce
• docker relay (npiperelay tool)
• ansible-container 0.9.2

Output: docker version

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:16:02 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:14:13 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output: docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: <unknown>
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Kernel Version: 4.4.0-17134-Microsoft
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 5
Total Memory: 7.998GiB
Name: Nexlo-PC
ID: AP3B:W3UC:QWB3:56PO:FGF3:WAN6:LL7O:S27M:6ZNE:3B4J:CGLV:MKNE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support

[Voronenko]

@Nexlo

Well, issue might relay mostly to your system.
As I said previously, one of my boxes is Windows NUC with 32G ram. I am running docker subsystem on native windows for performance considerations.

From the WSL (based on ubuntu:xenial) I am able to successfully compile, for example this demo
https://github.com/softasap/sa-container-bootstrap/tree/master/box-example/alpine-34

with ansible-container.

Python 2.7.12
docker daemon on localhost:2375 w/o TLS
docker 18.03.1-ce-win65(17513)

so I would say root cause is somewhere near setup you have mostly docker relay (npiperelay tool) ?

Can you try to simplify your windows setup ?

[Nexlo]

Hey Voronenko,

I was able to exclude "npiperelay tool" by simply setting my environment var 'DOCKER_HOST'.
export DOCKER_HOST=tcp://127.0.0.1:2375
My docker pull ubuntu:18.04 works again.

Sadly now I run into (#602 ).
If I get it right, the solution for this issue is using the 'unix' socket, instead of the 'tcp' one.
export DOCKER_HOST=unix:///var/run/docker.sock

In Windows we dont have file sockets, right? see.
...so 'npipetool' is actually the solution to use a unix file socket and routeing the socket itself via npipe to docker in windows. Within useing the unix socket the conductor container is able to work proper and doesnt loop back by tcp://127.0.0.1. (?^^)

Also mentioned in docker docs, see here.

So, ya... I'd say I am running circles^^

Let me mention again, actually with npipetool enabled the execution is running without any errors.

 (master *)$ ansible-container build
Building Docker Engine context...
Starting Docker build of Ansible Container Conductor image (please be patient)...
Parsing conductor CLI args.
Docker™ daemon integration engine loaded. Build starting.       project=ansible-test
Building service...     project=ansible-test service=web
Applied role ordereddict([('role', 'my-new-role')]) from cache  role=ordereddict([('role', 'my-new-role')]) service=web
Build complete. service=web
All images successfully built.
Conductor terminated. Cleaning up.      command_rc=0 conductor_id=2d95f82b6f232d155a3c8c1567bd37938f8ecc345570c140506236f18a4f9106 save_container=False

I only have this (for me) solid reproduceable fail, that nothing really happens to the containers... :/ (2x in Ubuntu 16.04 & 1x in Debian 9 stretch)

Can you try to simplify your windows setup ?

Sorry, what do you mean exactly? My windows has a lot of stuff installed - but involved into this topic should be only the WSL and "Docker for Windows"?

Other than that, I've researched for a PowerShell script to kickstart WSL:

New-Item -ItemType directory -Path C:\WSL\Ubuntu
Invoke-WebRequest -Uri https://aka.ms/wsl-ubuntu-1604 -OutFile C:\WSL\Ubuntu.zip -UseBasicParsing
Expand-Archive C:\WSL\Ubuntu.zip C:\WSL\Ubuntu

I appreciate every further help or hints :)

Thanks!