Problem: ModuleNotFoundError: No module named 'ansible' after upgrade to latest Semaphore release

[gitmarge]

Issue

Upgraded Semaphore with the latest .deb package 2.12.5 on an Ubuntu server virtual machine after several months of running on older release without issues.

After installation with

sudo dpkg -i semaphore.deb

and service start, the website comes up and tasks can be executed, but fail immediately with the following message:

4:28:39 PM Task 163646 added to queue
4:28:42 PM Started: 163646
4:28:42 PM Run TaskRunner with template: Ping
4:28:42 PM Preparing: 163646
4:28:42 PM installing static inventory
4:28:42 PM Traceback (most recent call last):
4:28:42 PM File "/home/sema/.local/bin/ansible-playbook", line 5, in
4:28:42 PM from ansible.cli.playbook import main
4:28:42 PM ModuleNotFoundError: No module named 'ansible'
4:28:42 PM Running app failed: exit status 1

Updating Python environment (3.10), all packages and Ansible via pip did not help solve the issue, neither did a reinstall of the Ansible package.

Semaphore is configured to run as a systemd service without a Python venv. Service comes up without issues.

Impact

Ansible (task execution)

Installation method

Package

Database

MySQL

Browser

Chrome, Microsoft Edge

Semaphore Version

v2.12.5

Ansible Version

ansible [core 2.17.8]
  config file = None
  configured module search path = ['/home/sema/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/sema/.local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/sema/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/sema/.local/bin/ansible
  python version = 3.10.12 (main, Jan 17 2025, 14:35:34) [GCC 11.4.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

Logs & errors

No response

Manual installation - system information

5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Configuration

systemd service file:

[Unit]
Description=Ansible Semaphore
Documentation=https://docs.ansible-semaphore.com/
Wants=network-online.target
After=network-online.target
ConditionPathExists=/usr/bin/semaphore
ConditionPathExists=/home/sema/semaphore/config.json

[Service]
User=sema
Group=sema
ExecStart=/usr/bin/semaphore server --config /home/sema/semaphore/config.json
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=10s

# automatically upgrade python modules at service start
ExecStartPre=/bin/bash -c 'python3 -m pip install --upgrade --user -r ~/requirements.txt'
# automatically upgrade ansible collection at service start
ExecStartPre=/bin/bash -c 'ansible-galaxy collection install --upgrade -r ~/requirements.yml'
# automatically upgrade ansibel roles at service start
ExecStartPre=/bin/bash -c 'ansible-galaxy role install --force -r ~/requirements.yml'

# environment filled with PATH, PYTHONPATH, Semaphore tmp_path
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/sema/.local/bin"
Environment="PYTHONPATH=/home/sema/.local/lib/python3.10/site-packages"
WorkingDirectory=/home/sema/semaphore/tmp

StandardOutput=journal
StandardError=journal
SyslogIdentifier=semaphore

[Install]
WantedBy=multi-user.target

Additional information

Running a simple playbook with different modules manually via the command line with user sema works perfectly fine

ansible-playbook -i ./inv pb.yml

TASK [Gathering Facts] *******************************************************************************************************************************************************************************************************************************************************
ok: [myhost]

TASK [Execute ping module (no ICMP ping)] ************************************************************************************************************************************************************************************************************************************
ok: [myhost]

TASK [Check service file state] **********************************************************************************************************************************************************************************************************************************************
ok: [myhost]

TASK [Get service names from target file] ************************************************************************************************************************************************************************************************************************************
changed: [myhost]

PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************************
myhost : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

One thing I noticed is that if I change the line

Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/sema/.local/bin"

in the systemd service file to

Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:~/.local/bin"

Semaphore crashes with the following message in the journalctl log, indicating the home directory is not set correctly with ~, however even if it finds the binaries the issue still stands:

Feb 10 16:11:25 semaphore-host semaphore[7052]: Server is running
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Task 163638 added to queue"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Task 163640 added to queue"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Task 163639 added to queue"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Task 163641 added to queue"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Task 163642 added to queue"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Set resource locker with TaskRunner 163638"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Task 163638 removed from queue"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Stopped running TaskRunner 163638"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=info msg="Release resource locker with TaskRunner 163638"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=warning msg="read |0: file already closed" error="Failed to read TaskRunner output"
Feb 10 16:15:00 semaphore-host semaphore[7052]: time="2025-02-10T16:15:00+01:00" level=warning msg="read |0: file already closed" error="Failed to read TaskRunner output"
Feb 10 16:15:00 semaphore-host semaphore[7052]: panic: exec: "ansible-playbook": executable file not found in $PATH
Feb 10 16:15:00 semaphore-host semaphore[7052]: goroutine 44 [running]:
Feb 10 16:15:00 semaphore-host semaphore[7052]: github.com/semaphoreui/semaphore/db_lib.AnsiblePlaybook.RunPlaybook({0x1e, {0x8, {0xc00003ce70, 0x17}, 0x6, {0xc0000ce2a0, 0x1f}, {0x0, 0x0}, 0xb, ...}, ...}, ...)
Feb 10 16:15:00 semaphore-host semaphore[7052]: #11/home/runner/work/semaphore/semaphore/db_lib/AnsiblePlaybook.go:49 +0x207
Feb 10 16:15:00 semaphore-host semaphore[7052]: github.com/semaphoreui/semaphore/db_lib.(*AnsibleApp).Run(0x0?, {{0xc0002e0500, 0xb, 0x10}, {0x0, 0x0, 0x0}, 0xc00061ba40, {0xbd67e0, 0xc000582440}, ...})
Feb 10 16:15:00 semaphore-host semaphore[7052]: #11/home/runner/work/semaphore/semaphore/db_lib/AnsibleApp.go:65 +0xca
Feb 10 16:15:00 semaphore-host semaphore[7052]: github.com/semaphoreui/semaphore/services/tasks.(*LocalJob).Run(0xc0002c0008, {0x0, 0x0}, 0x0, {0x0, 0x0})
Feb 10 16:15:00 semaphore-host semaphore[7052]: #11/home/runner/work/semaphore/semaphore/services/tasks/LocalJob.go:512 +0xa3f
Feb 10 16:15:00 semaphore-host semaphore[7052]: github.com/semaphoreui/semaphore/services/tasks.(*TaskRunner).run(0xc000607208)
Feb 10 16:15:00 semaphore-host semaphore[7052]: #11/home/runner/work/semaphore/semaphore/services/tasks/TaskRunner.go:163 +0x699
Feb 10 16:15:00 semaphore-host semaphore[7052]: created by github.com/semaphoreui/semaphore/services/tasks.(*TaskPool).Run in goroutine 9
Feb 10 16:15:00 semaphore-host semaphore[7052]: #11/home/runner/work/semaphore/semaphore/services/tasks/TaskPool.go:186 +0x4b6
Feb 10 16:15:00 semaphore-host systemd[1]: ansible-semaphore.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 10 16:15:00 semaphore-host systemd[1]: ansible-semaphore.service: Failed with result 'exit-code'.
Feb 10 16:15:00 semaphore-host systemd[1]: ansible-semaphore.service: Consumed 1.452s CPU time.
Feb 10 16:15:10 semaphore-host systemd[1]: ansible-semaphore.service: Scheduled restart job, restart counter is at 2.
Feb 10 16:15:10 semaphore-host systemd[1]: Stopped Ansible Semaphore.
Feb 10 16:15:10 semaphore-host systemd[1]: ansible-semaphore.service: Consumed 1.452s CPU time.

[fiftin]

Hi @gitmarge

I think problem in Python venv environment variable: VIRTUAL_ENV. Start from 2.11 you need manually specify which environment variable will be available for ansible-playbook. Configuration option:

• environment variable: SEMAPHORE_FORWARDED_ENV_VARS=["VIRTUAL_ENV"]
• config.json field: "forwarded_env_vars": ["VIRTUAL_ENV"].

This done for security reason.

---

It is a problem, I agree.

I implemented page for setup of Semaphore: https://semaphoreui.com/install/binary/2_12/config.

It is not complete. I improving it by the users feedback. And I think we need block "VENV settings". Thank you.

[gitmarge]

Hello @fiftin

Thanks for the reply.

However I don't quite understand what would need to be done. I do not use a Python venv, does Semaphore need to be explicitely configured for this use case?

The setup page is helpful for certain options, but entering a value in the SEMAPHORE_FORWARDED_ENV_VARS field does not function correctly and the config file has no name for this entry:

{
  "mysql": {
    "host": "localhost",
    "name": "semaphore",
    "user": "root"
  },
  "dialect": "mysql",
  "": "VIRTUAL_ENV",
  "cookie_hash": "REAqSjpYKrgKP6+ohQng+whzK7NrNhtVyUyJgSHJ8e0=",
  "cookie_encryption": "UAMKWkTHNTuO2FUqKLnfN/5yi5Tk8wuQ60X2YNscY2c=",
  "access_key_encryption": "hURaeKA6yXRtaDFFO4Ko2PQ1EYcOt45dJm/hG0GqBuI="
}

Or maybe I'm doing something wrong.

But I'm still not quite sure what would need to be configured and where I would configure it. I put the environment variable SEMAPHORE_FORWARDED_ENV_VARS=["VIRTUAL_ENV"] into my systemd service file and added the config.json field "forwarded_env_vars": ["VIRTUAL_ENV"], however that prevented Semaphore from starting up with the message:

panic: invalid character 'V' looking for beginning of value

Also adding just one of these lines did not work.

However I'm sure I am not intended to put these lines into the config and systemd service as is?

Thank you very much for your help, I'm looking forward to some more insight and clarification.

[fiftin]

Hi @gitmarge

My mistake, looks like you use Ansible installed with --user flag. It doesn't work for Semaphore.

You should use system installed Ansible or use VENV.

[gitmarge]

I see, thanks for coming back to me.

Was this a recent change? Because it was working fine with Ansible installed with the --user flag, until the upgrade to the latest Semaphore version.

In the meantime, I have rolled back the server to the state before the upgrade and it is working again.

Once I upgrade Semaphore, I will make sure to install Ansible via apt and test functionality beforehand.

Thanks again.

[fiftin]

@gitmarge hm, it is really strange... I don't know. Can you check environment variables like PYTHON_* in your system? Perhaps you can try to add them to SEMAPHORE_FORWARDED_ENV_VARS.