Summary
The issue here is that we pass the secret content as one of the args via CLI. This issue may affect any of our charms that are using: Juju (>=3.0), Juju secrets and not correctly capturing and processing subprocess.CalledProcessError.
There are two points I can see we may log this command, in different files:
First, if we have an error during a secret handling, we will get subprocess.CalledProcessError, which will contain the CLI comand + all its args. This is going to be logged in any logging level. This exception, if not caught by the charm, will bubble up to the /var/log/juju/ logs and syslog journal. Now, on Ubuntu 22.04, these logs are protected with:
$ juju ssh -m controller 0 -- ls -la /var/log/juju/
total 224
drwxr-xr-x 2 syslog adm 4096 Jul 14 10:59 .
drwxrwxr-x 9 root syslog 4096 Jul 14 10:58 ..
-rw-r----- 1 syslog adm 20124 Jul 14 11:10 audit.log
-rw-r----- 1 syslog adm 110432 Jul 14 11:10 logsink.log
-rw-r----- 1 syslog adm 80783 Jul 14 11:06 machine-0.log
-rw-r----- 1 syslog adm 766 Jul 14 11:10 machine-lock.log
-rw-r--r-- 1 root root 0 Jul 14 10:59 slow-query.log
-rw-r----- 1 syslog adm 3732 Jul 14 11:10 unit-controller-0.log
Second, I believe certain audit setups may log terminal commands, which would result in this command being logged with its secrets. I am not sure if this is done on ubuntu security benchmarks, such as CIS hardening.
We should keep in mind these logs may be copied or even backed up. Which exposes it to more services in the user's environment (e.g. my case, CI runs in GH - although these are dummy password generated per test only).
I'd say passing secrets straight via CLI is not advised. I see some ways out:
- Redacting: we know which commands and which args represent secrets, then we can redact these. It would also mean capturing a
subprocess.CalledProcessError, redacting its content and reissuing the same type of exception; this will not cover the case auditd is set to log CLI commands, if that is a risk
- Temp files: secret-add, for example, can use a secret file instead, as can be seen here. However, if we use a file, we need to be sure we will correctly remove it later
- stdin: I am not sure it is accepted by secret-* commands, but generally, secrets are not shown on CLI whilst typing them; I am not sure if auditd would capture that stdin
In any case, I recommend to discuss with security team what is the best path here.
Severity Rationale
This is a CWE-532. Potentially, these secrets can lead to privilege escalation but Ubuntu default is to have logs only accessible to adm group users. I will not add CWE-284 for now.
Marking this issue as "Moderate", as this report is not presenting a clear way on how to get access to the logs themselves: either getting local access to an adm group user (e.g. ubuntu) or recovering logs stored on a 3rd party service.
Details
2024-07-12T14:27:58.0175389Z unit-opensearch-3: 14:27:53 ERROR unit.opensearch/3.juju-log certificates:11: Uncaught exception while in charm code:
2024-07-12T14:27:58.0175524Z Traceback (most recent call last):
2024-07-12T14:27:58.0175957Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3180, in _run
2024-07-12T14:27:58.0176165Z result = subprocess.run(args, **kwargs) # type: ignore
2024-07-12T14:27:58.0176381Z File "/usr/lib/python3.10/subprocess.py", line 526, in run
2024-07-12T14:27:58.0176561Z raise CalledProcessError(retcode, process.args,
2024-07-12T14:27:58.0177749Z subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-opensearch-3/secret-add', '--label', 'opensearch:app:app-admin', '--owner', 'application', 'keystore-password-ca=aUE...')' returned non-zero exit status 1.
2024-07-12T14:27:58.0177765Z
2024-07-12T14:27:58.0178023Z The above exception was the direct cause of the following exception:
2024-07-12T14:27:58.0178038Z
2024-07-12T14:27:58.0178168Z Traceback (most recent call last):
2024-07-12T14:27:58.0178588Z File "/var/lib/juju/agents/unit-opensearch-3/charm/./src/charm.py", line 213, in <module>
2024-07-12T14:27:58.0178724Z main(OpenSearchOperatorCharm)
2024-07-12T14:27:58.0179129Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 548, in main
2024-07-12T14:27:58.0179237Z manager.run()
2024-07-12T14:27:58.0179640Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 527, in run
2024-07-12T14:27:58.0179745Z self._emit()
2024-07-12T14:27:58.0180150Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 516, in _emit
2024-07-12T14:27:58.0180359Z _emit_charm_event(self.charm, self.dispatcher.event_name)
2024-07-12T14:27:58.0180840Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/main.py", line 147, in _emit_charm_event
2024-07-12T14:27:58.0180974Z event_to_emit.emit(*args, **kwargs)
2024-07-12T14:27:58.0181537Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 348, in emit
2024-07-12T14:27:58.0181656Z framework._emit(event)
2024-07-12T14:27:58.0182091Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 860, in _emit
2024-07-12T14:27:58.0182211Z self._reemit(event_path)
2024-07-12T14:27:58.0182659Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 950, in _reemit
2024-07-12T14:27:58.0182770Z custom_handler(event)
2024-07-12T14:27:58.0183492Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/tls_certificates_interface/v3/tls_certificates.py", line 1811, in _on_relation_changed
2024-07-12T14:27:58.0183743Z self.on.certificate_available.emit(
2024-07-12T14:27:58.0184173Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 348, in emit
2024-07-12T14:27:58.0184291Z framework._emit(event)
2024-07-12T14:27:58.0184717Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 860, in _emit
2024-07-12T14:27:58.0184887Z self._reemit(event_path)
2024-07-12T14:27:58.0186171Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/framework.py", line 950, in _reemit
2024-07-12T14:27:58.0186306Z custom_handler(event)
2024-07-12T14:27:58.0187043Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 209, in _on_certificate_available
2024-07-12T14:27:58.0187320Z self.store_new_ca(self.charm.secrets.get_object(scope, cert_type.val))
2024-07-12T14:27:58.0187942Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 444, in store_new_ca
2024-07-12T14:27:58.0188242Z self._create_keystore_pwd_if_not_exists(Scope.APP, CertType.APP_ADMIN, "ca")
2024-07-12T14:27:58.0188981Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_tls.py", line 432, in _create_keystore_pwd_if_not_exists
2024-07-12T14:27:58.0189119Z self.charm.secrets.put_object(
2024-07-12T14:27:58.0189738Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 359, in put_object
2024-07-12T14:27:58.0189944Z self._add_or_update_juju_secret(scope, key, value, merge)
2024-07-12T14:27:58.0190652Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 272, in _add_or_update_juju_secret
2024-07-12T14:27:58.0190815Z return self._add_juju_secret(scope, key, value)
2024-07-12T14:27:58.0191462Z File "/var/lib/juju/agents/unit-opensearch-3/charm/lib/charms/opensearch/v0/opensearch_secrets.py", line 228, in _add_juju_secret
2024-07-12T14:27:58.0191660Z secret = scope_obj.add_secret(safe_value, label=label)
2024-07-12T14:27:58.0192110Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 477, in add_secret
2024-07-12T14:27:58.0192241Z id = self._backend.secret_add(
2024-07-12T14:27:58.0192686Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3624, in secret_add
2024-07-12T14:27:58.0192948Z result = self._run('secret-add', *args, return_output=True)
2024-07-12T14:27:58.0193356Z File "/var/lib/juju/agents/unit-opensearch-3/charm/venv/ops/model.py", line 3182, in _run
2024-07-12T14:27:58.0193489Z raise ModelError(e.stderr) from e
2024-07-12T14:27:58.0193685Z ops.model.ModelError: ERROR this unit is not the leader
2024-07-12T14:27:58.0193692Z
From CI: https://github.com/canonical/opensearch-operator/actions/runs/9908987369/job/27376377521?pr=364
PoC
- Deploy anything with juju
- Run a dummy secret-add call that will fail
- See the uncaught subprocess error
Impact
Juju secrets are generally composed of private keys, passwords, etc; generally valuable credentials that, if leaked, will likely allow an attacker to get privileged access to its target or other targets in the environment.
phvalguima Reporter
Summary
The issue here is that we pass the secret content as one of the args via CLI. This issue may affect any of our charms that are using: Juju (>=3.0), Juju secrets and not correctly capturing and processing
subprocess.CalledProcessError.There are two points I can see we may log this command, in different files:
First, if we have an error during a secret handling, we will get
subprocess.CalledProcessError, which will contain the CLI comand + all its args. This is going to be logged in any logging level. This exception, if not caught by the charm, will bubble up to the/var/log/juju/logs and syslog journal. Now, on Ubuntu 22.04, these logs are protected with:Second, I believe certain audit setups may log terminal commands, which would result in this command being logged with its secrets. I am not sure if this is done on ubuntu security benchmarks, such as CIS hardening.
We should keep in mind these logs may be copied or even backed up. Which exposes it to more services in the user's environment (e.g. my case, CI runs in GH - although these are dummy password generated per test only).
I'd say passing secrets straight via CLI is not advised. I see some ways out:
subprocess.CalledProcessError, redacting its content and reissuing the same type of exception; this will not cover the caseauditdis set to log CLI commands, if that is a riskIn any case, I recommend to discuss with security team what is the best path here.
Severity Rationale
This is a CWE-532. Potentially, these secrets can lead to privilege escalation but Ubuntu default is to have logs only accessible to
admgroup users. I will not add CWE-284 for now.Marking this issue as "Moderate", as this report is not presenting a clear way on how to get access to the logs themselves: either getting local access to an
admgroup user (e.g. ubuntu) or recovering logs stored on a 3rd party service.Details
From CI: https://github.com/canonical/opensearch-operator/actions/runs/9908987369/job/27376377521?pr=364
PoC
Impact
Juju secrets are generally composed of private keys, passwords, etc; generally valuable credentials that, if leaked, will likely allow an attacker to get privileged access to its target or other targets in the environment.