Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow passing tunnel token via the file system. #645

Closed
Cyb3r-Jak3 opened this issue May 14, 2022 · 31 comments · May be fixed by #1316
Closed

Allow passing tunnel token via the file system. #645

Cyb3r-Jak3 opened this issue May 14, 2022 · 31 comments · May be fixed by #1316

Comments

@Cyb3r-Jak3
Copy link

Describe the feature you'd like
I currently use named tunnels and run in docker containers. I would like to be able to use docker secrets with the containers. This is currently not possible because docker secrets are passed as a file and there is no way to read the file. It would be nice to be able to have --secret-file option, i.e. cloudflared tunnel run --token --secret-file or cloudflared tunnel run --token </path/to/token/file>.

Describe alternatives you've considered
Typically, you use cat in containers to read the files, but the rootless container does not have the cat executable so unable to use secrets that way

Additional context
Similar to #232.

I would be happy to work on this if it is something that is worth implementing.

@Cyb3r-Jak3 Cyb3r-Jak3 added Needs clarification Unable to move forward on the reported issue Type: Feature Request A big idea that would be split into smaller pieces labels May 14, 2022
@nmldiegues
Copy link
Contributor

You can see in:

→ cloudflared tunnel run --help | grep token
  --token value                                The Tunnel token. When provided along with credentials, this will take precedence. [$TUNNEL_TOKEN]

that it is also possible to pass the token as an environment variable. Those should be manageable as secrets in docker (compose, k8s, etc...) easily.

If you really want a file, then you can:

cloudflared tunnel token --cred-file /tmp/mysecret.json <TUNNEL>

And then you have /tmp/mysecret.json to put in your secret storage and to use.

All in all, I think there are enough ways to do this, and I would skip yet another way. Free free to open back and discuss further.

@nmldiegues nmldiegues removed Needs clarification Unable to move forward on the reported issue Type: Feature Request A big idea that would be split into smaller pieces labels May 16, 2022
@Cyb3r-Jak3
Copy link
Author

Cyb3r-Jak3 commented May 16, 2022

The purpose of docker secrets is to avoid putting secrets as environment variables. From my understanding of token --cred-file /tmp/mysecret.json <TUNNEL> is that I need to have to still have an origin cert (#603) which defeats remote managed tunnels.

I admit that this is an edge case, but the goal is to be able to use remote managed tunnels via docker secrets.

@nmldiegues
Copy link
Contributor

The purpose of docker secrets is to avoid putting secrets as environment variables.
I admit that this is an edge case, but the goal is to be able to use remote managed tunnels via docker secrets.

But that's my whole point: docker secrets can be injected into the container via both files as well as environment variables. The values still come from the secret storage. The way they are made available to the container is what differs.

@Cyb3r-Jak3
Copy link
Author

How do you inject secrets as environment variables? From the documentation I have seen, it is only possible as a file

As an example, I have this compose file with a file called token.secret in the same directory that contains a remote tunnel token.

version: "3.9"

secrets:
  cloudflare_token:
    file: "token.secret"

services:
  cloudflared:
    image: cloudflare/cloudflared:2022.5.1
    command: tunnel run
    secrets:
      - cloudflare_token
    environment:
      - TUNNEL_TOKEN=/run/secrets/cloudflare_token

Running docker-compose up gets

F:\git\cloudflare-secrets> docker-compose.exe up
[+] Running 1/1
 - Container cloudflare-secrets-cloudflared-1  Recreated                                                                                                                                                         1.4s 
Attaching to cloudflare-secrets-cloudflared-1
cloudflare-secrets-cloudflared-1  | Provided Tunnel token is not valid.
cloudflare-secrets-cloudflared-1  | See 'cloudflared tunnel run --help'.
cloudflare-secrets-cloudflared-1 exited with code 255

If I replace with

version: "3.9"

secrets:
  cloudflare_token:
    file: "token.secret"

services:
  cloudflared:
    image: cloudflare/cloudflared:2022.5.1
    command: tunnel run --token /run/secrets/cloudflare_token
    secrets:
      - cloudflare_token

I get the same output

@nmldiegues
Copy link
Contributor

The problem is that you are setting the TUNNEL_TOKEN To the file path, not to the token itself.
The env var needs to have the actual string contents of the token.

Maybe you want to use something like https://builtin.com/software-engineering-perspectives/docker-compose-environment-variables instead?

@Cyb3r-Jak3
Copy link
Author

The env var needs to have the actual string contents of the token.

The way that docker secrets work is that the secret info, in this case, the token, is passed to the container via a file.

In terms of Docker Swarm services, a secret is a blob of data, such as a password, SSH private key, SSL certificate, or another piece of data that should not be transmitted over a network or stored unencrypted in a Dockerfile or in your application’s source code
When you grant a newly-created or running service access to a secret, the decrypted secret is mounted into the container in an in-memory filesystem. The location of the mount point within the container defaults to /run/secrets/<secret_name>

Other images typically use an environment variable that ends in _FILE ie with mysql MYSQL_ROOT_PASSWORD_FILE instead of MYSQL_ROOT_PASSWORD_FILE

@beefstew809
Copy link

To reiterate, docker secrets allows us to have encrypted secrets at transit and at rest and makes it easier to keep tokens out of version control since it is not in the .env file. To the best of my (limited) knowledge, environment variables do not give us the same benefits of encryption at transit and rest. In my opinion, unless there is a more secure way to pass the token to a docker container, this should not be closed and should be worked on as a security enhancement.

@nmldiegues
Copy link
Contributor

In my opinion, unless there is a more secure way to pass the token to a docker container, this should not be closed and should be worked on as a security enhancement.

I want to reiterate on my comment in #645 (comment) above, where I said:

If you really want a file, then you can:

cloudflared tunnel token --cred-file /tmp/mysecret.json <TUNNEL>

And then you have /tmp/mysecret.json to put in your secret storage and to use

So there's a way to rely on a file, which is why I closed the issue, regardless of the discussion around the environment variables.

@Cyb3r-Jak3
Copy link
Author

If you really want a file, then you can:

cloudflared tunnel token --cred-file /tmp/mysecret.json <TUNNEL>

And then you have /tmp/mysecret.json to put in your secret storage and to use.

Doesn't this require having a cert.pem and a tunnel credentials file in the docker container?

With a docker-compose of

version: "3.9"

secrets:
  tunnel_file:
    file: "tunnel_info"

services:
  cloudflared:
    image: cloudflare/cloudflared:2022.5.1
    command: tunnel token --cred-file /run/secrets/tunnel_file 8be772befd147a8df540aae0fa15c047
    secrets:
      - tunnel_file
    volumes:
      - ./cloudflared:/etc/cloudflared/:ro

With ./cloudflared having a config.yml, cert.pem and tunnel credentials file. I can get it to work

Right now to get a connector online you run docker run cloudflare/cloudflared:2022.5.1 tunnel --no-autoupdate run --token <token id> where the token appears to be a base64 encoded JSON string that has a tunnel credential file information. The desired behavior is to be able to pass that same connector token is as a file.

@nmldiegues
Copy link
Contributor

Doesn't this require having a cert.pem and a tunnel credentials file in the docker container?

This would be done once per tunnel only, as part of your setup that would generate/handle/manage the secret so that it would be available in the future for your container to use.

Right now to get a connector online you run docker run cloudflare/cloudflared:2022.5.1 tunnel --no-autoupdate run --token where the token appears to be a base64 encoded JSON string that has a tunnel credential file information. The desired behavior is to be able to pass that same connector token is as a file.

Let me make it extra clear:

  1. you create the Tunnel in the UI
  2. you run cloudflared tunnel token --cred-file /some/path/tunnel.json <TUNNEL>
  3. now you make /some/path/tunnel.json available to your container via whatever volume
  4. the container command should be: tunnel --credentials-file /volume/path/tunnel.json run <TUNNEL>

Steps 1/2 can be skipped if you create the Tunnel via the CLI and manage it that way.

No need for the cert.pem.

@Cyb3r-Jak3
Copy link
Author

This would be done once per tunnel only, as part of your setup that would generate/handle/manage the secret so that it would be available in the future for your container to use.

Ah, this is the step that I was missing. I was attempting to do it all in the container 😅. Thanks for staying with me on this

@etienne-napoleone
Copy link

Thanks for the explanation, got it working this way.
Would have been awesome to also just make it a little bit more convenient? at least at the docker image level ...

@sderungs
Copy link

Following the conversation above and specially @nmldiegues last answer I understood that cert.pem would only be required to generate the tunnel.json file but that after that it's not needed anymore and only tunnel.json needs to be mounted into the Docker container.

However, if I do so the container complains at startup about the missing cert.pem.

2023-02-19T22:46:27Z INF Cannot determine default origin certificate path. No file cert.pem in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared] originCertPath=  
2023-02-19T22:46:27Z ERR You need to specify the origin certificate path by specifying the origincert option in the configuration file, or set TUNNEL_ORIGIN_CERT environment variable. See https://developers.cloudflare.com/argo-tunnel/reference/service/ for more information. originCertPath=
error parsing tunnel ID: Error locating origin cert: client didn't specify origincert path

Apologies if this question might sound dumb - but why does it still ask for the cert.pem? Shouldn't the tunnel.json have all the credential information?

After all, if I would not run the service in a stack but just as a single container with docker run cloudflare/cloudflared:latest tunnel --no-autoupdate run --token someTokenHere it would also not request that and also not generate an API token in my profile. I'm a bit at a loss here 😅

@ProfessorCha0s
Copy link

For anyone running into this later and a bit confused like I was, here is what you need to do (Replace <TUNNEL> with the name of your Tunnel as shown in the cloudflare dashboard)

  1. From your system where docker is running (not the container) you will need cloudflared installed (instructions outside the scope of this conversation, but easy to find). Then you can pull your token and temporarily store as a json file by running
    cloudflared tunnel token --cred-file /some/path/tunneltoken.json <TUNNEL>

  2. Read that file into your docker secret
    docker secret create cloudflareTunnelToken /some/path/tunneltoken.json

  3. (Optional) Now that you have it saved as a secret you can delete the tunneltoken.json file if you want.

  4. Finally you can use it in compose as in this example

version: '3.9'

secrets:
  cloudflareTunnelToken:
    external: true

services:
  cloudflared:
    image: cloudflare/cloudflared
    restart: unless-stopped    
    secrets:
      - cloudflareTunnelToken
    command: tunnel --credentials-file /run/secrets/cloudflareTunnelToken run <TUNNEL>

@josekommisar
Copy link

josekommisar commented Aug 25, 2023

I will post my experience to encourage others not to worry about the process.
I previously tried many combinations with environments or secrets to the Cloudflared container with no success.
Many hours, I should point out.
I performed the steps gathered from this thread, and BANG is working.
To expand a little on the steps of @ProfessorCha0s:
1- I made a micro cloud VM running Ubuntu and installed Cloudflared binary via ssh session, per instructions on the Cloudflare Zero Trust homepage. You will need the name of your tunnel.
2- After installing Cloudflared, you must run cloudflared tunnel login on the ssh session to generate a configuration file with your login details. The login is made via a URL you copy and run on your browser. The cert.pem file is the result output in the server. Otherwise, you cannot perform the next step.
3- After you have made login (and cert.pem generated), you can run the command cloudflared tunnel token --cred-file /some/path/tunneltoken.json <TUNNEL>
4- And finally, as pointed out by @ProfessorCha0s, you can use or copy the JSON file to another server with docker swarm, create a secret, deploy the cloudflared container with his compose file, and have your tunnel working, without worrying about disclosing your token.
I've done it that way to change the tunnel to another server with more resources, where I will run my apps.
My compose.yml file

secrets:
  tunneltoken:
    external: true

networks:
  cloud-overlay:
    external: true

services:
  cloudflared:
    image: cloudflare/cloudflared
    secrets:
      - tunneltoken
    command: tunnel --credentials-file /run/secrets/tunneltoken run my-cloudflare-tunnel
    deploy:
      replicas: 2
    networks:
      - cloud-overlay

I thank you all for the discussion and solution.

@catharsis71
Copy link

For anyone running into this later and a bit confused like I was

I'm still confused because this doesn't work

# docker logs cloudflared
2023-08-31T22:12:06Z ERR Cannot determine default origin certificate path. No file cert.pem in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]. You need to specify the origin certificate path by specifying the origincert option in the configuration file, or set TUNNEL_ORIGIN_CERT environment variable originCertPath=
error parsing tunnel ID: Error locating origin cert: client didn't specify origincert path
2023-08-31T22:12:07Z ERR Cannot determine default origin certificate path. No file cert.pem in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]. You need to specify the origin certificate path by specifying the origincert option in the configuration file, or set TUNNEL_ORIGIN_CERT environment variable originCertPath=
error parsing tunnel ID: Error locating origin cert: client didn't specify origincert path
2023-08-31T22:12:08Z ERR Cannot determine default origin certificate path. No file cert.pem in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]. You need to specify the origin certificate path by specifying the origincert option in the configuration file, or set TUNNEL_ORIGIN_CERT environment variable originCertPath=
error parsing tunnel ID: Error locating origin cert: client didn't specify origincert path
2023-08-31T22:12:09Z ERR Cannot determine default origin certificate path. No file cert.pem in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]. You need to specify the origin certificate path by specifying the origincert option in the configuration file, or set TUNNEL_ORIGIN_CERT environment variable originCertPath=
error parsing tunnel ID: Error locating origin cert: client didn't specify origincert path
2023-08-31T22:12:10Z ERR Cannot determine default origin certificate path. No file cert.pem in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]. You need to specify the origin certificate path by specifying the origincert option in the configuration file, or set TUNNEL_ORIGIN_CERT environment variable originCertPath=
error parsing tunnel ID: Error locating origin cert: client didn't specify origincert path

@catharsis71
Copy link

I will post my experience to encourage others not to worry about the process.

the Docker container still complains about lack of an origin certificate & will not run

how do you get the origin certificate into the container?

@josekommisar
Copy link

josekommisar commented Sep 1, 2023

Did you generate the cert.perm file and the .JSON cred-file and pass the JSON to the container?
It was the only way it worked for me.
I have no deep understanding of the method it exposes the string at run time. Why it works in the stack deploy on CLi ‘—-token abcdefg7654321zxcvbn’, and why it doesn’t with ‘--token $VAR-STRING’. The cloudflared image does not pass the variable/secret to the container at run time in a format it can work with, different implementation looks like. It expects a JSON format.
The route discussed here was through first making a normal package/process/binary install and generation of the auth chain needed. In my case using a temporary machine/server to just obtain the JSON credentials.

@artbird309
Copy link

@catharsis71 I had the same issue but I figured out that it was a permissions issue for the secret file. I was not able to figure out what uid the cloudflared user was running so I change my permissions to 404 (read for owner and others) on the tunneltoken secret file and it worked fine without the cert.pem.

@j0sh
Copy link

j0sh commented Sep 28, 2023

Adding another vote for being able to read the token directly from a file, rather than going through the steps to get the cert, JSON, UUID, etc - doing so kind of defeats the purpose of the one-shot remote setup process.

For reference, it is a convention for Docker env vars to have a _FILE suffix specifically for this type of use case [1], eg

TUNNEL_TOKEN_FILE=/run/secret/cf_tunnel_token

would basically translate to something like:

TUNNEL_TOKEN=$(cat /run/secret/cf_tunnel_token)

although of course this is just a convention and this type of translation isn't built into Docker - it needs explicit application support, hence this issue.

[1] Source - very last note on the bottom of the page

image

@oluceps
Copy link

oluceps commented Jan 12, 2024

EnvironmentFile option could pass secrets to app with env vars.

; env
TEST=1
...
EnvironmentFile=/some/env
ExecStart="echo $TEST"

It outputs 1 in stdin.

@ZJohnAsZ
Copy link

ZJohnAsZ commented Jan 23, 2024

Hi, I came across this discussion while trying to pass the token as a secret in my docker-compose file. Should I install and use docker swarm, just for this?

So far I went through the different steps to get the json file with the docker exec -it cloudflared way, so no need for vm or installing the cloudflared app directly on the OS.

1- Create a temporary folder in $DOCKERDIR and give it chmod 777, as to not get permissions issues when generating the json file
2- Add a temporary volume to the compose file
volumes:
- $DOCKERDIR/cf_token:/cf_token/
3- Start the cloudflared container
4- Generate the cert.pem at the container level
docker exec -it cloudflared cloudflared tunnel login

3- Generate and save the json cred-file
docker exec -it cloudflared cloudflared tunnel token --cred-file /cf_token/cf_tunnel_token.json <TUNNEL_NAME>

4- Not sure what to do from there install docker swarm to use it as a secret or load it from the json file with the tunnel --credentials-file /volume/path/tunnel.json run <TUNNEL> command.

I did not use docker swarm yet, would it change much to my single host orchestration to docker swarm init on my local personal server and then create secrets with all my files, instead of having a secrets folder of files? Thank's to anyone willing to reply.

@j0sh
Copy link

j0sh commented Jan 23, 2024

Hi, I came across this discussion while trying to pass the token as a secret in my docker-compose file. Should I install and use docker swarm, just for this?

@ZJohnAsZ Not to derail this topic but this gist shows my approach with Docker Compose. Basically invoke cloudflared in a shell (via multi-stage busybox build) and read the secrets file into a TUNNEL_TOKEN env var prior to execution.

@ZJohnAsZ
Copy link

@ZJohnAsZ Not to derail this topic but this gist shows my approach with Docker Compose. Basically invoke cloudflared in a shell (via multi-stage busybox build) and read the secrets file into a TUNNEL_TOKEN env var prior to execution.

@j0sh Thank you for the reply. I am fairly new to docker/containers, I will explore the multi-stage way. New things to learn. :)

@adam-stamand
Copy link

Thanks @j0sh for the workaround, it worked well for me.

+1 to address this the right way with a TUNNEL_TOKEN_FILE; not sure of the author's hesitation there.

@bfg100k
Copy link

bfg100k commented Jul 11, 2024

+1 for TUNNEL_TOKEN_FILE

@mbratchikov
Copy link

Using Google Secret Manager with Google Kubernetes Engine has limitations - secrets can be only read from file. It will be very nice to read tunnel_token from the file.
+1 for TUNNEL_TOKEN_FILE

@mikhail5555
Copy link

mikhail5555 commented Jul 19, 2024

If anyone stumbes on this thread while researching it for NixOS, you can use the TUNNEL_TOKEN env variable to easily inject it into the service:

{
  pkgs,
  config,
  ...
}: {
  age.secrets.cloudflared_env = {
    file = ./secrets/cloudflared_env.age;
    owner = "cloudflared";
    group = "cloudflared";
  };

  # https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes
  boot.kernel.sysctl."net.core.rmem_max" = 7500000;
  boot.kernel.sysctl."net.core.wmem_max" = 7500000;

  users.users.cloudflared = {
    group = "cloudflared";
    isSystemUser = true;
  };
  users.groups.cloudflared = {};

  systemd.services.cloudflared = {
    wantedBy = ["multi-user.target"];

    requires = ["network-online.target"];
    after = ["network-online.target"];

    serviceConfig = {
      Restart = "always";
      EnvironmentFile = config.age.secrets.cloudflared_env.path;
      User = "cloudflared";
      Group = "cloudflared";
    };

    script = ''
      ${pkgs.unstable.cloudflared}/bin/cloudflared --no-autoupdate tunnel run
    '';
  };
}

with cloudflared_env looking as:

TUNNEL_TOKEN=<your actual token>

So this technically is not needed for this, even though it would still be a nice addition to be able to 'easily' inject it without utilizing the EnvironmentFile.

@tetious
Copy link

tetious commented Jul 27, 2024

I'm uncertain why there is so much push-back to making this more ergonomic by following the "standards" established by other docker containers. Is there something I'm missing that makes adding this painful or insecure?

The provided workarounds are serviceable, but all have drawbacks and are varying degrees of unfriendly to use.

Please reconsider making this nicer by implementing TUNNEL_TOKEN_FILE.

@DenisMir
Copy link

DenisMir commented Aug 17, 2024

WTF it is 2024 and there is still no support for TUNNEL_TOKEN_FILE. This must be a joke. Wake up Cloudflare and follow the conventions!

@ehellman
Copy link

This should really be reopened. Cloudflare tunnels are widely used and not having TUNNEL_TOKEN_FILE available for docker secrets is a security concern. The current workarounds still make it more of a hassle to set up than it needs to be. I run maybe 60 services in my swarm, cloudflared is the ONLY one where I have to use 1Password Secrets directly for the token inside an .env file instead of using docker secrets like every other one supports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.