Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis logs have errors using the default values.yaml from application version 3.6.0 chart version 2.6.0 #1764

Open
edison-vflow opened this issue Sep 20, 2024 · 2 comments

Comments

@edison-vflow
Copy link
Contributor

Describe the bug

After deploying Connaisseur application version 3.6.1 and chart version 2.6.1 on AWS EKS v1.30, the redis pod logs show errors

│ 123086:C 20 Sep 2024 15:28:25.062 # Failed opening the temp RDB file temp-123086.rdb (in server root dir /data) for saving: Permission denied           
│ 1:M 20 Sep 2024 15:28:25.163 # Background saving error

`
The full log looks something like this

1:M 20 Sep 2024 15:32:43.073 * 1 changes in 3600 seconds. Saving...                                                                                     │
│ 1:M 20 Sep 2024 15:32:43.073 * Background saving started by pid 123439                                                                                  │
│ 123439:C 20 Sep 2024 15:32:43.074 # Failed opening the temp RDB file temp-123439.rdb (in server root dir /data) for saving: Permission denied           │
│ 1:M 20 Sep 2024 15:32:43.174 # Background saving error                                                                                                  │
│ 1:M 20 Sep 2024 15:32:45.488 . 6 clients connected (0 replicas), 1060744 bytes in use                                                                   │
│ 1:M 20 Sep 2024 15:32:48.863 - Accepted 127.0.0.1:50826                                                                                                 │
│ 1:M 20 Sep 2024 15:32:48.864 - Client closed connection id=18276 addr=127.0.0.1:50826 laddr=127.0.0.1:6380 fd=20 name= age=0 idle=0 flags=N db=0 sub=0  │
│ 1:M 20 Sep 2024 15:32:48.894 - Accepted 127.0.0.1:50838                                                                                                 │
│ 1:M 20 Sep 2024 15:32:48.896 - Client closed connection id=18277 addr=127.0.0.1:50838 laddr=127.0.0.1:6380 fd=20 name= age=0 idle=0 flags=N db=0 sub=0  │
│ 1:M 20 Sep 2024 15:32:49.009 * 1 changes in 3600 seconds. Saving...                                                                                     │
│ 1:M 20 Sep 2024 15:32:49.010 * Background saving started by pid 123452                                                                                  │
│ 123452:C 20 Sep 2024 15:32:49.011 # Failed opening the temp RDB file temp-123452.rdb (in server root dir /data) for saving: Permission denied           │
│ 1:M 20 Sep 2024 15:32:49.111 # Background saving error                                                                                                  │
│ 1:M 20 Sep 2024 15:32:50.519 . 6 clients connected (0 replicas), 1060744 bytes in use                                                                   │
│ 1:M 20 Sep 2024 15:32:55.056 * 1 changes in 3600 seconds. Saving...                                                                                     │
│ 1:M 20 Sep 2024 15:32:55.056 * Background saving started by pid 123453                                                                                  │
│ 123453:C 20 Sep 2024 15:32:55.057 # Failed opening the temp RDB file temp-123453.rdb (in server root dir /data) for saving: Permission denied           │
│ 1:M 20 Sep 2024 15:32:55.157 # Background saving error

Expected behavior

  • This new version of Connaisseur uses redis caching.
    We were using chart version 2.0 application version 3.0.0 which did not have redis.
    The expectation is that for the new versions with redis, the logs will be clean without errors.

Optional: To reproduce

To reproduce, install Connaisseur application version 3.6.1 and chart version 2.6.1 on AWS EKS v1.30

Optional: Versions (please complete the following information as relevant):

  • OS: Amazon Linux
  • Kubernetes Cluster: EKS 1.30
  • Notary Server:
  • Container registry: containerd
  • Connaisseur: chart 2.6.1 application 3.6.1
  • Other:

Optional: Additional context

  • Using the default values for Connaisseur application version 3.6.1 and chart version 2.6.1 results in the redis log errors
  • To try resolve the failure in the logsFailed opening the temp RDB file temp-123086.rdb (in server root dir /data) for saving: Permission denied , we also tried adding below to values.yaml in order for redis not to run under restricted security context.
    This also didnt work
redis:
    securityContext: {}
@edison-vflow
Copy link
Contributor Author

cc @phbelitz @chrysogonus

@phbelitz
Copy link
Member

@edison-vflow hmm. unfortunately i can't replicate this in a minikube setup, but i did some digging. the /data directory inside the redis container is owned by a redis user (with id 999) and our default security context sets permissions to 10001. from that alone it should be expected for the error to occurr, but since you removed the security context and the error still persisted ... things don't make sense.

could you try to set the runAsUser and runAsGroup fields for redis to 999, uninstall Connaisseur and the reinstall? in theory that should do the trick, if not, i'd be curious what user is running inside the redis pod (you can do kubectl exec -it -n connaisseur connaisseur-redis-<random-id> -- bash to get inside the pod and run id to get the user id). maybe also check the permissions on the /data directory, they should belong to redis:redis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants