Replies: 4 comments
-
|
The logging suggests that you are using a WRONG password to access Redis. |
Beta Was this translation helpful? Give feedback.
-
|
If it is a WRONG password for Redis/MemoryDB, then why is it we were still able to list the contents of the mountpoint and even write to it (see the output of For example, when the MemoryDB password does expire (IAM authentication) we will get: but that wasn't the case when those WRONGPASS errors were appearing in the logs. More on the IAM authentication method for MemoryDB: From our discussions with AWS, while the IAM token is valid to use as a password within 15 minutes of obtaining it, once the connection to MemoryDB is established, it would remain active until the expiration of the IAM role of the credentials used. Additionally, we used an assumed role for which the credentials can last for a maximum of 12 hours because the instance profile credentials (from We are generating the MemoryDB (which requires AWS SigV4 signing of various parameters) password from the following steps:
Question: If JuiceFS does indeed try to establish new connections when read/writes increase, that could explain why the password is no longer valid, if more than 15 minutes have elapsed from the time the mount command was issued. Is there a way we can make JuiceFS spawn multiple Redis connections (that are kept alive, so that MemoryDB doesn't terminate them) at the start so that read/write processes could just reuse these opened connections, rather than opening new ones on demand? Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
The go-redis use a pool of connections to access MemoryDB, so the additional connections are created based on request/workload. A single connection can only handle a request/response at a time. We could ask go-redis for this feature, but even with that, I think it will still be easy to run into this problem. |
Beta Was this translation helpful? Give feedback.
-
|
Can JuiceFS implement native refresh-able IAM authentication using instance profile credentials, for both S3 and MemoryDB? |
Beta Was this translation helpful? Give feedback.
-
What happened:
We perform git clone of a large opensource codebase with hundres of thousands of files into a juicefs filesystem. The checkouts fail towards the end with the following git error:
fatal: cannot pread pack file: Input/output error fatal: fetch-pack: invalid index-pack outputIn the juicefs client logs, there are multiple occurrences of the following:
The
WRONGPASSerror seems to indicate that it is related to redis access but the juicefs mountpoint is fully accessible:and subsequently
head-ing the file shows the binary data writtenWhat you expected to happen:
Checkout should complete without errors.
How to reproduce it (as minimally and precisely as possible):
Clone an OSS project with many small files and long history (e.g. linux kernel, gcc or llvm) into a juicefs system with the mount flags detailed in the environment below.
Anything else we need to know?
Not sure how relevant but the EC2 clients that access the juicefs mount will be writing to the same volume but not the same set of files. Each client would write to its own subdirectory in the mount at any given time. Also, juicefs binary is running as root (via sudo)
Environment:
juicefs --version) or Hadoop Java SDK version:arm64 1.2.0+2024-06-18.873c47bself compiled as static binaryAWS Graviton3 EC2 instance running
juicefs version 1.2.0+2024-06-18.873c47bwith the following mount options:
juicefs mount --no-usage-report --writeback --no-syslog --background --cache-dir=/tmp/juicefs_cache --cache-size=1024000 --buffer-size=1024 --max-uploads=40 --log=/var/log/juicefs.log rediss://ciagent:****@clustercfg.abc.def.memorydb.region.amazonaws.comcat /etc/os-release):PRETTY_NAME="Ubuntu 24.04.1 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.1 LTS (Noble Numbat)" VERSION_CODENAME=noble ...uname -a):Linux ip-10-252-53-17 6.8.0-1014-aws #15-Ubuntu SMP Thu Aug 8 20:05:03 UTC 2024 aarch64 aarch64 aarch64 GNU/LinuxBeta Was this translation helpful? Give feedback.
All reactions