How to handle dragonfly_pipeline_queue_length that hangs forever (pipeline hangs)?
#3997
-
Problem
(below is a Python code, but I think it is irrelevant here as the problem is related only to the DragonflyDB internal pipeline execution) pipe = redis_client.pipeline()
key_count = 65536
for i in count():
if i == key_count:
break
pipe.hset(hash_name, mapping={str(i): str(j) for j in range(10_000)})
pipe.execute()results in
where Later in the log I have Versions env:
- name: DFLY_cache_mode
value: "false"
- name: DFLY_enable_heartbeat_eviction
value: "false"
- name: DFLY_dbnum
value: "1"
- name: DFLY_proactor_threads
value: "2"
- name: DFLY_dbfilename
value: dump
- name: DFLY_maxmemory
value: "5100273664"
- name: DFLY_logtostdout
value: "true"
- name: DFLY_aclfile
value: /dragonfly/snapshots/acl.file
- name: HEALTHCHECK_PORT
value: "9999"
image: docker.dragonflydb.io/dragonflydb/dragonfly:v1.24.0QuestionWhat to do if a pipeline hangs forever inside DragonflyDB after I call Update:A smaller pipeline worked. Now I have an assumption that for a too large pipeline DragonflyDB hangs forever silently. How can it abort the pipeline and raise an error to the client? RelatedI tried to search for "pipeline", "dragonfly_pipeline_queue_length", "Some commands are still being dispatched", but I found no related open issues or discussions. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
|
Can you please provide a minimal reproducible example? This snippet is not clear to me. Are you saying it happens all the time? |
Beta Was this translation helpful? Give feedback.
-
|
Now I can confirm that, unlike Redis, DragonflyDB cannot handle a pipeline that is above a certain length and queue hangs forever silently (no errors, no warnings in the log). MRE (minimal reproducible example)
import os
import redis
from redis import BlockingConnectionPool, Redis
r = Redis(
connection_pool=BlockingConnectionPool(
max_connections=10, # https://redis.uptrace.dev/guide/go-redis-debugging.html#connection-pool-size
timeout=60, # https://redis.uptrace.dev/guide/go-redis-debugging.html#timeouts
host=os.environ["REDIS_HOST"],
port=int(os.environ["REDIS_PORT"]),
password=os.environ["REDIS_PASSWORD"],
socket_keepalive=True,
retry_on_error=[
ConnectionError,
redis.exceptions.ConnectionError,
redis.exceptions.TimeoutError,
],
protocol=3,
)
)
r.ping()
pipe = r.pipeline()
for i in range(1_000_000):
pipe.hset("testhash", mapping={str(i): str(i)})
print("Executing...")
pipe.execute()
print(f"Executed. {r.hlen("testhash")=}")
r.delete("testhash") # Clean up.
r.close()DragonflyDB result
My attempts to make DragonflyDB workingI read https://www.dragonflydb.io/docs/managing-dragonfly/known-limitations but since in the cloud my pipeline hangs with 65536 commands but locally works, I do not think that we are under the mentioned limitations. I tried to set Maybe we are under a known limitation (that you know, but I do not know), but DragonflyDB must raise an error or at least say something in the log. Now DragonflyDB takes a large pipeline and does nothing silently. Redis resultNo problems. It takes seconds to run the same MRE on Redis VS DragonflyDB performance for large pipelinesRunning the MRE above for
The larger the pipeline, the slower DragonflyDB is compared to Redis. |
Beta Was this translation helpful? Give feedback.
-
|
I decided to run my MRE against every DragonflyDB version one by one.
@romange I hope that now you can bisect |
Beta Was this translation helpful? Give feedback.
-
|
Hello @alessio-locatelli, thank you very much for your efforts! I can explain why this occurs. When Dragonfly Connection replies, it writes to the output socket. Now, with the Python client, when you send 1 million requests, Dragonfly begins processing them and sending replies. However, the client doesn't read these responses until it finishes sending all 1 million requests. At some point, Dragonfly's networking buffers become saturated, and the connection stops processing requests. This situation results in a deadlock, as the client tries writing more requests without reading the replies. While it would be beneficial to have an option to configure the size of the client output buffer, allowing Dragonfly to further aggregate replies even with a blocked socket, this feature is not currently available. As a workaround, limiting batch sizes to reasonable numbers (around 100-1000) is recommended, as pipelining performance tends to plateau beyond these sizes. |
Beta Was this translation helpful? Give feedback.
-
|
I have not explained why there is a regression. The PR that causes it is #3152 Before that Dragonfly avoided the deadlock scenario above by reading all the input data from the socket into its memory buffers. Once it did that, the python client could proceed with consuming the replies and the deadlock did not happen. This PR introduced limits to that: Dragonfly stopped reading requests if it had more than K bytes in pipeline buffers per IO thread. To summarize: I must say it was not easy for me to debug the problem, we should probably add some log warnings or metrics if this scenario happens. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks again for putting an effort to present the problem in most clear way, and even explore whether this is a regression and when it appeared 🍻 |
Beta Was this translation helpful? Give feedback.
I have not explained why there is a regression. The PR that causes it is #3152
Before that Dragonfly avoided the deadlock scenario above by reading all the input data from the socket into its memory buffers. Once it did that, the python client could proceed with consuming the replies and the deadlock did not happen.
So Dragonly just read infinite number requests - it's a weakness that could potentially lead to OOM.
This PR introduced limits to that: Dragonfly stopped reading requests if it had more than K bytes in pipeline buffers per IO thread.
pipeline_buffer_limitis the flag that controls that and I just confirmed thatdocker run --network=host docker.dragonflydb.io/dragonflydb/dragon…