Skip to content

Commit f5a9127

Browse files
committed
modernize the timeouts docs a little
listen_disabled_num hasn't mattered in ages since we default to maxconns_fast - also 32bit servers are rare now.
1 parent e89358e commit f5a9127

File tree

1 file changed

+15
-20
lines changed

1 file changed

+15
-20
lines changed

content/troubleshooting/timeouts.md

+15-20
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,17 @@ weight = 1
77
## Troubleshooting Timeouts
88

99
Client complaining about "timeout errors", but not sure how to track it down?
10-
Here's a simple utility for examining your situation.
10+
This page will guide you through the most common causes.
1111

12-
### First, check listen_disabled_num
12+
### Check your firewalls
1313

14-
Before you go ahead with troubleshooting, you'll want to telnet to your
15-
memcached instance and run `stats`, then look for "listen_disabled_num". This
16-
is a poorly named counter which describes how many times you've reached
17-
maxconns. Each time memcached hits maxconns it will delay new connections,
18-
which means you'll possibly get timeouts.
14+
You may need to tune or disable firewalls that exist between the client and
15+
memcached. Most firewalls track the connections in use (connection tracking)
16+
and hitting that limit will often cause timeout errors as existing TCP
17+
connections are dropped.
1918

20-
Also, disable or tune any firewalls you may have in the way.
19+
You will have to refer to your firewall vendor's documentation to figure this
20+
out.
2121

2222
### Then, carefully check the usual suspects
2323

@@ -27,18 +27,12 @@ memcached to disk periodically.
2727
Is the machine overloaded? 0% CPU idle with a load of 400 and memcached
2828
probably isn't getting enough CPU time. You can try `nice` or `renice`, or
2929
just run less on the machine. If you're severely overloaded on CPU, you might
30-
notice the mc_conn_tester below reporting very high wait times for `set`
30+
notice the "mc_conn_tester" below reporting very high wait times for `set`
3131
commands.
3232

33-
Is the memcached server 32bit? 32bit hosts have less memory available to the
34-
kernel for TCP sockets and friends. We've observed some odd behavior under
35-
large numbers of open sockets and high load with 32bit systems. Strongly
36-
consider going 64bit, as it may help some hard to trace problems go away,
37-
including segfaults due to the 2/4g memory limit.
38-
3933
### Next, mc_conn_tester.pl
4034

41-
Fetch this:
35+
Download this perl script:
4236

4337
https://www.memcached.org/files/mc_conn_tester.pl
4438

@@ -55,7 +49,7 @@ Options:
5549
[...etc...]
5650
```
5751

58-
This is a minimal utility for testing a quick routine with a memcached
52+
This is a small utility for testing a quick routine directly against a memcached
5953
instance. It will connect, attempt a couple sets, attempt a few gets, then loop and
6054
repeat.
6155

@@ -102,7 +96,7 @@ with `Fail:` as usual.
10296

10397
### You're probably dropping packets.
10498

105-
In most cases, where listen_disabled_num doesn't apply, you're likely dropping
99+
In many cases, you're likely dropping
106100
packets for some reason. Either a firewall is in the way and has run out of
107101
stateful tracking slots, or your network card or switch is dropping packets.
108102

@@ -113,7 +107,8 @@ Fail: (timeout: 1) (elapsed: 1.00145602) (conn: 0.00000000) (set: 0.00000000) (g
113107
```
114108

115109
... where `conn:` and the rest are all zero. So the test was not able to
116-
connect to memcached.
110+
connect to memcached. You may also see random numbers across conn/set/get
111+
mixed in with total failures.
117112

118113
On most systems SYN retries are 3 seconds, which is awfully long. Losing a
119114
single SYN packet will certainly mean a timeout. This is easily proven:
@@ -166,4 +161,4 @@ them here.
166161
### But your utility never fails!
167162

168163
Odds are good your client has a bug :( Try reaching out to the client author
169-
for help.
164+
for help. If not, please reach out to us for support via github discussions.

0 commit comments

Comments
 (0)