Connection reset by peer - recvfrom(2)

[mbajur]

Hey! I'm using Kamal for some time now and i'm loving it, however one thing is ocassionally popping up and I'm not able to deal with it in any way.

I'm running Kamal via github action and, for some deploy action runs, the kamal deploy command fails with ERROR (Errno::ECONNRESET): Exception while executing on host x.x.x.x: Connection reset by peer - recvfrom(2) error. From what I've managed to find on the internet, generally speaking that's because some command is taking too much time to complete without any output which makes ssh client consider the connection idle and kills the connection. In my case, i noticed that it's mostly happening when I'm having a change in my app's Gemfile which expires bundle install Dockerfile step and makes the process of deployment longer - which in turn makes the connection timing out.

I tried several ways of fixing that, including creating a ~/.ssh/config with content similar to that, i tried setting sshkit.pool_idle_timeout to something big like 1800, i tried adding verbose flag to my deploy commands to make them produce more output more often and i tried couple of other solutions i can't even remember now but still, the issue persists.

Does anyone have any idea what can i do to overcome this? Thank you in advance for any clues 🙏

---

Last couple of lines from the output:

 DEBUG [eb139e80] 	#27 [auth] yyy/xxx-build-cache:pull,push token for ghcr.io
 DEBUG [eb139e80] 	#27 DONE 0.0s
 DEBUG [eb139e80] 	
 DEBUG [eb139e80] 	#26 exporting cache to registry
 DEBUG [eb139e80] 	#26 writing layer sha256:0bdba67fb31514893f00c4502057ee9902a3b1d9216d5c33533c8453f81be11f 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:0f4b88c0a77456714eb5c607d4c906192f31cf907cda9c211ead67f92d92bd53
 DEBUG [eb139e80] 	#26 writing layer sha256:0f4b88c0a77456714eb5c607d4c906192f31cf907cda9c211ead67f92d92bd53 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:181a929aca279dacfa581184704dd3cc1a778d1404fa4ae768a2cf0223fabd89 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:225fdd30e1a3eb664decd3c5720a9500ef3ce275075e7c61bb904f0b4d4b2f46
 DEBUG [eb139e80] 	#26 writing layer sha256:225fdd30e1a3eb664decd3c5720a9500ef3ce275075e7c61bb904f0b4d4b2f46 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:244c32eb9e5eb1efaaf0a302ff37ba1955e5930c428c04bf10c5fcfb1f809544
 DEBUG [eb139e80] 	#26 writing layer sha256:244c32eb9e5eb1efaaf0a302ff37ba1955e5930c428c04bf10c5fcfb1f809544 0.3s done
 DEBUG [eb139e80] 	#26 writing layer sha256:273a9c7400d30c78542aff1b2c41a969cf0b443ae09a4f862fabd234f1e31972
 DEBUG [eb139e80] 	#26 writing layer sha256:273a9c7400d30c78542aff1b2c41a969cf0b443ae09a4f862fabd234f1e31972 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:40f1960d23e9faf0b21ae6c98d9872c5733e77129acffa647593df3180f52024
 DEBUG [eb139e80] 	#26 writing layer sha256:40f1960d23e9faf0b21ae6c98d9872c5733e77129acffa647593df3180f52024 0.3s done
 DEBUG [eb139e80] 	#26 writing layer sha256:49034c2773add59ea1c9d16a5f6b7c8587c3d55300174c44cb1c416f14388570
 DEBUG [eb139e80] 	#26 writing layer sha256:49034c2773add59ea1c9d16a5f6b7c8587c3d55300174c44cb1c416f14388570 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:4cec535da207f5d811fda01a346f5561fd2f77c2d09a080925b7455b84f0959e 0.0s done
 DEBUG [eb139e80] 	#26 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
 DEBUG [eb139e80] 	#26 writing layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:64594aaf68b9ad87731f869d2c6e0ab40d1dfae4831113e28c33a591e7898bfe 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:699c8a97647f5789e5850bcf1a3d5afe9730edb654e1ae0714d5bd198a67a3ed
 DEBUG [eb139e80] 	#26 writing layer sha256:699c8a97647f5789e5850bcf1a3d5afe9730edb654e1ae0714d5bd198a67a3ed 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:72cb2c87d3c7522e1871409dc04a8c9ae1ea9b4dc5187e80f65ac434f8d39cee
 DEBUG [eb139e80] 	#26 writing layer sha256:72cb2c87d3c7522e1871409dc04a8c9ae1ea9b4dc5187e80f65ac434f8d39cee 2.3s done
 DEBUG [eb139e80] 	#26 writing layer sha256:782595eaf17afceb87a3798d2feb7b231063a8581bcea3e75b1b24632437e29a
 DEBUG [eb139e80] 	#26 writing layer sha256:782595eaf17afceb87a3798d2feb7b231063a8581bcea3e75b1b24632437e29a 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:86cd158b89fde67a8a4c428a844985f930eba450ec3fde1c9ef5df3884128c62 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:9652ab75d067d96802e372513e3b19714d3574be70104050e5725571b285b9d1
 DEBUG [eb139e80] 	#26 writing layer sha256:9652ab75d067d96802e372513e3b19714d3574be70104050e5725571b285b9d1 0.3s done
 DEBUG [eb139e80] 	#26 writing layer sha256:996078afd01541e4ab745bdf7d69b4382cef12cb47196e3e701557d5a44561a4
 DEBUG [eb139e80] 	#26 writing layer sha256:996078afd01541e4ab745bdf7d69b4382cef12cb47196e3e701557d5a44561a4 1.8s done
 DEBUG [eb139e80] 	#26 writing layer sha256:a226e961cfaa2d1d171e429e9db314feec6201f5cba1487b20e7aee311e4a54f
 DEBUG [eb139e80] 	#26 writing layer sha256:a226e961cfaa2d1d171e429e9db314feec6201f5cba1487b20e7aee311e4a54f 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:bc59010be9c2d0b78b8eadc6f03970af3c1cbb1b48a596114a76ffb1e84967ad 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:c639f3f8e728d3e532b9c0f79c55bf43ea4230206fa3ac7a9eae35d61cef077a
 DEBUG [eb139e80] 	#26 writing layer sha256:c639f3f8e728d3e532b9c0f79c55bf43ea4230206fa3ac7a9eae35d61cef077a 0.4s done
 DEBUG [eb139e80] 	#26 writing layer sha256:cbeee2cd28f5677b50d5dde725a2660c1edbca56e25fc5a1fbc29dc68806fca8
 DEBUG [eb139e80] 	#26 writing layer sha256:cbeee2cd28f5677b50d5dde725a2660c1edbca56e25fc5a1fbc29dc68806fca8 0.1s done
 DEBUG [eb139e80] 	#26 writing layer sha256:faa2a10a81f17d75ef8bcc3aeec9681dddbc2d5c70ce394552ead371484fc295
 DEBUG [eb139e80] 	#26 writing layer sha256:faa2a10a81f17d75ef8bcc3aeec9681dddbc2d5c70ce394552ead371484fc295 0.3s done
 DEBUG [eb139e80] 	#26 writing config sha256:dbec83753ae4f6a687f43501a9b532515424083ae76dffdaa3ae1b81f9ec4794
 DEBUG [eb139e80] 	#26 writing config sha256:dbec83753ae4f6a687f43501a9b532515424083ae76dffdaa3ae1b81f9ec4794 0.4s done
 DEBUG [eb139e80] 	#26 writing cache manifest sha256:b665ca7e2d2e2854282fb1beb6831c47c377df7d25f9c5a1c3854576f643fbdd
 DEBUG [eb139e80] 	#26 preparing build cache for export 7.6s done
 DEBUG [eb139e80] 	#26 writing cache manifest sha256:b665ca7e2d2e2854282fb1beb6831c47c377df7d25f9c5a1c3854576f643fbdd 0.5s done
 DEBUG [eb139e80] 	#26 DONE 7.6s
 DEBUG [eb139e80] 	
 DEBUG [eb139e80] 	 3 warnings found (use docker --debug to expand):
 DEBUG [eb139e80] 	 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 41)
 DEBUG [eb139e80] 	 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 42)
 DEBUG [eb139e80] 	 - LegacyKeyValueFormat: "ENV key=value" should be used instead of legacy "ENV key value" format (line 43)
  INFO [eb139e80] Finished in 379.274 seconds with exit status 0 (successful).
D, [2025-01-02T08:32:32.934504 #2328] DEBUG -- socket[85c]: using encrypt-then-mac
D, [2025-01-02T08:32:32.934659 #2328] DEBUG -- socket[85c]: queueing packet nr 13 type 90 len 32
D, [2025-01-02T08:32:32.934728 #2328] DEBUG -- socket[85c]: read 84 bytes
D, [2025-01-02T08:32:32.934823 #2328] DEBUG -- socket[85c]: received packet nr 20 type 82 len 16
I, [2025-01-02T08:32:32.934858 #2328]  INFO -- net.ssh.connection.session[8d4]: global request failure
D, [2025-01-02T08:32:32.934877 #2328] DEBUG -- net.ssh.connection.keepalive[910]: keepalive response successful. Missed 0 keepalives
D, [2025-01-02T08:32:32.935002 #2328] DEBUG -- socket[85c]: read 0 bytes
D, [2025-01-02T08:32:32.935064 #2328] DEBUG -- socket[85c]: read 0 bytes
  Finished all in 382.7 seconds
  ERROR (Errno::ECONNRESET): Exception while executing on host x.x.x.x: Connection reset by peer - recvfrom(2)
Error: Process completed with exit code 1.

[tuladhar]

I'd suggest, try setting ssh.log_level: debug in Kamal config and share the deploy output.

[mbajur]

I updated my original question with sample output

[apatel-redi]

@tuladhar did you find a solution to this? I am having same issue. I am upgrading my ruby version to 3.4.1 and that is when this issue started occurring for me