Does this still work?

[nelg]

Hi,

I've had issues with this not working, although it used to work.

It seems that when it deletes the default route:

# switch the default route to eth1
ip route del default dev eth0

The nat instance then looses all internet connectivity.

Does this still work for you?

[nelg]

To get my one working, I ended up making the changes as per nelg@e4a0b33

[DesAWSume]

Hi nelg

If I just wanna setup a Linux 2 NAT instance, don't wanna IaC to provision all other infra. which commands I should run to be able to have a Amazon Linux 2 NAT working?

Thanks in advance.

[szromek]

@nelg I am experiencing the same issue as you did and your fix seems to solve the problem. Could you provide @int128 with a PR that could be tested, merged and published to Terraform registry, so the whole module would be operational again?

[nelg]

@nelg I am experiencing the same issue as you did and your fix seems to solve the problem. Could you provide @int128 with a PR that could be tested, merged and published to Terraform registry, so the whole module would be operational again?

Sure, will do

[nelg]

Here is the PR #37

[arjitj2]

This issue and your fix solved 5+ hours of debugging work for me. Thank you and I hope it gets merged soon.

[int128]

It seems NAT connection is lost after the NAT instance is rebooted.

ip route del default dev eth0 command is needed to change the default route to eth1 to fix the source IP, because an EIP of eth0 will be changed when the instance is recreated by Auto Scaling Group.

I noticed the route table is broken after reboot as follows:

## When an instance is created

ssm-user@ip-172-18-138-43 bin]$ ip ro
default via 172.18.128.1 dev eth1 metric 10001
169.254.169.254 dev eth0
172.18.128.0/20 dev eth0 proto kernel scope link src 172.18.138.43
172.18.128.0/20 dev eth1 proto kernel scope link src 172.18.132.145

ssm-user@ip-172-18-138-43 bin]$ sudo reboot

## After reboot

ssm-user@ip-172-18-138-43 bin]$ ip ro
default via 172.18.128.1 dev eth0
default via 172.18.128.1 dev eth1 metric 10001
169.254.169.254 dev eth0
172.18.128.0/20 dev eth0 proto kernel scope link src 172.18.138.43
172.18.128.0/20 dev eth1 proto kernel scope link src 172.18.132.145

Finally I could fixed this problem by removing the config of eth0:

sudo rm /etc/sysconfig/network-scripts/ifcfg-eth0

I will add it to the script.

[int128]

I think #42 resolved the issue. Please let me know if the issue still occurs.

[nelg]

I have tested version 2.0.1 release on terraform registry, and it doesn't work.. still have eth0 as the default route, so the instance can't send traffic to the internet.

which version should I test?

[nelg]

I'm quite keen to get a version of this published on the registry that works. Rather than me publishing a copy of your one, can we work together to get it working, if you have time sometime in the next couple of weeks.

My solution is working for us, but it's not perfect and ends up with 2 default routes, and two interfaces in the same subnet.
The two ENI's attached, 1 has a public IP and a private IP, the other just has a private IP. We have to route out the one that has a public IP to get to the internet.

[JulianCBC]

Yeah, this latest fix is bogus.

I built this module from the example in README.md and this is my NAT instance's networking details after a reboot:

sh-4.2$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:e8:a4:c9:de:f6 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:33:c7:03:41:92 brd ff:ff:ff:ff:ff:ff
    inet 10.0.128.88/24 brd 10.0.128.255 scope global dynamic eth1
       valid_lft 3401sec preferred_lft 3401sec
    inet6 fe80::433:c7ff:fe03:4192/64 scope link
       valid_lft forever preferred_lft forever
sh-4.2$ ip route
default via 10.0.128.1 dev eth1 metric 10001
10.0.128.0/24 dev eth1 proto kernel scope link src 10.0.128.88
sh-4.2$ sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  anywhere             anywhere

Long story short, it appears this module is broken, I tried downgrading to 2.0.0, but after that I couldn't even connect to the EC2 instance via SSM to debug this.

[nelg]

When you tried using it, did you have an eip assigned to the nat instance? It has to be created externally to the module, then passed in. I had problems when I didn't assign an EIP.

…

On Sat, 23 Jul 2022, 2:53 AM Julian Calaby, ***@***.***> wrote: Yeah, this latest fix is bogus. I built this module from the example in README.md and this is my NAT instance's networking details *after a reboot*: sh-4.2$ ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 06:e8:a4:c9:de:f6 brd ff:ff:ff:ff:ff:ff 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 06:33:c7:03:41:92 brd ff:ff:ff:ff:ff:ff inet 10.0.128.88/24 brd 10.0.128.255 scope global dynamic eth1 valid_lft 3401sec preferred_lft 3401sec inet6 fe80::433:c7ff:fe03:4192/64 scope link valid_lft forever preferred_lft forever sh-4.2$ ip route default via 10.0.128.1 dev eth1 metric 1000110.0.128.0/24 dev eth1 proto kernel scope link src 10.0.128.88 sh-4.2$ sudo iptables -t nat -L Chain PREROUTING (policy ACCEPT) target prot opt source destination Chain INPUT (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- anywhere anywhere Long story short, it appears this module is broken, I tried downgrading to 2.0.0, but after that I couldn't even connect to the EC2 instance via SSM to debug this. — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAY4PN3FAU7YM5G4MY4ERLVVKYXHANCNFSM4XXI7VQA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[JulianCBC]

Yep, had to reorganise stuff so I could, but I did have an EIP on the NAT instance when I did my first round of testing with version 2.0.1.

My initial testing of this module failed to produce a working internet connection on the NAT instance or an instance on a private subnet, so it looks like something's misconfigured or missing. For the record, it's possible that the "something missing" is entirely my fault.

My understanding is that NAT gateways work like this: private host -> network interface -> NAT -> route table -> internet. So therefore how we get to the internet shouldn't matter, which makes the act of deleting the eth0 configuration script and therefore leaving that interface unconfigured after a reboot seem bogus as it shouldn't matter. That said, all my previous hacking has used separate interfaces for the input and output sides of the NAT gateway, so it's quite possible it'll all work on one interface and leaving eth0 unconfigured is correct.

I suspect that I've made a mistake somewhere here, but I also know that the NAT gateway should have had internet access in my testing, and the fact that it doesn't is concerning. I'm going to try a couple of other options then maybe return to this depending on the outcome. fck-nat seems promising if I can figure out a simple way to Terraformise it's setup.

(Another thing that stood out is that the ENI handling needs to be smarter: we should be able to detect whether it's already connected or somehow still in-use (e.g. after an instance is terminated) and respond appropriately.)