corridor-init-forwarding not idempotent

[adrelanos]

corridor-init-forwarding is currently not idempotent. Re-running corridor-init-forwarding will fail. This is problematic, because then the systemd service cannot be successfully restarted.

user@corridor:~$ sudo service corridor-init-forwarding status
● corridor-init-forwarding.service - corridor's forwarding
   Loaded: loaded (/lib/systemd/system/corridor-init-forwarding.service; enabled)
   Active: failed (Result: exit-code) since Tue 2016-07-05 19:49:35 CEST; 8min ago
  Process: 11067 ExecStart=/usr/sbin/corridor-init-forwarding (code=exited, status=1/FAILURE)
 Main PID: 11067 (code=exited, status=1/FAILURE)

Jul 05 19:49:35 corridor corridor-init-forwarding[11067]: iptables: Chain already exists.
Jul 05 19:49:35 corridor systemd[1]: corridor-init-forwarding.service: main process exited, code=exited, status=1/FAILURE
Jul 05 19:49:35 corridor systemd[1]: Failed to start corridor's forwarding.
Jul 05 19:49:35 corridor systemd[1]: Unit corridor-init-forwarding.service entered failed state.

user@corridor:~$ sudo sh -x -e /usr/sbin/corridor-init-forwarding
+ corridor-load-ipset-relays --init
+ corridor-load-ipset-logged --init
+ iptables -w -N CORRIDOR
iptables: Chain already exists

Could you make corridor-init-forwarding idempotent please? (That would help with Debian packaging. #10)

[rustybird]

Added corridor-stop-{forwarding,snat} (also as ExecStop= actions for corridor-init-{forwarding,snat}.service) to allow stopping and restarting. Note that RemainAfterExit=yes is used, so at least systemctl start was already idempotent.

[adrelanos]

This is better. Not idempotent, though. Perhaps good enough.

When manually running corridor-stop-forwarding, next time systemd will register an unsuccessful stop.

user@corridor:~$ sudo service corridor-init-forwarding status
● corridor-init-forwarding.service - corridor's forwarding
   Loaded: loaded (/lib/systemd/system/corridor-init-forwarding.service; enabled)
   Active: active (exited) since Wed 2016-07-06 21:49:28 CEST; 5min ago
  Process: 4522 ExecStop=/usr/sbin/corridor-stop-forwarding (code=exited, status=0/SUCCESS)
  Process: 4529 ExecStart=/usr/sbin/corridor-init-forwarding (code=exited, status=0/SUCCESS)
 Main PID: 4529 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/corridor-init-forwarding.service

Jul 06 21:49:28 corridor corridor-init-forwarding[4529]: net.ipv4.ip_forward = 1
Jul 06 21:49:28 corridor corridor-init-forwarding[4529]: sysctl: cannot stat /proc/sys/net/ipv6/ip_forward: No such file or directory
Jul 06 21:49:28 corridor systemd[1]: Started corridor's forwarding.

user@corridor:~$ sudo /usr/sbin/corridor-stop-forwarding
net.ipv4.ip_forward = 0

user@corridor:~$ sudo service corridor-init-forwarding restart
user@corridor:~$ sudo service corridor-init-forwarding status
● corridor-init-forwarding.service - corridor's forwarding
   Loaded: loaded (/lib/systemd/system/corridor-init-forwarding.service; enabled)
   Active: active (exited) since Wed 2016-07-06 21:54:58 CEST; 3s ago
  Process: 4592 ExecStop=/usr/sbin/corridor-stop-forwarding (code=exited, status=2)
  Process: 4597 ExecStart=/usr/sbin/corridor-init-forwarding (code=exited, status=0/SUCCESS)
 Main PID: 4597 (code=exited, status=0/SUCCESS)

Jul 06 21:54:58 corridor corridor-init-forwarding[4597]: net.ipv4.ip_forward = 1
Jul 06 21:54:58 corridor corridor-init-forwarding[4597]: sysctl: cannot stat /proc/sys/net/ipv6/ip_forward: No such file or directory
Jul 06 21:54:58 corridor systemd[1]: Started corridor's forwarding.

I mean this line: Process: 4592 ExecStop=/usr/sbin/corridor-stop-forwarding (code=exited, status=2). Dunno if this is an issue.

---

Stopping init forwarding.

user@corridor:~$ sudo service corridor-init-forwarding stop

user@corridor:~$ sudo service corridor-init-forwarding status
● corridor-init-forwarding.service - corridor's forwarding
   Loaded: loaded (/lib/systemd/system/corridor-init-forwarding.service; enabled)
   Active: inactive (dead) since Wed 2016-07-06 21:57:29 CEST; 989ms ago
  Process: 4644 ExecStop=/usr/sbin/corridor-stop-forwarding (code=exited, status=0/SUCCESS)
  Process: 4597 ExecStart=/usr/sbin/corridor-init-forwarding (code=exited, status=0/SUCCESS)
 Main PID: 4597 (code=exited, status=0/SUCCESS)

Jul 06 21:54:58 corridor corridor-init-forwarding[4597]: net.ipv4.ip_forward = 1
Jul 06 21:54:58 corridor corridor-init-forwarding[4597]: sysctl: cannot stat /proc/sys/net/ipv6/ip_forward: No such file or directory
Jul 06 21:54:58 corridor systemd[1]: Started corridor's forwarding.
Jul 06 21:57:29 corridor systemd[1]: Stopping corridor's forwarding...
Jul 06 21:57:29 corridor corridor-stop-forwarding[4644]: net.ipv4.ip_forward = 0
Jul 06 21:57:29 corridor systemd[1]: Stopped corridor's forwarding.

So far so good.

Now lets manually start init forwarding.

user@corridor:~$ sudo /usr/sbin/corridor-init-forwarding
net.ipv4.ip_forward = 1
sysctl: cannot stat /proc/sys/net/ipv6/ip_forward: No such file or directory

Next time we try to start init forwarding from systemd it will fail.

user@corridor:~$ sudo service corridor-init-forwarding start
Job for corridor-init-forwarding.service failed. See 'systemctl status corridor-init-forwarding.service' and 'journalctl -xn' for details.

And then it is impossible to recover from this by systemd stop / start / restart.

user@corridor:~$ sudo service corridor-init-forwarding stop

user@corridor:~$ sudo service corridor-init-forwarding status
● corridor-init-forwarding.service - corridor's forwarding
   Loaded: loaded (/lib/systemd/system/corridor-init-forwarding.service; enabled)
   Active: failed (Result: exit-code) since Wed 2016-07-06 22:00:32 CEST; 44s ago
  Process: 4644 ExecStop=/usr/sbin/corridor-stop-forwarding (code=exited, status=0/SUCCESS)
  Process: 4883 ExecStart=/usr/sbin/corridor-init-forwarding (code=exited, status=1/FAILURE)
 Main PID: 4883 (code=exited, status=1/FAILURE)

Jul 06 22:00:32 corridor corridor-init-forwarding[4883]: iptables: Chain already exists.
Jul 06 22:00:32 corridor systemd[1]: corridor-init-forwarding.service: main process exited, code=exited, status=1/FAILURE
Jul 06 22:00:32 corridor systemd[1]: Failed to start corridor's forwarding.
Jul 06 22:00:32 corridor systemd[1]: Unit corridor-init-forwarding.service entered failed state.
Jul 06 22:01:11 corridor systemd[1]: Stopped corridor's forwarding.
user@corridor:~$ sudo service corridor-init-forwarding start
Job for corridor-init-forwarding.service failed. See 'systemctl status corridor-init-forwarding.service' and 'journalctl -xn' for details.

user@corridor:~$ sudo service corridor-init-forwarding start
Job for corridor-init-forwarding.service failed. See 'systemctl status corridor-init-forwarding.service' and 'journalctl -xn' for details.

user@corridor:~$ sudo service corridor-init-forwarding status
● corridor-init-forwarding.service - corridor's forwarding
   Loaded: loaded (/lib/systemd/system/corridor-init-forwarding.service; enabled)
   Active: failed (Result: exit-code) since Wed 2016-07-06 22:03:44 CEST; 1s ago
  Process: 4644 ExecStop=/usr/sbin/corridor-stop-forwarding (code=exited, status=0/SUCCESS)
  Process: 4976 ExecStart=/usr/sbin/corridor-init-forwarding (code=exited, status=1/FAILURE)
 Main PID: 4976 (code=exited, status=1/FAILURE)

Jul 06 22:03:44 corridor corridor-init-forwarding[4976]: iptables: Chain already exists.
Jul 06 22:03:44 corridor systemd[1]: corridor-init-forwarding.service: main process exited, code=exited, status=1/FAILURE
Jul 06 22:03:44 corridor systemd[1]: Failed to start corridor's forwarding.
Jul 06 22:03:44 corridor systemd[1]: Unit corridor-init-forwarding.service entered failed state.

Not sure if this is an issue, if that would happen with other systemd daemons / manual interaction also.

[rustybird]

Hmm, yes. I even have a commit that makes -stop idempotent, but it takes 15-20 lines to do it right. I'm on the fence whether this is too much for such a corner case; e.g. it seems fairly intuitive that the problem described can be fixed by running the inverse action in the same way as the original action (either manually or via systemctl).