Keep Track of Dynamic TCP Sessions to Allow Better Scoped Network Policy

[mike-carrigan24]

Describe the problem/challenge you have
Certain TCP protocols (ex. FTP, SFTP, NFSv3) use dynamic ports for certain services (NFSv3 for example uses them for NLM, MNT, and STATD). These are necessary for these protocols to operate properly. Due to the dynamic nature of these ports, scoping Network Policy to allow these protocols is difficult. Accounting for dynamic ports often requries provisioning all TCP highports or, in some instances, all TCP ports. This requires Network Policies to be provisioned with a significantly wider scope than would be desired.

Describe the solution you'd like
Ideally, when a dynamic session is stood up as a result of one of these protocols, this session is evaluated with respect to the protocol that spawned it instead of being evaluated as a new session. This would allow Network Policy for these protocols to be better scoped and easier to manage.

Anything else you would like to add?
While these protocols sometimes have the ability to set a specific port range for dynamic services, they can be difficult to configure, especially at scale.

Also, I believe Open vSwitch connection tracking has the ability to do this with the "ct_tp_dst" field, however I am not super familiar with it so I may be mistaken.

[antoninbas]

Thanks for submitting this.
My understanding is that the recommendation when using NFSv3 with a firewall (which would include Antrea NetworkPolicy implementation) is to assign static ports for these services which use dynamic ports by default?

Ideally, when a dynamic session is stood up as a result of one of these protocols, this session is evaluated with respect to the protocol that spawned it instead of being evaluated as a new session.

I believe that ports are allocated and "distributed" out-of-band by something called rpcbind? This happens at the app layer, not at the transport layer. As a result, Antrea would have no knowledge of which port have been assigned for the dynamic NFS services. The only way to know them would be to intercept these packets, parse them and extract that information. In. other words, from the perspective of Antrea / OVS / conntrack, there is nothing that ties the new sessions (for NLM, MNT, and STATD) to your original NFS session (presumably using port 2049) or RPC session (presumably using port 111).

ct_tp_dst is the original destination port prior to NAT being performed. For example, when you access a K8s Service using the ClusterIP and Service port, ct_tp_dst will "preserve" the value of the original Service port, while the destination port in the packet is rewritten (DNAT) to the target port (i.e. port on which the Pod Endpoints are listening). We are still talking about the same connection here, not a separate but "related" connection (again, "related" here is application-specific - in this case, NFS-specific).

I know that Suricata has some support for NFS, so a L7NetworkPolicy may be an option here (we would need to add support for NFS). But I don't know the extent of that support, nor how the individual services are handled. I believe that NFSv3 is quite ancient as well, and that NFSv4 is recommended?

My take is pretty much the same for FTP. Note that passive FTP was listed as one of the motivating use cases for port range support in K8s NetworkPolicies: https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/2079-network-policy-port-range/README.md#story-3---containerized-passive-ftp-server. Here as well, we could add FTP support to L7NetworkPolicy given that Suricata supports it.

[mike-carrigan24]

Thanks @antoninbas,

My understanding is that the recommendation when using NFSv3 with a firewall (which would include Antrea NetworkPolicy implementation) is to assign static ports for these services which use dynamic ports by default?

You are correct that for the majority of these protocols, the recommendation is to assign static ports for these services. Unfortunately, this can be sometimes difficult to do in practice and may sometimes cause unnecessary pain for users configuring these services.

I believe that ports are allocated and "distributed" out-of-band by something called rpcbind? This happens at the app layer, not at the transport layer. As a result, Antrea would have no knowledge of which port have been assigned for the dynamic NFS services. The only way to know them would be to intercept these packets, parse them and extract that information. In. other words, from the perspective of Antrea / OVS / conntrack, there is nothing that ties the new sessions (for NLM, MNT, and STATD) to your original NFS session (presumably using port 2049) or RPC session (presumably using port 111).

ct_tp_dst is the original destination port prior to NAT being performed. For example, when you access a K8s Service using the ClusterIP and Service port, ct_tp_dst will "preserve" the value of the original Service port, while the destination port in the packet is rewritten (DNAT) to the target port (i.e. port on which the Pod Endpoints are listening). We are still talking about the same connection here, not a separate but "related" connection (again, "related" here is application-specific - in this case, NFS-specific).

This information is really helpful and makes sense as to why this may not be feasible/possible.

The use of Suricata L7NetworkPolicy to inspect this traffic is interesting and seems similar to other implementations of stateful firewalls for these session types (ex. Fortinet session helpers. If L7NetworkPolicy were implemented for FTP or similar protocols, would the goal be to enforce additional constraints on the connectivity or to specifically handle dynamic sessions?

Thanks!

[antoninbas]

The use of Suricata L7NetworkPolicy to inspect this traffic is interesting and seems similar to other implementations of stateful firewalls for these session types (ex. Fortinet session helpers. If L7NetworkPolicy were implemented for FTP or similar protocols, would the goal be to enforce additional constraints on the connectivity or to specifically handle dynamic sessions?

I am not very familiar with Suricata FTP support. My expectation is that we would be able to restrict traffic to FTP control and data packets, regardless of which destination port is used for data packets (assuming FTP passive mode). I have to say the FTP support in Suricata is not super well documented.

From the Antrea side, a policy could look like this:

apiVersion: crd.antrea.io/v1beta1
kind: NetworkPolicy
metadata:
  name: ingress-ftp
spec:
  priority: 5
  tier: application
  appliedTo:
    - podSelector:
        matchLabels:
          app: ftp-server
  ingress:
    - name: allow-ftp
      action: Allow
      from:
        - podSelector:
            matchLabels:
              app: client
      l7Protocols:
        - ftp:
            fileName: foobar.txt # optionally filter by filename (pattern supported?)
    - name: drop-other   # Drop all other inbound traffic (non FTP)
      action: Drop

We are pretty much a passthrough to Suricata for the l7 rules, and we would generate Suricata rules based on that policy.
I assume that Suricata maintains a state machine, and that once the control connection is established, the data connection will be allowed for the specified file paths (if applicable), and not before then.

I am even less familiar with NFS support in Suricata :P. If you are interested, and willing to help test it, we could start with FTP support?

[mike-carrigan24]

Hey @antoninbas,

That makes sense, thanks!

Yeah I would be interested/willing to help test FTP support. I can work on getting Suricata set up in my environment. Please let me know how I can help!