Ingress for all chain nodes

[Anmol1696]

Overview

Currently we are using kubectl port-forward to get access to the RPC ports and expose the chains. We relay on this for both local as well as remote cluster on a k8s digital ocean.

Proposal

Inorder to unify the experince of everyone down the line, we dont expect people to use kubectl at all. The proposal here is to use ingress to expose the RPC ports for various chains. We can use the following ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: node-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx-example
  rules:
  - host: "rpc.osmosis-1.starship.one"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: osmosis-1-genesis
            port:
              number: 26657
  - host: "rest.osmosis-1.starship.one"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: osmosis-1-genesis
            port:
              number: 1317

There are a coupld of considerations to be made here:

• Do we want to expose all the validator nodes independently with ingress, rpc.<node>.<chain>.shuttle.one?
• New service per validator node?
• TLS need to end on ingress for remote

For local setup, need to map the hosts to the IPs and for remote need to create cloudflare rules with tls ending secrets.

[Anmol1696]

For local setup: https://mjpitz.com/blog/2020/10/21/local-ingress-domains-kind/

It might make more sense to divide the ingress problem into local and on cloud.

• For local we can have something like: https://mjpitz.com/blog/2020/10/21/local-ingress-domains-kind/ to perform all the routing
• For Cloud: We might want to use cloud provider based option

Port-forwarding can also be standardized with shell scripts to make the process smooth for users.

[Anmol1696]

In https://github.com/osmosis-labs/mesh-security-infra/pull/8, got the ingress working with following setting:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
    cert-manager.io/issuer: "cert-issuer"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - explorer.thestarship.io
      secretName: explorer.nginx-ingress-tls
    - hosts:
        - registry.thestarship.io
      secretName: registry.nginx-ingress-tls
    - hosts:
        - rest.mesh-1-genesis.thestarship.io
      secretName: rest.mesh-1-genesis.nginx-ingress-tls
    - hosts:
        - rpc.mesh-1-genesis.thestarship.io
      secretName: rpc.mesh-1-genesis.nginx-ingress-tls
  rules:
    - host: "explorer.thestarship.io"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: explorer
                port:
                  name: http
    - host: "registry.thestarship.io"
      http:
        paths:
          - pathType: ImplementationSpecific
            path: "/(.*)"
            backend:
              service:
                name: registry
                port:
                  name: http
    - host: "rest.mesh-1-genesis.thestarship.io"
      http:
        paths:
          - pathType: ImplementationSpecific
            path: "/(.*)"
            backend:
              service:
                name: mesh-1-genesis
                port:
                  name: rest
          - pathType: ImplementationSpecific
            path: "/faucet/(.*)"
            backend:
              service:
                name: mesh-1-genesis
                port:
                  name: faucet
          - pathType: ImplementationSpecific
            path: "/exposer/(.*)"
            backend:
              service:
                name: mesh-1-genesis
                port:
                  name: exposer
    - host: "rpc.mesh-1-genesis.thestarship.io"
      http:
        paths:
          - pathType: ImplementationSpecific
            path: "/(.*)"
            backend:
              service:
                name: mesh-1-genesis
                port:
                  name: rpc

Open Questions

Couple of open questions now?

1. Make part of starship config? or leave it to developers?

Should ingress be made part of the starship config file, or left to the overlays with generated yaml files.

Can look in config like:

ingress:
  enabled: true
  ingressClassName: nginx
  domain: "thestarship.io" ## specifies the domain that is configured
  ## maybe
  cert-manager:
    issuer: "cert-issuer"
  ## TODO: better name
  genesis-only: true ## if false, will create rules for all the validator nodes, exposing each one individually

From this following domains need to be predefined: explorer.<domain>, registry.<domain>, <chain-name>.<domain>...

Ingress.yaml file will keep getting longer, it would be really benificial to be able to generate ingress yaml file.

Why this should not be done?

• Too many pre-requists, devs can shoot themselves in foot.
• Might be dependent on underlying cloud cluster and implementations? (might have to test in multiple places).

Max capacity

How will ingress effect max number of validator nodes that we can setup down the line?
How many ingress rules are allowed
What is the additional overhead?

Should we make ingress-controller and cert-manager part of deployment

Maybe not since this can be cluster wide, we can not assume people just have the whole cluster for Starship only.
This is what makes ingress and certs even more tricky, we wont be able to enforce something from Starship.

[Anmol1696]

Cert-manager also configured with: https://cert-manager.io/docs/tutorials/acme/nginx-ingress/

[Anmol1696]

Fk. Ingress seems to be frozen. Newer paradigm of Gateway API seems to be hot thing.
https://kubernetes.io/docs/concepts/services-networking/gateway/

We should move away from ingress as well then, before we start.