Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Globalconnections overloadmanager #6308

Closed
wants to merge 3 commits into from
Closed

Globalconnections overloadmanager #6308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
621ca8f
Add new bootstrap flag for connection limit
seth-epps Mar 28, 2024
4666ff0
[Minor][Overload Manager] Allow setting global connection limit resou…
seth-epps Mar 28, 2024
08ba591
pr num update
seth-epps Mar 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions apis/projectcontour/v1alpha1/contourconfig.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,13 @@ type EnvoyConfig struct {
// Network holds various configurable Envoy network values.
// +optional
Network *NetworkParameters `json:"network,omitempty"`

// OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
// the envoy overload manager actions, such as global connection limits, enforced.
//
// This is disbled by default
// +optional
OMEnforcedHealth *HealthConfig `json:"om_enforced_health,omitempty"`
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be camel case

}

// DebugConfig contains Contour specific troubleshooting options.
Expand Down
18 changes: 18 additions & 0 deletions apis/projectcontour/v1alpha1/contourconfig_helpers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,10 @@
return fmt.Errorf("invalid envoy configuration: %v", err)
}

if err := healthEndpointsInConflict(e.Metrics, e.Health, e.OMEnforcedHealth); err != nil {
return fmt.Errorf("invalid envoy configuration: %v", err)
}

if err := e.Logging.Validate(); err != nil {
return err
}
Expand Down Expand Up @@ -313,3 +317,17 @@
}
return nil
}

// healthEndpointsInConflict returns an error if the same address and port are used between the overload manager enforced health listener and metrics
// _or_ the stats listener. Since the metrics would be validated against the stats listener already we only need to check that the overload manager
// health listener is not in conflict with either metrics or stats
func healthEndpointsInConflict(metrics *MetricsConfig, stats *HealthConfig, omEnforcedHealth *HealthConfig) error {

Check failure on line 324 in apis/projectcontour/v1alpha1/contourconfig_helpers.go

View workflow job for this annotation

GitHub Actions / lint 

File is not `gofumpt`-ed with `-extra` (gofumpt)
switch {
case omEnforcedHealth != nil && stats != nil && omEnforcedHealth.Address == stats.Address && omEnforcedHealth.Port == stats.Port:
return fmt.Errorf("cannot use the same port for health checks and overload-manager enforced health checks")
case omEnforcedHealth != nil && metrics != nil && omEnforcedHealth.Address == metrics.Address && omEnforcedHealth.Port == metrics.Port:
return fmt.Errorf("cannot use the same port for metrics and overload-manager enforced health checks")
default:
return nil
}
}
47 changes: 47 additions & 0 deletions changelogs/unreleased/6308-sethepps-minor.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
## Overload Manager - Max Global Connections

Introduces an envoy bootstrap flag to enable the [global downstream connection limit overload manager resource monitors](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/resource_monitors/downstream_connections/v3/downstream_connections.proto#envoy-v3-api-msg-extensions-resource-monitors-downstream-connections-v3-downstreamconnectionsconfig).

The new flag can be passed as an integer flag to the contour bootstrap subcommand, `overload-dowstream-max-conn`.

```sh
contour bootstrap --help
INFO[0000] maxprocs: Leaving GOMAXPROCS=10: CPU quota undefined
usage: contour bootstrap [<flags>] <path>

Generate bootstrap configuration.


Flags:
-h, --[no-]help Show context-sensitive help (also try --help-long and --help-man).
--log-format=text Log output format for Contour. Either text or json.
--admin-address="/admin/admin.sock"
Path to Envoy admin unix domain socket.
--admin-port=ADMIN-PORT DEPRECATED: Envoy admin interface port.
--dns-lookup-family=DNS-LOOKUP-FAMILY
Defines what DNS Resolution Policy to use for Envoy -> Contour cluster name lookup. Either v4, v6, auto, or all.
--envoy-cafile=ENVOY-CAFILE
CA Filename for Envoy secure xDS gRPC communication. ($ENVOY_CAFILE)
--envoy-cert-file=ENVOY-CERT-FILE
Client certificate filename for Envoy secure xDS gRPC communication. ($ENVOY_CERT_FILE)
--envoy-key-file=ENVOY-KEY-FILE
Client key filename for Envoy secure xDS gRPC communication. ($ENVOY_KEY_FILE)
--namespace="projectcontour"
The namespace the Envoy container will run in. ($CONTOUR_NAMESPACE)
--overload-dowstream-max-conn=OVERLOAD-DOWSTREAM-MAX-CONN
Defines the Envoy global downstream connection limit
--overload-max-heap=OVERLOAD-MAX-HEAP
Defines the maximum heap size in bytes until overload manager stops accepting new connections.
--resources-dir=RESOURCES-DIR
Directory where configuration files will be written to.
--xds-address=XDS-ADDRESS xDS gRPC API address.
--xds-port=XDS-PORT xDS gRPC API port.
--xds-resource-version="v3"
The versions of the xDS resources to request from Contour.

Args:
<path> Configuration file ('-' for standard output).
```

As part of this change, we also set the `ignore_global_conn_limit` flag to `true` on the existing admin listeners such that envoy remains live, ready, and serving stats even though it is rejecting downstream connections.
To add some flexibility for health checks, in addition to adding a new bootstrap flag there is a new configuration option for the envoy health config to enforce the envoy overload manager actions, namely rejecting requests. This "advanced" configuration gives the operator the ability to configure readiness and liveness to handle taking pods out of the pool of pods that can service k8s service traffic.
1 change: 1 addition & 0 deletions cmd/contour/bootstrap.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ func registerBootstrap(app *kingpin.Application) (*kingpin.CmdClause, *envoy.Boo
bootstrap.Flag("envoy-cert-file", "Client certificate filename for Envoy secure xDS gRPC communication.").Envar("ENVOY_CERT_FILE").StringVar(&config.GrpcClientCert)
bootstrap.Flag("envoy-key-file", "Client key filename for Envoy secure xDS gRPC communication.").Envar("ENVOY_KEY_FILE").StringVar(&config.GrpcClientKey)
bootstrap.Flag("namespace", "The namespace the Envoy container will run in.").Envar("CONTOUR_NAMESPACE").Default("projectcontour").StringVar(&config.Namespace)
bootstrap.Flag("overload-dowstream-max-conn", "Defines the Envoy global downstream connection limit").Int64Var(&config.GlobalDownstreamConnectionLimit)
bootstrap.Flag("overload-max-heap", "Defines the maximum heap size in bytes until overload manager stops accepting new connections.").Uint64Var(&config.MaximumHeapSizeBytes)
bootstrap.Flag("resources-dir", "Directory where configuration files will be written to.").StringVar(&config.ResourcesDir)
bootstrap.Flag("xds-address", "xDS gRPC API address.").StringVar(&config.XDSAddress)
Expand Down
3 changes: 3 additions & 0 deletions cmd/contour/contour.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,9 @@ func main() {
if err := envoy.ValidAdminAddress(bootstrapCtx.AdminAddress); err != nil {
log.WithField("flag", "--admin-address").WithError(err).Fatal("failed to parse bootstrap args")
}
if err := envoy.ValidConnectionLimit(bootstrapCtx.GlobalDownstreamConnectionLimit); err != nil {
log.WithField("flag", "--overload-dowstream-max-conn").WithError(err).Fatal("failed to parse bootstrap args")
}
if err := envoy_v3.WriteBootstrap(bootstrapCtx); err != nil {
log.WithError(err).Fatal("failed to write bootstrap configuration")
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/contour/serve.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,7 +493,7 @@ func (s *Server) doServe() error {
}

resources := []xdscache.ResourceCache{
xdscache_v3.NewListenerCache(listenerConfig, *contourConfiguration.Envoy.Metrics, *contourConfiguration.Envoy.Health, *contourConfiguration.Envoy.Network.EnvoyAdminPort),
xdscache_v3.NewListenerCache(listenerConfig, *contourConfiguration.Envoy.Metrics, *contourConfiguration.Envoy.Health, *contourConfiguration.Envoy.Network.EnvoyAdminPort, contourConfiguration.Envoy.OMEnforcedHealth),
xdscache_v3.NewSecretsCache(envoy_v3.StatsSecrets(contourConfiguration.Envoy.Metrics.TLS)),
&xdscache_v3.RouteCache{},
&xdscache_v3.ClusterCache{},
Expand Down
9 changes: 9 additions & 0 deletions cmd/contour/servecontext.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -495,6 +495,14 @@ func (ctx *serveContext) convertToContourConfigurationSpec() contour_v1alpha1.Co
Port: ctx.statsPort,
}

var envoyOMEnforcedHealthListenerConfig contour_v1alpha1.HealthConfig
if ctx.Config.OMEnforcedHealthListener != nil {
envoyOMEnforcedHealthListenerConfig = contour_v1alpha1.HealthConfig{
Address: ctx.Config.OMEnforcedHealthListener.Address,
Port: ctx.Config.OMEnforcedHealthListener.Port,
}
}

// Override metrics endpoint info from config files
//
// Note!
Expand Down Expand Up @@ -581,6 +589,7 @@ func (ctx *serveContext) convertToContourConfigurationSpec() contour_v1alpha1.Co
XffNumTrustedHops: &ctx.Config.Network.XffNumTrustedHops,
EnvoyAdminPort: &ctx.Config.Network.EnvoyAdminPort,
},
OMEnforcedHealth: &envoyOMEnforcedHealthListenerConfig,
},
Gateway: gatewayConfig,
HTTPProxy: &contour_v1alpha1.HTTPProxyConfig{
Expand Down
28 changes: 28 additions & 0 deletions examples/contour/01-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -516,6 +516,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 523 in examples/contour/01-crds.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down Expand Up @@ -4201,6 +4215,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 4222 in examples/contour/01-crds.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down
28 changes: 28 additions & 0 deletions examples/render/contour-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -736,6 +736,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 743 in examples/render/contour-deployment.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down Expand Up @@ -4421,6 +4435,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 4442 in examples/render/contour-deployment.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down
28 changes: 28 additions & 0 deletions examples/render/contour-gateway-provisioner.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -527,6 +527,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 534 in examples/render/contour-gateway-provisioner.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down Expand Up @@ -4212,6 +4226,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 4233 in examples/render/contour-gateway-provisioner.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down
28 changes: 28 additions & 0 deletions examples/render/contour-gateway.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -552,6 +552,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 559 in examples/render/contour-gateway.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down Expand Up @@ -4237,6 +4251,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 4258 in examples/render/contour-gateway.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down
28 changes: 28 additions & 0 deletions examples/render/contour.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -736,6 +736,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 743 in examples/render/contour.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down Expand Up @@ -4421,6 +4435,20 @@
format: int32
type: integer
type: object
om_enforced_health:
description: |-
OMEnforcedHealth defines the endpoint Envoy uses to serve health checks with
the envoy overload manager actions, such as global connection limits, enforced.
This is disbled by default

Check failure on line 4442 in examples/render/contour.yaml

View workflow job for this annotation

GitHub Actions / Codespell 

disbled ==> disabled
properties:
address:
description: Defines the health address interface.
minLength: 1
type: string
port:
description: Defines the health port.
type: integer
type: object
service:
description: |-
Service holds Envoy service parameters for setting Ingress status.
Expand Down
16 changes: 16 additions & 0 deletions internal/envoy/bootstrap.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,12 @@ type BootstrapConfig struct {
// MaximumHeapSizeBytes specifies the number of bytes that overload manager allows heap to grow to.
// When reaching the set threshold, new connections are denied.
MaximumHeapSizeBytes uint64

// GlobalDownstreamConnectionLimit specifies the maximum number of global open downstream connections to envoy.
// When at the limit, envoy will reject connections at L4.
//
// Although a valid limit must be >= 0, to avoid overflow we require the type be int64
GlobalDownstreamConnectionLimit int64
}

// GetXdsAddress returns the address configured or defaults to "127.0.0.1"
Expand Down Expand Up @@ -130,6 +136,16 @@ func ValidAdminAddress(address string) error {
return nil
}

// ValidConnectionLimit checks if the supplied
// envoy global connection limit is an integer
// greater than or equal to 0.
func ValidConnectionLimit(limit int64) error {
if limit < 0 {
return fmt.Errorf("invalid value %d, cannot be < 0", limit)
}
return nil
}

func stringOrDefault(s, def string) string {
if s == "" {
return def
Expand Down
Loading
Loading