http-routing.html.md.erb

---
title: HTTP Routing
owner: Routing
---

<strong><%= modified_date %></strong>

This topic describes how the Gorouter, the main component in Cloud Foundry's [routing](./architecture/index.html#routing) tier, routes HTTP traffic within Cloud Foundry (CF).


## <a id="http-headers"></a> HTTP Headers

#### <a id="header-size-limit"></a> Header Size Limit

Gorouter has a limit of 1MB for HTTP Headers.

#### <a id="X-Forwarded-Proto"></a> X-Forwarded-Proto

`X-Forwarded-Proto` header gives the scheme of the HTTP request from the client. In requests forwarded to backends, Gorouter sets the scheme to HTTP if the client made an insecure request (on port 80) or HTTPS if the client made a secure request (on port 443). Developers can configure their apps to reject insecure requests by inspecting the HTTP headers of incoming traffic and rejecting traffic that includes `X-Forwarded-Proto` with the scheme of HTTP. 

#### <a id="X-Forwarded-For"></a> X-Forwarded-For

If `X-Forwarded-For` is present, the Gorouter appends the load balancer's IP address to it and forwards the list. If `X-Forwarded-For` is not present, then the Gorouter sets it to the IP address of the load balancer in the forwarded request (some load balancers masquerade the client IP). If a load balancer sends the client IP using the PROXY protocol, then the Gorouter uses the client IP address to set `X-Forwarded-For`. 

If your load balancer terminates TLS on the client side of the Gorouter, it must append these headers to requests forwarded to the Gorouter. <%= vars.http_routing %>

### <a id="zipkin-headers"></a> HTTP Headers for Zipkin Tracing

Zipkin is a tracing system that enables app developers to troubleshoot failures or latency issues. Zipkin provides the ability to trace requests and responses across distributed systems. See [Zipkin.io](http://zipkin.io/) for more information.

<%= vars.zipkin_enable_link %>

* If the `X-B3-TraceId` and `X-B3-SpanId` HTTP headers are not present in the request, the Gorouter generates values for these and inserts the headers into the request forwarded to an application. These values are also found in the Gorouter access log message for the request: `x_b3_traceid` and `x_b3_spanid`.
* If the `X-B3-TraceId` and `X-B3-SpanId` HTTP headers are present in the request, the Gorouter forwards them unmodified. In addition to these trace and span ids, the Gorouter access log message for the request includes `x_b3_parentspanid`.

Developers can then add Zipkin trace IDs to their application logging in order to trace app requests and responses in Cloud Foundry.

After adding Zipkin HTTP headers to app logs, developers can use `cf logs myapp` to correlate the trace and span ids logged by the Gorouter with the trace ids logged by their app. To correlate trace IDs for a request through multiple apps, each app must forward appropriate values for the headers with requests to other applications.

### <a id="app-instance-routing"></a> HTTP Headers for App Instance Routing

Developers who want to obtain debug data for a specific instance of an app can use the HTTP header `X-CF-APP-INSTANCE` to make a request to an app instance.

Perform the following steps to make an HTTP request to a specific app instance:

1. Obtain the GUID of your app:
	<pre class="terminal">$ cf app YOUR-APP --guid</pre>
1. List your app instances and retrieve the index number of the instance you want to debug:
	<pre class="terminal">$ cf app YOUR-APP</pre>
1. Make a request to the app route using the HTTP header `X-CF-APP-INSTANCE` set to the concatenated values of the app GUID and the instance index:
	<pre class="terminal">$ curl app.example.com -H "X-CF-APP-INSTANCE":"YOUR-APP-GUID:YOUR-INSTANCE-INDEX"</pre>

### <a id="forward-client-cert"></a> Forward Client Certificate to Applications

Applications that require mutual TLS (mTLS) need metadata from client certificates to authorize requests. Cloud Foundry supports this use case without bypassing layer-7 load balancers and the Gorouter.

The HTTP header `X-Forwarded-Client-Cert` (XFCC) may be used to pass the originating client certificate along the data path to the application. Each component in the data path must trust that the back end component has not allowed the header to be tampered with.

If you configure the load balancer to terminate TLS and set the XFCC header from the received client certificate, then you must also configure the load balancer to strip this header if it is present in client requests. This configuration is required to prevent spoofing of the client certificate.

The following sections describe supported deployment configurations.

<% if vars.product_name == 'CF' %>
<%= partial 'oss_xfcc_loadbalancer' %>
<% else %>
<%= partial '../console/xfcc_loadbalancer' %>
<% end %>

<% if vars.product_name == 'CF' %>
<% else %>
<%= partial '../console/xfcc_haproxy' %>
<% end %>

<% if vars.product_name == 'CF' %>
<%= partial 'oss_xfcc_router' %>
<% else %>
<%= partial '../console/xfcc_router' %>
<% end %>

## <a id="client-side-tls"></a>Client-Side TLS

Depending on your needs, you can configure your deployment to terminate TLS at the Gorouter, at the Gorouter and the load balancer, or at the load balancer only. <%= vars.http_routing %>


## <a id="tls-to-back-end"></a> TLS to Apps and Other Back End Services

The Gorouter supports TLS and mutual authentication to back end destinations, including app instances, platform services, and any other routable endpoints.

<% if vars.product_name == 'CF' %>
This is enabled by configuring the following manifest properties:

- `router.backends.enable_tls: true`
- `router.ca_certs`: must include the CA certificate used to sign any back end to which the Gorouter initiates a TLS handshake.

The Gorouter does not automatically initiate TLS handshakes with back end destinations. To register that the Gorouter must initiate a TLS handshake before forwarding HTTP requests to a component, the component must include the following optional fields in its `route.register` message to NATS:

- `"tls_port"`: The port to which the Gorouter should open a connection and initiate a TLS handshake
- `"server_cert_domain_san"`: An identifier for the back end which the Gorouter expects to find in the Domain Subject Alternative Name of the certificate presented by the back end. This is used to prevent misrouting for back ends whose IPs and/or ports are expected to change; see [Consistency](#consistency).

To enable TLS from the Gorouter to application instances, some additional configuration is required in addition to the properties above; see [Configuring Consistency with TLS](#with-tls).

Authors of routable components can configure [Route Registrar](https://github.com/cloudfoundry/route-registrar) to send the necessary NATS message automatically.

To enable mutual authentication between the Gorouter and back ends, operators configure the Gorouter with a certificate and private key using the following manifest properties:

- `router.backends.cert_chain`
- `router.backends.private_key`

The Gorouter presents the certificate if requested by the back end during the TLS handshake.
<% else %>
This is enabled by following the steps in [Configure Validation of App Instance Identity with TLS](#app-valid-tls-config). 
<% end %>

## <a id="consistency"></a> Preventing Misrouting

As CF manages and balances apps, the internal IP address and ports for app instances change. To keep the Gorouter's routing tables current, a Route-Emitter on each Diego cell sends a periodic update to all Gorouters through NATS, reminding them of the location of all app instances; the default frequency of these updates is 20 seconds. The Gorouter tracks a time-to-live (TTL) for each route to back end mapping; this TTL defaults to 120 seconds and is reset when the Gorouter receives an updated registration message.

Network partitions or NATS failures can cause the Gorouter's routing table to fall out of sync, as CF continues to re-create containers across hosts to keep apps running. This can lead to routing of requests to incorrect destinations.

You can configure the Gorouter to handle this problem in two ways:

<table border="1" id="gorouter-modes" class="nice">
  <tr><th>Consistency mode</th>
  	<th>Security of traffic between Gorouter and Containers</th>
  	<th>Gorouter Route Pruning</th>
  </tr><tr>
  	<td><a href="#with-tls"><strong>With TLS Enabled</strong></a></td>
  	<td>Encrypted via TLS</td>
  	<td>Routes are pruned on failure of TLS handshake only. See <a href="#when-tls-prune">Gorouter TLS pruning behavior.</a></td>
  </tr><tr>
  	<td><a href="#without-tls"><strong>Without TLS Enabled</strong></a></td>
  	<td>Plain text</td>
  	<td>Routes are pruned on TTL expiry</td>
  </tr>
</table>

Both of these consistency modes are described below.

### <a id="with-tls"></a> With TLS Enabled

This consistency mode is newer and has the following benefits:

- Improved availability for applications by keeping routes in the Gorouter's routing table when TTL expires
- Increased guarantees against misrouting by validating the identity of backends before forwarding requests
- Increased security by encrypting data in flight from the Gorouter to backends

<p class='note warning'><strong>WARNING:</strong> TLS routing requires an additional 32&nbsp;MB of RAM capacity on your Diego cells per app instance, as well as additional CPU capacity. Check the total amount of Diego cell memory available to allocate in your environment, and if it is less than 32&nbsp;MB times the number of running app instances, scale out your Diego cells.</p>

<p class='note warning'><strong>WARNING:</strong> You may see an increase of memory and CPU usage for your Gorouters after enabling TLS routing. Check the total amount of memory and CPU usage of the Gorouters in your environment, and if they are close to the size limit, consider scaling out your Gorouters before enabling TLS routing.</p>

In this mode, the Diego Route-Emitters send a modified route registration message to NATS that includes a unique identifier for the app instance, as well instructions to use TLS when communicating with the instance. See [TLS to Apps and Other Back End Services](#tls-to-back-end) for details.

Before forwarding traffic to an app instance, the Gorouter initiates a TLS handshake with an [Envoy proxy](https://www.envoyproxy.io/) running in each app container. In the TLS handshake, the Envoy proxy presents a certificate generated by Diego for each container which uniquely identifies the container using the same app instance identifier sent by the Route-Emitter, configured in the certificate as a domain Subject Alternative Name (SAN).

If the Gorouter confirms that the app instance identifier in the certificate matches the one received in the route registration message, the Gorouter forwards the HTTP request over the TLS session, and the Envoy proxy then forwards it to the app process. If the instance identifiers do not match, the Gorouter removes the app instance from its routing table and transparently retries another instance of the app.

#### <a id="app-valid-tls-config"></a> Configure Validation of App Instance Identity with TLS

<% if vars.product_name == 'CF' %>
<%= partial 'router_app_tls_oss' %>
<% else %>
<%= partial '../console/router_app_tls_pcf' %>
<% end %>

### <a id="without-tls"></a> Without TLS Enabled

In this consistency mode, the Diego Route-Emitters on each cell send route registration messages that include instructions for the Gorouter to send unencrypted requests to the app instance. If the Gorouter does not receive an update for the route within the time-to-live (TTL) interval, the route is pruned from the Gorouter's routing table. See [TLS to Apps and Other Back End Services](#tls-to-back-end) for details.

This pruning method was developed before the alternative was available.

<% if vars.product_name.include? "CF" %>

### <a id="config-by-instance-group"></a> Consistency Mode Can Differ by Instance Group

The Gorouter can [validate app instance identity using TLS](#with-tls) only when Diego cells are configured appropriately. Because cells are configured for TLS through the instance group that they belong to, the Gorouter can run in different consistency modes with cells in different instance groups. For example, the Gorouter can communicate over TLS and validate the cells in one Isolation Segment, while communicating with cells in another Isolation Segment via plain text and [without validating instance identity](#without-tls).

Currently, only Linux cells support the Gorouter validating app instance identities [using TLS](#with-tls). With Windows cells, the Gorouter runs [without TLS enabled](#without-tls), forwarding requests to Windows apps over plain text and pruning based on route TTL.

<% end %>


## <a id="round-robin"></a> Round-Robin Load Balancing

The Gorouter uses the round-robin algorithm for load balancing incoming requests to application instances. The Gorouter maintains a dynamically updated list of application instances for each route, and forwards each request for a given route to the next application instance in the list.


## <a id="sockets"></a> WebSockets

WebSockets is a protocol providing bi-directional communication over a single, long-lived TCP connection, commonly implemented by web clients and servers. WebSockets are initiated through HTTP as an upgrade request. The Gorouter supports this upgrade handshake, and holds the TCP connection open with the selected application instance. <%= vars.port_limitations_1 %>


## <a id="sessions"></a> Session Affinity

The Gorouter supports session affinity, or _sticky sessions_, for incoming HTTP requests to compatible apps.

With sticky sessions, when multiple instances of an app are running on CF, requests from a particular client always reach the same app instance. This allows apps to store session data specific to a user session.

* To support sticky sessions, configure your app to return a `JSESSIONID` cookie in responses. The app generates a `JSESSIONID` as a long hash in the following format:
<pre class="terminal">
1A530637289A03B07199A44E8D531427
</pre>

* If an app returns a `JSESSIONID` cookie to a client request, the CF routing tier generates a unique `VCAP_ID` for the app instance based on its GUID in the following format:
<pre class="terminal">
323f211e-fea3-4161-9bd1-615392327913
</pre>

* On subsequent requests, the client must provide both the `JSESSIONID` and `VCAP_ID` cookies.

The CF routing tier uses the `VCAP_ID` cookie to forward client requests to the same app instance every time. The `JSESSIONID` cookie is forwarded to the app instance to enable session continuity. If the app instance identified by the `VCAP_ID` crashes, the Gorouter attempts to route the request to a different instance of the app. If the Gorouter finds a healthy instance of the app, it initiates a new sticky session.

<p class="note"><strong>Note:</strong> CF does not persist or replicate HTTP session data across app instances. If an app instance crashes or is stopped, session data for that instance is lost. If you require session data to persist across crashed or stopped instances, or to be shared by all instances of an app, store session data in a CF marketplace service that offers data persistence.</p>


## <a id="keepalive"></a> Keepalive Connections

### <a id='front-end'></a> From Front End Clients

The Gorouter supports keepalive connections from clients and does not close the TCP connection with clients immediately after returning an HTTP response. Clients are responsible for closing these connections.

### <a id='back-end'></a> To Back End Servers

If keepalive connections are disabled, the Gorouter closes the TCP connection with an app instance or system component after receiving an HTTP response.

If keepalive connections are enabled, the Gorouter maintains established TCP connections to back ends. the Gorouter supports up to 100 idle connections to each back end:

* If an idle connection exists for a given back end, the Gorouter reuses it to route subsequent requests.
* If no idle connection exists for this back end, the Gorouter creates a new connection.

<% if vars.product_name == "CF" %> <%= vars.keepalive %> <% else %> <% end %>


## <a id="retry"></a> Transparent Retries

If the Gorouter cannot establish a TCP connection with a selected application instance, the Gorouter considers the instance ineligible for requests for 30 seconds and transparently attempts to connect to another application instance. Once the Gorouter has established a TCP connection with an application instance, the Gorouter forwards the HTTP request.

When you deploy an app that requires Diego cells to restart or recreate, the app may not respond to a Gorouter request before the keepalive connection breaks. The following table describes how the Gorouter behaves if it cannot establish a TCP connection to an app:

<table>
  <tr>
    <th>If the Gorouter...</th>
    <th>and the backend...</th>
    <th>then the Gorouter...</th>
  </tr><tr>
    <td>cannot establish a TCP connection to a backend</td>
    <td>n/a</td>
    <td>retries another backend, no more than 3 times</td>
  </tr><tr>
    <td>establishes a TCP connection to a backend and forwards the request</td>
    <td>does not respond</td>
    <td>waits 15 minutes for a response, and if it errors, does not retry another backend</td>
  </tr><tr>
    <td>establishes a TCP connection to a backend and forwards the request</td>
    <td>returns a TCP connection error</td>
    <td>returns an error to the client, marks backend ineligible, and does not retry another backend</td>
</table>

In all cases, if the app still does not respond to the request, the Gorouter returns a [`502` error](https://docs.cloudfoundry.org/adminguide/troubleshooting-router-error-responses.html).