The glb-director component is the main DPDK application that processes inbound packets and encapsulates them using GUE to server(s) designated by the GLB hasing algorithm.
There are 2 places the director can be configured:
/etc/default/glb-director- Configuration around the DPDK EAL runtime and other similar configuration options, used by systemd./etc/glb/director.conf- The default location for the configuration of how the director processes packets on CPU cores and on ports/NICs.
GLB_DIRECTOR_CONFIG_FILE specifies the config file to use, by default /etc/glb/director.conf.
GLB_DIRECTOR_FORWARDING_TABLE specifies the binary forwarding table to use. By default, the package references a non-healthchecked file, which should manually be updated, /etc/glb/forwarding_table.bin. To enable the glb-healthcheck service to work with the director, this will typically be changed to /etc/glb/forwarding_table.checked.bin.
GLB_DIRECTOR_EAL_ARGS specifies the options to be provided to the DPDK Environment Abstraction Layer at startup. Typically this is where you would configure which CPU cores (lcore by DPDK definition) will be made available to DPDK, and which NICs (ports by DPDK definition) will be provided. There are many other options that may be useful depending on the environment.
GLB Director requires that at least 3 cores are provided. The first (master lcore) just does house-keeping operations like loading forwarding tables and processing signals, while all other cores are used to perform actions (described below). Review the DPDK documentation for details on best practises here, however we typically advise using the Linux kernel isolcpus option to prevent Linux from scheduling work on DPDK cores. On CPUs that support hyperthreads, we only run DPDK on one of the hyperthreads and leave the other empty (both hyperthreads would be included in isolcpus in this case where hyperthreads are still left enabled on other cores for Linux processes - alternatively, hyperthreading could be disabled entirely).
required
GLB Director doesn't handle ARP requests or responses for simplicity, since Linux is already expected to be providing a full TCP/IP stack on the same NIC. Instead, this option should be configured to be the MAC of the router or top of rack switch that the director host is connected to. GLB will use this as the destination MAC in every forwarded packet.
required
Similarly, this specifies the source IP to use for outbound forwarded packets. Typically, this will be the same IP as the machine itself, and Linux will usually be responding to all requests on that IP.
optional
GLB Director hashes based on src/dst fields in L3/L4 packet headers. This field is a hash configuring which header fields to use as part of hash calculation:
src_addr(bool) - The source IP address of the packetdst_addr(bool) - The destination IP address of the packetsrc_port(bool) - The source port of the inner TCP/UDP packetdst_port(bool) - The destination port of the inner TCP/UDP packet
If hash_fields is not specified, it defaults to just source IP. If the field is specified, any values not included default to false. At least one field must be included (set to true).
Changing this value will cause any existing connections to re-hash to new servers. To perform this gracefully, see alt_hash_fields.
Example:
{ "src_addr": true, "src_port": true }
optional
To allow safe migration between 2 selections of hash fields without breaking running connections. hash_fields can be sets to a new value and alt_hash_fields can temporarily specify the previously configured value of hash_fields. The hops that would have previously been used will be appended as a 3rd/4th hop, essentially gracefully draining those nodes of connections that matched them using the previous hash configuration. Once migration is complete, the field can be removed.
optional
By default, GLB Director will forward ICMP pings to one of the backends. If this isn't desired, it can be disabled.
required
The number of NIC RX/TX queues to configure on each NIC that is bound to DPDK. This must not be larger than the maximum number supported by your NIC. Note that since GLB Director/DPDK processes packets extremely fast and can spread work amongst cores, this can typically be as low as 1 to support 10G NIC line rate.
optional
By default, GLB Director will tell the NIC to drop packets when no RX descriptors are available. In certain DPDK drivers (like the E1000 used in Vagrant), this isn't supported, and so may be disabled.
required
GLB Director processes packets from a NIC RX queue, encapsulates them, and drops them on a NIC TX queue. A flow path configures a mapping from inbound to outbound queue, such as:
{ "rx_port": 0, "rx_queue": 0, "tx_port": 0, "tx_queue": 0 }
Generally, and in the default configuration, packets from NIC/Port 0 RX queue 0 are processed and written out to NIC/Port 0 TX queue 0. However, if you have multiple NICs, multiple queues, or want to receive and transmit packets across ports/queues, one or more flow paths can be constructed to do this.
In a typical bonded NIC environment, this might be configured as:
{ "rx_port": 0, "rx_queue": 0, "tx_port": 0, "tx_queue": 0 },
{ "rx_port": 1, "rx_queue": 0, "tx_port": 1, "tx_queue": 0 }
The above bonded NIC configuration is reflected in this diagram:
required
In GLB_DIRECTOR_EAL_ARGS above, DPDK provided GLB Director with a set of lcores. In this section, they are configured to perform specific tasks.
The master lcore should not be listed here and defaults to performing housekeeping work.
Each other core should be configured as one of 2 potential modes. Keys in this setting are lcore-N where N is the DPDK lcore number.
"lcore-1": {
"rx": true,
"tx": true,
"flow_paths": [0],
"dist": true,
"num_dist_workers": 1,
"kni": true
},
A distributor lcore performs at minimum rx work (receiving from a NIC queue), dist work (distributing packets to workers), and tx work (pushing to a NIC queue). Currently all three of these must be enabled on the same lcore (though this may change in the future).
flow_paths specifies a list of the indexes into the previously defined set of flow_paths. A single distributor can process packets for multiple flows (e.g. on the bonded setup above, this may be configured as [0,1]).
num_dist_workers specifies how many worker cores will reference this distributor core.
kni is an optional extension that uses DPDK's Kernel NIC Interface to enable passthrough of non-forwarded packets to Linux. If this is enabled, a new vglb_kniN interface will be created for each NIC provided to DPDK, and any packets that don't match forwarding table binds will arrive on that interface. Packets sent to the interface will be sent out the NIC transparently. This allows a configuration where DPDK owns the entire NIC, but services like BGP can be run on Linux over the vglb_kniN interfaces. If using DPDK Flow Bifurcation, this can be disabled, since only relevant packets are ever sent to DPDK.
"lcore-2": {
"work": true,
"work_source": 1
}
Worker lcores take packets received by the distributor, hash them, encapsulate them, then send them back to the distributor to be sent to the NIC. For more details of the technical implementation of this and why it's used, see DPDK Packet Distributor Library.
work_source configures the lcore index of the distributor core that work will be sourced from. The number of cores that reference any given distributor should match that distributor core's num_dist_workers.
optional
By default, GLB Director uses UDP port 28125 to send messages to statsd. This configuration option allows selecting a different port as the target for statsd messages.