The section gives a simplified description of the path a packet goes through from the time it enters the ASIC until it is scheduled for transmission. It then continues to describe the various quality of service (QoS) mechanisms supported by the ASIC and how they affect the previously described path. ##### Table of Contents 1. [Packet's Path](#packets-path) 2. [Priority Assignment](#priority-assignment) 1. [Trust PCP](#trust-pcp) 2. [Trust DSCP](#trust-dscp) 3. [Default Priority](#default-priority) 4. [Priority Update in Forwarded IPv4 Packets](#priority-update-in-forwarded-ipv4-packets) 5. [DSCP Rewrite](#dscp-rewrite) 3. [Data Center Bridging](#data-center-bridging) 1. [Enhanced Transmission Selection (ETS)](#enhanced-transmission-selection-ets) 2. [Priority-based Flow Control (PFC)](#priority-based-flow-control-pfc) 1. [PFC Tx](#pfc-tx) 2. [PFC Rx](#pfc-rx) 4. [PAUSE Frames](#pause-frames) 5. [Shared Buffers](#shared-buffers) 1. [Admission Rules](#admission-rules) 2. [Pool Size](#pool-size) 3. [Pool Threshold](#pool-threshold) 4. [Pool Binding](#pool-binding) 5. [Pool Occupancy](#pool-occupancy) 6. [Handling of BUM Traffic](#handling-of-bum-traffic) 7. [Spectrum-1 Shared Buffer Configuration](#spectrum-1-shared-buffer-configuration) 8. [Spectrum-2 Shared Buffer Configuration](#spectrum-2-shared-buffer-configuration) 5. [Further Resources](#further-resources) The Packet's Path ----------------- When a packet enters the chip, it is assigned Switch Priority (SP), an internal identifier that informs how the packet is treated in context of other traffic. The assignment is made based on packet headers and switch configuration—see [Priority Assignment](#priority-assignment) for details. Afterwards, the packet is directed to a priority group (PG) buffer in the port's headroom based on its SP. The port's headroom buffer is used to store incoming packets on the port while they go through the switch's pipeline and also to store packets in a lossless flow if they are not allowed to enter the switch's shared buffer. However, if there is no room for the packet in the headroom, it gets dropped. Mapping an SP to a PG buffer is explained in the [ETS section](#enhanced-transmission-selection-ets). Once outside the switch's pipeline, the packet's ingress port, internal SP, egress port, and traffic class (TC) are known. Based on these parameters the packet is classified to ingress and egress buffers in the switch's shared buffer. In order for it to be eligible to enter the shared buffer, certain quotas associated with the packet need to be checked. These quotas and their configuration are described in the [shared buffers section](#shared-buffers). The packet stays in the shared buffer until it is transmitted from its designated egress port. The packet is queued for transmission according to its Traffic Class (TC). Once in its assigned queue, the packet is scheduled for transmission based on the chosen traffic selection algorithm (TSA) employed on the TC and its various parameters. The mapping from SP to TC and TSA configuration are discussed in the [ETS section](#enhanced-transmission-selection-ets). Packets which are not eligible to enter the shared buffer stay in the headroom if they are associated with a lossless flow (mapped to a lossless PG). Otherwise, they are dropped. The configuration of lossless flows is discussed in the [PFC](#priority-based-flow-control-pfc) and [PAUSE frames](#pause-frames) sections. #### Features by Version | Kernel Version | | |:--------------:|:---------------------------------------------------------------- | | **4.7** | Shared buffers, Trust PCP | | **4.19** | Trust DSCP, DSCP rewrite, `net.ipv4.ip_forward_update_priority` | | | Dedicated pool & TCs for BUM traffic | | **4.20** | BUM pool & TCs exposed in devlink-sb | | Minimum shaper configured on BUM TCs | | **5.1**(*) | Spectrum-2 support | **Note:** Kernel 5.1 is currently in development. Priority Assignment ------------------- Switch Priority (SP) of a packet can be derived from packet headers, or assigned by default. Which headers are used when deriving packet priority depends on configuration of Trust Level of the port through which a given packet ingresses. `mlxsw` currently recognizes two trust levels: "Trust PCP" (or Trust L2; this is the default) and "Trust DSCP" (or Trust L3). #### Trust PCP By default, ports are in trust-PCP mode. In that case, the switch will prioritize a packet based on its [IEEE 802.1p][1] priority. This priority is specified in the packet's [Priority Code Point (PCP)][2] field, which is part of the packet's VLAN header. The mapping between PCP and SP is 1:1 and cannot be changed. If the packet does not have 802.1p tagging, it is assigned [port-default priority](#default-priority). #### Trust DSCP Each port can be configured to set Switch Priority of packets based on the [DSCP field][9] of IP and IPv6 headers. The priority is assigned to individual DSCP values by [Data Center Bridging toolset](#data-center-bridging), using LLDP "APP" TLVs with selector 5 for DSCP. The syntax is: ``` $ lldptool -T -i -V APP app=,5, # Insert rule. $ lldptool -T -i -V APP -d app=,5, # Delete rule. $ lldptool -t -i -V APP -c app # Show rules. ``` After the first APP rule is added for a given port, this port's trust level is toggled to DSCP. It stays in this mode until all DSCP APP rules are removed again. **Note:** The support for the DSCP APP rules in openlldpad was introduced by [this patch][8]. If the above commands give a "selector out of range" error, the reason is that the package that you are using does not carry the patch. For example, to assign priority 3 to packets with DSCP 24: ``` $ lldptool -T -i sw1p5 -V APP app=3,5,24 ``` Linux DCB and openlldp allow a very flexible configuration of DSCP-to-SP mapping, to the point of permitting configuration of two priorities for the same DSCP value. These conflicts are resolved in favor of the highest configured priority. For example: ``` $ lldptool -T -i sw1p5 -V APP app=3,5,24 # Configure 24->3. $ lldptool -T -i sw1p5 -V APP app=2,5,24 # Keep 24->3. $ lldptool -T -i sw1p5 -V APP -d app=3,5,24 # Configure 24->2. ``` When a packet arrives with a DSCP value that does not have a corresponding APP rule, the default priority that the packet is assigned is 0. This can be changed through LLDP APP rules with selector 1 (Ethertype) and PID 0. Such rules denote default priority: ``` $ lldptool -T -i -V APP app=,1,0 ``` As with the DSCP APP rules, `lldptool` allows configuration of several default priorities. Again, `mlxsw` chooses the highest one that is configured. Trust-DSCP mode disables PCP prioritization even for non-IP packets that have no DSCP value, and even if they have 802.1p tagging. Such packets get the [port-default priority](#default-priority) assigned instead. **Note:** The Spectrum chip supports also a "trust both" mode, where Switch Priority is assigned based on PCP when DSCP is not available, however this mode is currently not supported by `mlxsw`. **Note:** The use of selector 5 is described in the draft standard 802.1Qcd/D2.1 Table D-9. The use of selector 1 is standardized in 802.1Q-2014 in the same Table D-9. #### Default Priority A default value is assigned to packet's SP when: - it does not have an 802.1p tagging and ingresses through a trust-PCP port - it is a non-IPv4/non-IPv6 packet and ingresses through a trust-DSCP port The default SP is 0, and cannot currently be reconfigured. #### Priority Update in Forwarded IPv4 Packets In Linux, the priority of forwarded IPv4 SKBs is updated according to the TOS value in IPv4 header. The mapping is fixed, and cannot be configured. `mlxsw` configures the switch to update packet priority after routing as well using the same values. However unlike the software path, this is done for both IPv4 and IPv6. **Note:** The actual mapping from TOS to SKB priority is shown in man tc-prio, section "QDISC PARAMETERS", parameter "priomap". As of 4.19, it is possible to turn this behavior off through sysctl: ``` $ sysctl -w net.ipv4.ip_forward_update_priority=0 ``` This disables IPv4 priority update after forwarding in slow path, as well as both IPv4 and IPv6 post-routing priority update in the chip. #### DSCP Rewrite Packets that ingress the switch through a port that is in trust-DSCP mode, will have their DSCP value updated as they egress the switch. The same DSCP APP rules that are used for packet prioritization are used to configure the rewrite as well. If several priorities end up resolving to the same DSCP value (as they probably will), the highest DSCP is favored. For example: ``` $ lldptool -T -i sw1p5 -V APP app=3,5,26 # Configure 3->26 $ lldptool -T -i sw1p5 -V APP app=3,5,24 # Keep 3->26 $ lldptool -T -i sw1p5 -V APP -d app=3,5,26 # Configure 3->24 ``` it is worth repeating that the decision to rewrite is made as the packet ingresses the switch through a trust-DSCP port, however the rewrite map to use is taken from the egress port. As a consequence, in mixed-trust switches, if a packet ingresses through a trust-PCP port, and egresses through a trust-DSCP port, its DSCP value will stay intact. That means that a DSCP value can leak from one domain to another, where it may have a different meaning. Therefore when running a switch with a mixture of trust levels, one needs to be careful that this is not a problem. Data Center Bridging -------------------- The ASIC supports certain enhancements defined under DCB. For an overview of DCB and particularly DCB operation in Linux please refer to [this article][3]. The DCB configuration described below is done using the [Open-LLDP][4] software package. Therefore, it is advised to first go over the [LLDP document](Link-Layer-Discovery-Protocol). To enable DCB support in the `mlxsw_spectrum` driver, `CONFIG_MLXSW_SPECTRUM_DCB=y` is needed. #### Enhanced Transmission Selection (ETS) In Linux, packets are stored in a data structure called [socket buffer (SKB)][5]. One of the fields of the SKB is the `priority` field. Under DCB operation, the `priority` is used to determine the packet's TC based on the user priority to TC mapping. In the ASIC, the entity corresponding to SKB priority is the Switch Priority (SP). By default, in Linux, all priorities are mapped to TC 0. To configure a different mapping, use `lldptool` to configure the `up2tc` mapping in `ETS-CFG` TLV. E.g.: ``` $ lldptool -T -i sw1p5 -V ETS-CFG up2tc=0:0,1:1,2:2,3:3,4:4,5:5,6:6,7:7 ``` The above command creates 1:1 mapping from SP to TC. In addition, the command creates 1:1 mapping between SP and PG buffer, as mapping different flows to different TCs at egress also implies that they should be separated at ingress. Since there is 1:1 mapping between PCP value and SP, the aforementioned effectively means that packets with different PCP values are to be directed to different buffers at ingress and different TCs at egress. Two TSAs are supported by `mlxsw`: Strict Priority (the default) and ETS. To configure TSA, configure the `tsa` property in `ETS-CFG` TLV. For example, to select Strict Priority: ``` $ lldptool -T -i sw1p5 -V ETS-CFG \ tsa=0:strict,1:strict,2:strict,3:strict,4:strict,5:strict,6:strict,7:strict \ ``` **Note:** In case of a congestion, if packets are not admitted to shared buffer, they will be dropped. That is the case even if those packets are of higher priority than the ones already admitted to the shared buffer. Make sure that the quotas are configured in such a way that there is always room in the egress buffer for the high-priority traffic to be admitted. ETS is implemented in the ASIC using a [weighted round robin (WRR)][6] algorithm. The device requires that the sum of the weights used amounts to `100`. Otherwise, changes do not take effect. To set the TSA for each TC to ETS, run: ``` $ lldptool -T -i sw1p5 -V ETS-CFG \ tsa=0:ets,1:ets,2:ets,3:ets,4:ets,5:ets,6:ets,7:ets \ tcbw=12,12,12,12,13,13,13,13 ``` To allow neighbouring hosts to know about the ETS configuration, run: ``` $ lldptool -T -i sw1p5 -V ETS-CFG enableTx=yes ``` This can be verified on a neighbouring host by running: ``` hostA$ lldptool -i enp6s0 -n -t -V ETS-CFG IEEE 8021QAZ ETS Configuration TLV Willing: yes CBS: not supported MAX_TCS: 8 PRIO_MAP: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 TC Bandwidth: 12% 12% 12% 12% 13% 13% 13% 13% TSA_MAP: 0:ets 1:ets 2:ets 3:ets 4:ets 5:ets 6:ets 7:ets ``` #### Priority-based Flow Control (PFC) Packets which are not eligible to enter the shared buffer can stay in the headroom, if they are associated with a lossless flow. To configure a PG as lossless, configure `enabled` attribute of `PFC` TLV using `lldptool`. For example, to enable PFC for priorities 1, 2 and 3, run: ``` $ lldptool -T -i sw1p5 -V PFC enabled=1,2,3 ``` This command enables PFC for both Rx and Tx. **Note:** Setting a priority as lossless and mapping it to a PG buffer along with lossy priorities yields unpredictable behavior. Besides enabling PFC, [Shared Buffers](#shared-buffers) need to be configured suitably as well. Packets associated with a lossless flow can only stay in the headroom if the following conditions hold: * The pool associated with PFC priorities and TCs is static. * The per-Port and per-{Port, TC} quotas of the egress port are set to the maximum allowed value. * When binding a lossless PG to a pool, the size of the pool and the quota assigned to the {Port, PG} must be non-zero. ##### PFC Tx Packets with PFC-enabled priorities are allowed to stay in their assigned PG buffer in the port's headroom, but if the PG buffer is full they are dropped. To prevent that from happening, once the Xoff threshold in the PG buffer is reached the switch sends a PFC packet telling the transmitter to stop sending (Xoff) traffic for all the priorities sharing the same PG. However, traffic can still ingress through the port until the transmitter receives and processes the PFC packet. This amount of traffic is known as the PFC delay allowance and can be set by running the following (in bits): ``` $ lldptool -T -i sw1p5 -V PFC delay=32768 ``` The resulting PG buffer: ``` +----------------+ + | | | | | | | | | | | | | | | | | | Delay | | | | | | | | | | | | | | | Xon/Xoff threshold +----------------+ + | | | | | | 2 * MTU | | | +----------------+ + ``` If packets continue to be dropped, then the delay value should be increased. (Maximum configurable delay is 65535 bit times.) For PFC to work properly, both sides of the link need to be configured correctly. To do so, the willing bit needs to be set. Run: ``` $ lldptool -T -i sw1p5 -V PFC willing=no $ lldptool -T -i sw1p5 -V PFC enableTx=yes ``` And on `hostA`: ``` $ lldptool -T -i enp6s0 -V PFC willing=yes ``` When `hostA` receives the switch's PFC TLV, it will use its settings: ``` hostA$ lldptool -t -i enp6s0 -V PFC IEEE 8021QAZ PFC TLV Willing: yes MACsec Bypass Capable: no PFC capable traffic classes: 8 PFC enabled: 1 2 3 ``` Once the amount of data in the PG buffer goes below the threshold, a PFC packet is transmitted telling the other side to resume transmission (Xon). ##### PFC Rx When a PFC packet is received by a port, it stops the TCs to which the priorities set in the PFC packet are mapped. **Note:** A PFC packet received for a PFC enabled priority stops lossy priorities from transmitting if they are mapped to the same TC as the lossless priority. PAUSE Frames ------------ To enable PAUSE frames on a port, run: ``` $ ethtool -A sw1p5 autoneg off rx on tx on ``` To query PAUSE frame parameters, run: ``` $ ethtool -a sw1p5 Pause parameters for sw1p5: Autonegotiate: off RX: on TX: on ``` Unlike PFC configuration, it is not possible to set the delay parameter. Therefore, this delay is automatically set according to the worst case scenario. **Note:** Enabling PAUSE frames on a port that has too many PG buffers configured can result in a failed operation due to the size limit of the port's headroom. **Note:** It is not possible to have both PFC and PAUSE frames enabled on a port at the same time. Trying to do so generates an error. Shared Buffers -------------- As explained above, packets are admitted to the switch's shared buffer from the port's headroom and stay there until they are transmitted out of the switch. The device has two types of pools, ingress and egress. The pools are used as containers for packets and allow a user to limit the amount of traffic: * From a port * From a {Port, PG buffer} * To a port * To a {Port, TC} The limit can be either a specified amount of bytes (static) or a percentage of the remaining free space (dynamic). #### Admission Rules Once out of the switch's pipeline, a packet is admitted into the shared buffer only if all four quotas mentioned above are below the configured threshold: - `Ingress{Port}.Usage < Thres` - `Ingress{Port,PG}.Usage < Thres` - `Egress{Port}.Usage < Thres` - `Egress{Port,TC}.Usage < Thres` A packet admitted to the shared buffer updates all four usages. #### Pool Size To configure a pool's size and threshold type, run: ``` $ devlink sb pool set pci/0000:03:00.0 pool 0 size 12401088 thtype dynamic ``` To see the current settings of a pool, run: ``` $ devlink sb pool show pci/0000:03:00.0 pool 0 ``` **Note:** Control packets (e.g. LACP, STP) use ingress pool number 3 and cannot be bound to a different pool. It is therefore important to configure it using a suitable size. #### Pool Threshold Limiting the usage of a flow in a pool can be done by using either a static or dynamic threshold. The threshold type is a pool property and is set as follows: ``` $ devlink sb pool set pci/0000:03:00.0 pool 0 size 12401088 thtype static ``` To set a dynamic threshold, run: ``` $ devlink sb pool set pci/0000:03:00.0 pool 0 size 12401088 thtype dynamic ``` #### Pool Binding To bind packets originating from a {Port, PG} to an ingress pool, run: ``` $ devlink sb tc bind set pci/0000:03:00.0/1 tc 0 type ingress pool 0 th 9 ``` Or use port name instead: ``` $ devlink sb tc bind set sw1p17 tc 0 type ingress pool 0 th 9 ``` If the pool's threshold is dynamic, then the value specified as the threshold is used to calculate the `alpha` parameter: ``` alpha = 2 ^ (th - 10) ``` The range of the passed value is between 3 and 16. The computed `alpha` is used to determine the maximum usage of the flow according to the following formula: ``` max_usage = alpha / (1 + alpha) * Free_Buffer ``` Where `Free_Buffer` is the amount of non-occupied buffer in the relevant pool. The following table shows the possible `th` values and their corresponding maximum usage: | th | alpha | max_usage | |:----:|:----------:|:-----------:| | 3 | 0.0078125 | 0.77% | | 4 | 0.015625 | 1.53% | | 5 | 0.03125 | 3.03% | | 6 | 0.0625 | 5.88% | | 7 | 0.125 | 11.11% | | 8 | 0.25 | 20% | | 9 | 0.5 | 33.33% | | 10 | 1 | 50% | | 11 | 2 | 66.66% | | 12 | 4 | 80% | | 13 | 8 | 88.88% | | 14 | 16 | 94.11% | | 15 | 32 | 96.96% | | 16 | 64 | 98.46% | To see the current settings of binding of {Port, PG} to an ingress pool, run: ``` $ devlink sb tc bind show sw1p17 tc 0 type ingress sw1p17: sb 0 tc 0 type ingress pool 0 threshold 10 ``` Similarly for egress, to bind packets directed to a {Port, TC} to an egress pool, run: ``` $ devlink sb tc bind set sw1p17 tc 0 type egress pool 4 th 9 ``` If the pool's threshold is static, then its value is treated as the maximum number of bytes that can be used by the flow. The admission rule requires that the port's usage is also smaller than the maximum usage. To set a threshold for a port, run: ``` $ devlink sb port pool set sw1p17 pool 0 th 15 ``` The static threshold can be used to set minimal and maximal usage. To set minimal usage, the static threshold should be set to 0, in which case the flow never enters the specified pool. Maximal usage can be configured by setting the threshold to the pool's size or larger. #### Pool Occupancy It is possible to take a snapshot of the shared buffer usage with the following command: ``` $ devlink sb occupancy snapshot pci/0000:03:00.0 ``` Once the snapshot is taken, the user may query the current and maximum usage by: * Pool * {Port, Pool} * {Port, PG} * {Port, TC} This is especially useful when trying to determine optimal sizes and thresholds. **Note:** The snapshot is not atomic. However, the interval between the different measurements composing it is as minimal as possible, thus making the snapshot as accurate as possible. **Note:** Queries following a failed snapshot are invalid. To monitor the current and maximum occupancy of a port in a pool and to display current and maximum usage of {Port, PG/TC} in a pool, run: ``` $ devlink sb occupancy show sw1p17 sw1p17: pool: 0: 0/0 1: 0/0 2: 0/0 3: 0/0 4: 0/0 5: 0/0 6: 0/0 7: 0/0 8: 0/0 itc: 0(0): 0/0 1(0): 0/0 2(0): 0/0 3(0): 0/0 4(0): 0/0 5(0): 0/0 6(0): 0/0 7(0): 0/0 etc: 0(4): 0/0 1(4): 0/0 2(4): 0/0 3(4): 0/0 4(4): 0/0 5(4): 0/0 6(4): 0/0 7(4): 0/0 8(8): 0/0 9(8): 0/0 10(8): 0/0 11(8): 0/0 12(8): 0/0 13(8): 0/0 14(8): 0/0 15(8): 0/0 ``` To clear the maximum usage (watermark), run: ``` $ devlink sb occupancy clearmax pci/0000:03:00.0 ``` A new maximum usage is tracked from the time the clear operation is performed. #### Handling of BUM Traffic Flood traffic is subject to special treatment by the ASIC. Such traffic is commonly referred to as BUM, for broadcast, unknown-unicast and multicast packets. BUM traffic is prioritized and scheduled separately from other traffic, using egress TCs 8–15, as opposed to TCs 0–7 for unicast traffic. Thus packets that would otherwise be assigned to some TC X, are assigned to TC X+8 instead, if they are BUM packets. The pairs of unicast and corresponding BUM TCs are then configured to strictly prioritize the unicast traffic: TC 0 is strictly prioritized over TC 8, 1 over 9, etc. This configuration is necessary to mitigate an issue in Spectrum chips where an overload of BUM traffic shuts all unicast traffic out of the system. However, strictly prioritizing unicast traffic has, under sustained unicast overload, the effect of blocking e.g. ARP traffic. These packets are admitted to the system, stay in the queues for a while, but if the lower-numbered TC stays occupied by unicast traffic, their lifetime eventually expires, and these packets are dropped. To prevent this scenario, a minimum shaper of 200Mbps is configured on the higher-numbered TCs to allow through a guaranteed trickle of BUM traffic even under unicast overload. #### Spectrum-1 Shared Buffer Configuration By default, two ingress pools and one egress pool are used. In addition to that, pool 8 is dedicated for [BUM traffic](#handling-of-bum-traffic). ``` $ devlink sb pool show pci/0000:03:00.0: sb 0 pool 0 type ingress size 12440064 thtype dynamic cell_size 96 sb 0 pool 1 type ingress size 0 thtype dynamic cell_size 96 sb 0 pool 2 type ingress size 0 thtype dynamic cell_size 96 sb 0 pool 3 type ingress size 200064 thtype dynamic cell_size 96 sb 0 pool 4 type egress size 13232064 thtype dynamic cell_size 96 sb 0 pool 5 type egress size 0 thtype dynamic cell_size 96 sb 0 pool 6 type egress size 0 thtype dynamic cell_size 96 sb 0 pool 7 type egress size 0 thtype dynamic cell_size 96 sb 0 pool 8 type egress size 15990816 thtype static cell_size 96 ``` In addition to configuring TC to recognize and handle BUM traffic, on Spectrum-1, all such traffic is also assigned to a separate pool 8: ``` $ devlink sb pool show pci/0000:03:00.0 pool 8 pci/0000:03:00.0: sb 0 pool 8 type egress size 15990816 thtype static cell_size 96 $ for i in {8..15}; do > devlink sb tc bind show pci/0000:03:00.0/1 tc $i type egress > done enp3s0np33: sb 0 tc 8 type egress pool 8 threshold 15990816 enp3s0np33: sb 0 tc 9 type egress pool 8 threshold 15990816 enp3s0np33: sb 0 tc 10 type egress pool 8 threshold 15990816 enp3s0np33: sb 0 tc 11 type egress pool 8 threshold 15990816 enp3s0np33: sb 0 tc 12 type egress pool 8 threshold 15990816 enp3s0np33: sb 0 tc 13 type egress pool 8 threshold 15990816 enp3s0np33: sb 0 tc 14 type egress pool 8 threshold 15990816 enp3s0np33: sb 0 tc 15 type egress pool 8 threshold 15990816 $ devlink sb port pool show pci/0000:03:00.0/1 pool 8 enp3s0np33: sb 0 pool 8 threshold 90048 ``` The number 15990816, which occurs in two contexts above, represents the total shared buffer size reported by the system. Technically the corresponding configuration is an infinity, but there's no way to convey that in devlink. #### Spectrum-2 Shared Buffer Configuration By default, two ingress pools and one egress pool are used. There is no dedicated MC pool, but the switch is still configured in MC-aware mode. ``` # devlink sb pool show pci/0000:06:00.0: sb 0 pool 0 type ingress size 40960080 thtype dynamic cell_size 144 sb 0 pool 1 type ingress size 0 thtype static cell_size 144 sb 0 pool 2 type ingress size 0 thtype static cell_size 144 sb 0 pool 3 type ingress size 200016 thtype dynamic cell_size 144 sb 0 pool 4 type egress size 40960080 thtype dynamic cell_size 144 sb 0 pool 5 type egress size 0 thtype static cell_size 144 sb 0 pool 6 type egress size 0 thtype static cell_size 144 sb 0 pool 7 type egress size 0 thtype static cell_size 144 ``` Further Resources ----------------- 1. [Wikipedia article on DCB][7] 2. man lldptool-ets 3. man lldptool-pfc 4. man lldptool-app 5. man ethtool 6. man devlink 7. man devlink-sb [1]: https://en.wikipedia.org/wiki/IEEE_802.1p [2]: https://en.wikipedia.org/wiki/IEEE_802.1Q#Frame_format [3]: http://open-lldp.org/dcb_overview [4]: http://open-lldp.org/ [5]: http://vger.kernel.org/~davem/skb.html [6]: https://en.wikipedia.org/wiki/Weighted_round_robin [7]: https://en.wikipedia.org/wiki/Data_center_bridging [8]: https://github.com/intel/openlldp/pull/9 [9]: https://en.wikipedia.org/wiki/Differentiated_services