rfc9318.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rfc [
  <!ENTITY nbsp    "&#160;">
  <!ENTITY zwsp   "&#8203;">
  <!ENTITY nbhy   "&#8209;">
  <!ENTITY wj     "&#8288;">
]>

<!-- generated by https://github.com/cabo/kramdown-rfc2629 version 1.3.32 -->

<rfc xmlns:xi="http://www.w3.org/2001/XInclude" ipr="trust200902" docName="draft-iab-mnqeu-report-04" number="9318" submissionType="IAB" category="info" consensus="true" obsoletes="" updates="" xml:lang="en" tocInclude="true" sortRefs="true" symRefs="true" version="3">

<!-- [rfced] 
b) The boilerplate that follows the Abstract appears to be missing.
-->

<!-- Wes: sounds fine -->

<!-- xml2rfc v2v3 conversion 3.14.1 -->
  <front>
    <title abbrev="Measuring Network Quality for End-Users">IAB Workshop Report: Measuring Network Quality for End-Users</title>
    <seriesInfo name="RFC" value="9318"/>
    <author initials="W." surname="Hardaker" fullname="Wes Hardaker">
      <address>
        <email>ietf@hardakers.net</email>
      </address>
    </author>
    <author initials="O." surname="Shapira" fullname="Omer Shapira">
      <address>
        <email>omer_shapira@apple.com</email>
      </address>
    </author>
    <date year="2022" month="September"/>

    <keyword>QoE</keyword>
    <keyword>QoS</keyword>
    <keyword>Quality of Service</keyword>
    <keyword>Quality of Experience</keyword>
    <keyword>Measurement</keyword>
    <keyword>End User</keyword>

    <abstract>
      <t>The Measuring Network Quality for End-Users workshop was held
virtually by the Internet Architecture Board (IAB) on September 14-16, 2021.
This report summarizes the workshop, the topics discussed, and some
      preliminary conclusions drawn at the end of the workshop.</t>
      <t>Note that this document is a report on the proceedings of the 
      workshop.  The views and positions documented in this report are 
      those of the workshop participants and do not necessarily reflect IAB 
      views and positions. </t>
    </abstract>
  </front>
  <middle>
    <section anchor="introduction" numbered="true" toc="default">
      <name>Introduction</name>
      <t>The Internet Architecture Board (IAB) holds occasional workshops designed to
consider long-term issues and strategies for the Internet, and to suggest
future directions for the Internet architecture.  This long-term planning
function of the IAB is complementary to the ongoing engineering efforts
      performed by working groups of the Internet Engineering Task Force (IETF).</t>
      <t>The Measuring Network Quality for End-Users workshop <xref target="WORKSHOP" format="default"/> was held
virtually by the Internet Architecture Board (IAB) on September 14-16, 2021.
This report summarizes the workshop, the topics discussed, and some preliminary
conclusions drawn at the end of the workshop.</t>
      <section anchor="problem-space" numbered="true" toc="default">
        <name>Problem Space</name>
        <t>The Internet in 2021 is quite different from what it was 10 years ago. Today, it
is a crucial part of everyone's daily life. People use the Internet for their
social life, for their daily jobs, for routine shopping, and for keeping up
with major events. An increasing number of people can access a gigabit
connection, which would be hard to imagine a decade ago. Additionally, thanks to
improvements in security, people trust the Internet for financial
banking transactions, purchasing goods, and everyday bill payments.</t>
        <t>At the same time, some aspects of the end-user experience have not
improved as much.  Many users have typical connection latencies that
remain at decade-old levels.  Despite significant reliability
improvements in data center environments, end users also still often see
interruptions in service. Despite algorithmic advances in the field of
control theory, one still finds that the queuing delays in the
last-mile equipment exceeds the accumulated transit delays. Transport
improvements, such as QUIC, Multipath TCP, and TCP Fast Open, are still
not fully supported in some networks. Likewise, various advances in
the security and privacy of user data are not widely supported, such
as encrypted DNS to the local resolver.</t>
        <t>Some of the major factors behind this lack of progress is the popular
perception that throughput is often the sole measure of the quality of
Internet connectivity. With such a narrow focus, the Measuring Network
Quality for End-Users workshop aimed to discuss various topics:</t>
        <ul spacing="normal">
          <li>What is user latency under typical working conditions?</li>
          <li>How reliable is connectivity across longer time periods?</li>
          <li>Do networks allow the use of a broad range of protocols?</li>
          <li>What services can be run by network clients?</li>
          <li>What kind of IPv4, NAT, or IPv6 connectivity is offered, and are there
firewalls?</li>
          <li>What security mechanisms are available for local services, such as DNS?</li>
          <li>To what degree are the privacy, confidentiality, integrity, and authenticity
	  of user communications guarded?</li>
          <li>Improving these aspects of network quality will likely depend on
measuring and exposing metrics in a meaningful way to all involved
parties, including to end users. Such measurement and exposure of
the right metrics will allow service providers and network operators
to concentrate focus on their users' experience and will
simultaneously empower users to choose the Internet Service
Providers (ISPs) that can deliver the best experience based on their needs.</li>
          <li>What are the fundamental properties of a network that contributes to
a good user experience?</li>
          <li>What metrics quantify these properties, and how can we collect such metrics in a
practical way?</li>
          <li>What are the best practices for interpreting those metrics and incorporating
them in a decision-making process?</li>
          <li>What are the best ways to communicate these properties to service providers
and network operators?</li>
          <li>How can these metrics be displayed to users in a meaningful way?</li>
        </ul>
      </section>
    </section>
    <section anchor="workshop-agenda" numbered="true" toc="default">
      <name>Workshop Agenda</name>
      <t>The Measuring Network Quality for End-Users workshop was divided into the
following main topic areas; see further discussion in Sections <xref target="discussions" format="counter"/> and <xref target="conclusions" format="counter"/>:</t>
      <ul spacing="normal">
        <li>Introduction overviews and a keynote by Vint Cerf</li>
        <li>Metrics considerations</li>
        <li>Cross-layer considerations</li>
        <li>Synthesis</li>
        <li>Group conclusions</li>
      </ul>
    </section>
    <section anchor="positionpapers" numbered="true" toc="default">
      <name>Position Papers</name>
      <t>The following position papers were received for consideration by the
workshop attendees.  The workshop's web page <xref target="WORKSHOP" format="default"/> contains
archives of the papers, presentations, and recorded videos.</t>
      <ul spacing="normal">
        <li>Ahmed Aldabbagh. "Regulatory perspective on measuring network quality for end users" <xref target="Aldabbagh2021" format="default"/></li>
        <li>Al Morton. "Dream-Pipe or Pipe-Dream: What Do Users Want (and how can we assure it)?" <xref target="I-D.morton-ippm-pipe-dream" format="default"/></li>
        <li>Alexander Kozlov. "The 2021 National Internet Segment Reliability Research"</li>
        <li>Anna Brunstrom. "Measuring network quality - the MONROE experience"</li>
        <li>Bob Briscoe, Greg White, Vidhi Goel, and Koen De Schepper. "A Single Common Metric to Characterize Varying Packet Delay" <xref target="Briscoe2021" format="default"/></li>
        <li>Brandon Schlinker. "Internet Performance from Facebook's Edge" <xref target="Schlinker2019" format="default"/></li>
        <li>Christoph Paasch, Kristen McIntyre, Randall Meyer, Stuart Cheshire, and Omer Shapira. "An end-user approach to the Internet Score" <xref target="McIntyre2021" format="default"/></li>
        <li>Christoph Paasch, Randall Meyer, Stuart Cheshire, and Omer Shapira. "Responsiveness under Working Conditions" <xref target="I-D.cpaasch-ippm-responsiveness" format="default"/></li>
        <li>Dave Reed and Levi Perigo. "Measuring ISP Performance in Broadband America: A Study of Latency Under Load" <xref target="Reed2021" format="default"/></li>
        <li>Eve M.&nbsp;Schooler and Rick Taylor. "Non-traditional Network Metrics"</li>
        <li>Gino Dion. "Focusing on latency, not throughput, to provide better internet  experience and network quality" <xref target="Dion2021" format="default"/></li>
        <li>Gregory Mirsky, Xiao Min, Gyan Mishra, and Liuyan Han. "The error performance metric in a packet-switched network" <xref target="Mirsky2021" format="default"/></li>
        <li>Jana Iyengar. "The Internet Exists In Its Use" <xref target="Iyengar2021" format="default"/></li>
        <li>Jari Arkko and Mirja Kuehlewind. "Observability is needed to improve network quality" <xref target="Arkko2021" format="default"/></li>
        <li>Joachim Fabini. "Network Quality from an End User Perspective" <xref target="Fabini2021" format="default"/></li>
        <li>Jonathan Foulkes. "Metrics helpful in assessing Internet Quality" <xref target="Foulkes2021" format="default"/></li>
        <li>Kalevi Kilkki and Benajamin Finley. "In Search of Lost QoS" <xref target="Kilkki2021" format="default"/></li>
        <li>Karthik Sundaresan, Greg White, and Steve Glennon. "Latency Measurement: What is latency and how do we measure it?"</li>
        <li>Keith Winstein. "Five Observations on Measuring Network Quality for Users of Real-Time Media Applications"</li>
        <li>Ken Kerpez, Jinous Shafiei, John Cioffi, Pete Chow, and Djamel Bousaber. "Wi-Fi and Broadband Data" <xref target="Kerpez2021" format="default"/></li>
        <li>Kenjiro Cho. "Access Network Quality as Fitness for Purpose"</li>
        <li>Koen De Schepper, Olivier Tilmans, and Gino Dion. "Challenges and opportunities of hardware support for Low Queuing Latency without Packet Loss" <xref target="DeSchepper2021" format="default"/></li>
        <li>Kyle MacMillian and Nick Feamster. "Beyond Speed Test: Measuring Latency Under  Load Across Different Speed Tiers" <xref target="MacMillian2021" format="default"/></li>
        <li>Lucas Pardue and Sreeni Tellakula. "Lower-layer performance not indicative of upper-layer success" <xref target="Pardue2021" format="default"/></li>
        <li>Matt Mathis. "Preliminary Longitudinal Study of Internet Responsiveness" <xref target="Mathis2021" format="default"/></li>
        <li>Michael Welzl. "A Case for Long-Term Statistics" <xref target="Welzl2021" format="default"/></li>
        <li>Mikhail Liubogoshchev. "Cross-layer Cooperation for Better Network Service" <xref target="Liubogoshchev2021" format="default"/></li>
        <li>Mingrui Zhang, Vidhi Goel, and Lisong Xu. "User-Perceived Latency to Measure CCAs" <xref target="Zhang2021" format="default"/></li>
        <li>Neil Davies and Peter Thompson. "Measuring Network Impact on Application Outcomes Using Quality Attenuation" <xref target="Davies2021" format="default"/></li>
        <li>Olivier Bonaventure and Francois Michel. "Packet delivery time as a tie-breaker for assessing Wi-Fi access points" <xref target="Michel2021" format="default"/></li>
        <li>Pedro Casas. "10 Years of Internet-QoE Measurements. Video, Cloud,
Conferencing, Web and Apps. What do we Need from the Network Side?" <xref target="Casas2021" format="default"/></li>
        <li>Praveen Balasubramanian. "Transport Layer Statistics for Network Quality" <xref target="Balasubramanian2021" format="default"/></li>
        <li>Rajat Ghai. "Using TCP Connect Latency for measuring CX and Network
Optimization" <xref target="Ghai2021" format="default"/></li>
        <li>Robin Marx and Joris Herbots. "Merge Those Metrics: Towards Holistic (Protocol) Logging" <xref target="Marx2021" format="default"/></li>
        <li>Sandor Laki, Szilveszter Nadas, Balazs Varga, and Luis M.
Contreras. "Incentive-Based Traffic Management and QoS Measurements" <xref target="Laki2021" format="default"/></li>
        <li>Satadal Sengupta, Hyojoon Kim, and Jennifer Rexford. "Fine-Grained RTT Monitoring Inside the Network" <xref target="Sengupta2021" format="default"/></li>
        <li>Stuart Cheshire. "The Internet is a Shared Network" <xref target="Cheshire2021" format="default"/></li>
        <li>Toerless Eckert and Alex Clemm. "network-quality-eckert-clemm-00.4"</li>
        <li>Vijay Sivaraman, Sharat Madanapalli, and Himal Kumar. "Measuring Network Experience Meaningfully, Accurately, and Scalably" <xref target="Sivaraman2021" format="default"/></li>
        <li>Yaakov (J) Stein. "The Futility of QoS" <xref target="Stein2021" format="default"/></li>
      </ul>
    </section>
    <section anchor="discussions" numbered="true" toc="default">
      <name>Workshop Topics and Discussion</name>
      <t>The agenda for the three-day workshop was broken into four separate
sections that each played a role in framing the discussions. The
workshop started with a series of introduction and problem space
presentations (<xref target="introduction-section"/>), followed by metrics considerations
(<xref target="discussion-metrics" format="default"/>), cross-layer considerations
(<xref target="discussions-cross-layer" format="default"/>), and a synthesis discussion (<xref target="synthesis" format="default"/>).
After the four subsections concluded, a follow-on discussion was held
to draw conclusions that could be agreed upon by workshop participants
(<xref target="conclusions" format="default"/>).</t>
      <section anchor="introduction-section" numbered="true" toc="default">
        <name>Introduction and Overviews</name>
        <t>The workshop started with a broad focus on the state of user Quality
of Service (QoS) and Quality of Experience (QoE) on the Internet today.
The goal of the introductory talks was to set the stage for the
workshop by describing both the problem space and the current
solutions in place and their limitations.</t>
        <t>The introduction presentations provided views of existing QoS and QoE
measurements and their effectiveness. Also discussed was the
interaction between multiple users within the network, as well as the
interaction between multiple layers of the OSI stack.  Vint Cerf
provided a keynote describing the history and importance of the
topic.</t>
        <section anchor="dicsucssion-intro-keynote" numbered="true" toc="default">
          <name>Key Points from the Keynote by Vint Cerf</name>
          <t>We may be operating in a networking space with dramatically different
parameters compared to 30 years ago. This differentiation justifies
reconsidering not only the importance of one metric over the other
but also reconsidering the entire metaphor.</t>
          <t>It is time for the experts to look at not only adjusting TCP but
also exploring other protocols, such as QUIC has done lately. It's
important that we feel free to consider alternatives to TCP. TCP is
not a teddy bear, and one should not be afraid to replace it with a
transport layer with better properties that better benefit its users.</t>
          <t>A suggestion: we should consider exercises to identify desirable
properties. As we are looking at the parametric spaces, one can
identify "desirable properties", as opposed to "fundamental
properties", for example, a low-latency property. An example coming
from the Advanced Research Projects Agency (ARPA): you want to know where the missile is now, not where it
was. Understanding drives particular parameter creation and selection
in the design space.</t>
          <t>When parameter values are changed in extreme, such as connectiveness,
alternative designs will emerge. One case study of note is the
interplanetary protocol, where "ping" is no longer indicative of
anything useful. While we look at responsiveness, we should not ignore
connectivity.</t>
          <t>Unfortunately, maintaining backward compatibility is painful. The work
on designing IPv6 so as to transition from IPv4 could have been done
better if the backward compatibility was considered.
It is too late for IPv6, but it is not too late to consider this issue for potential future problems.</t>
          <t>IPv6 is still not implemented fully everywhere.  It's been a long road
to deployment since starting work in 1996, and we are still not
there. In 1996, the thinking was that it was quite easy to implement
IPv6, but that failed to hold true. In 1996, the dot-com boom began,
where a lot of money was spent quickly, and the moment was not caught in
time while the market expanded exponentially. This should serve as a
cautionary tale.</t>
          <t>One last point: consider performance across multiple hops in the
Internet. We've not seen many end-to-end metrics, as successfully
developing end-to-end measurements across different network and
business boundaries is quite hard to achieve. A good question to ask
when developing new protocols is "will the new protocol work across
multiple network hops?"</t>
          <t>Multi-hop networks are being gradually replaced by humongous, flat
networks with sufficient connectivity between operators so that
systems become 1 hop, or 2 hops at most, away from each other
(e.g., Google, Facebook, and Amazon). The fundamental architecture of the
Internet is changing.</t>
        </section>
        <section anchor="discussion-introductions" numbered="true" toc="default">
          <name>Introductory Talks</name>
          <t>The Internet is a shared network built on IP protocols using
packet switching to interconnect multiple autonomous networks. The
Internet's departure from circuit-switching technologies allowed it to
scale beyond any other known network design. On the other hand, the
lack of in-network regulation made it difficult to ensure the best
experience for every user.</t>
          <t>As Internet use cases continue to expand, it becomes increasingly more
difficult to predict which network characteristics correlate with
better user experiences. Different application classes, e.g., video
streaming and teleconferencing, can affect user experience in ways that are complex
and difficult to measure. Internet utilization shifts rapidly
during the course of each day, week, and year, which further
complicates identifying key metrics capable of predicting a good user
experience.</t>
          <t>QoS initiatives attempted to overcome these
difficulties by strictly prioritizing different types of
traffic. However, QoS metrics do not always correlate with user
experience. The utility of the QoS metric is further limited by the
difficulties in building solutions with the desired QoS
characteristics.</t>
          <t>QoE initiatives attempted to integrate the
psychological aspects of how quality is perceived and create
statistical models designed to optimize the user experience. Despite
these high modeling efforts, the QoE approach proved beneficial in
certain application classes. Unfortunately, generalizing the models
proved to be difficult, and the question of how different applications
affect each other when sharing the same network remains an open problem.</t>
          <t>The industry's focus on giving the end user more throughput/bandwidth
led to remarkable advances. In many places around the world, a home
user enjoys gigabit speeds to their ISP.  This
is so remarkable that it would have been brushed off as science
fiction a decade ago. However, the focus on increased capacity came at
the expense of neglecting another important core metric: latency. As
a result, end users whose experience is negatively affected by high
latency were advised to upgrade their equipment to get more
throughput instead. <xref target="MacMillian2021" format="default"/> showed that sometimes such an
upgrade can lead to latency improvements, due to the economical
reasons of overselling the "value-priced" data plans.</t>
          <t>As the industry continued to give end users more throughput, while
mostly neglecting latency concerns, application designs started to
employ various latency and short service disruption hiding techniques.
For example, a user's web browser performance experience is closely
tied to the content in the browser's local cache. While such
techniques can clearly improve the user experience when using stale
data is possible, this development further decouples user experience
from core metrics.</t>
          <t>In the most recent 10 years, efforts by Dave Taht and the bufferbloat
society have led to significant progress in updating queuing algorithms to
reduce latencies under load compared to simpler FIFO
queues. Unfortunately, the home router industry has yet to implement
these algorithms, mostly due to marketing and cost concerns. Most home
router manufacturers depend on System on a Chip (SoC) acceleration to
create products with a desired throughput. SoC manufacturers opt for
simpler algorithms and aggressive aggregation, reasoning that a
higher-throughput chip will have guaranteed demand. Because consumers
are offered choices primarily among different high-throughput devices,
the perception that a higher throughput leads to higher a QoS continues to strengthen.</t>
          <t>The home router is not the only place that can benefit from clearer
	  indications of acceptable performance for users.
Since users perceive the Internet via the lens of applications, it
is important that we call upon application vendors to adopt solutions
that stress lower latencies.  Unfortunately, while bandwidth is straightforward to
measure, responsiveness is trickier. Many applications have found a
set of metrics that are helpful to their realm but do not generalize
well and cannot become universally applicable. Furthermore, due to the
highly competitive application space, vendors may have economic
reasons to avoid sharing their most useful metrics.</t>
</section>
        <section anchor="discussion-introductions-summary" numbered="true" toc="default">
          <name>Introductory Talks - Key Points</name>
          <ol spacing="normal" type="1">
	    <li>Measuring bandwidth is necessary but is not alone sufficient.</li>
            <li>In many cases, Internet users don't need more bandwidth but rather 
	     need "better bandwidth", i.e., they need other connectivity improvements.</li>
            <li>Users perceive the quality of their Internet connection based
on the applications they use, which are affected by a combination
of factors. There's little value in exposing a typical user to the
entire spectrum of possible reasons for the poor performance
perceived in their application-centric view.</li>
            <li>Many factors affecting user experience are outside the users'
sphere of control. It's unclear whether exposing users to these
other factors will help them understand the state of their network
performance. In general, users prefer simple, categorical
choices (e.g., "good", "better", and "best" options).</li>
            <li>The Internet content market is highly competitive, and many
applications develop their own "secret sauce".</li>
          </ol>
        </section>
      </section>
      <section anchor="discussion-metrics" numbered="true" toc="default">
        <name>Metrics Considerations</name>
        <t>In the second agenda section, the workshop continued its discussion
about metrics that can be used instead of or in addition to available
bandwidth. Several workshop attendees presented deep-dive studies on
measurement methodology.</t>
        <section anchor="common-performance-metrics" numbered="true" toc="default">
          <name>Common Performance Metrics</name>
          <t>Losing Internet access entirely is, of course, the worst user
experience. Unfortunately, unless rebooting the home router restores
connectivity, there is little a user can do other than contacting
their service provider. Nevertheless, there is value in the systematic
collection of availability metrics on the client side; these can help
the user's ISP localize and resolve issues faster while enabling
users to better choose between ISPs. One can measure availability
directly by simply attempting connections from the client side to
distant locations of interest. For example, Ookla's <xref target="Speedtest" format="default"/>
uses a large number of Android devices to measure network and cellular
availability around the globe. Ookla collects hundreds of millions of
data points per day and uses these for accurate availability
reporting. An alternative approach is to derive availability from the
failure rates of other tests. For example, <xref target="FCC_MBA" format="default"/> and
            <xref target="FCC_MBA_methodology" format="default"/> use thousands of off-the-shelf routers, with measurement software developed by
<xref target="SamKnows" format="default"/>. These routers perform an array of network tests and
report availability based on whether test connections were successful or
not.</t>
          <t>Measuring available capacity can be helpful to end users, but it is
even more valuable for service providers and application
developers. High-definition video streaming requires significantly
more capacity than any other type of traffic. At the time of the
workshop, video traffic constituted 90% of overall Internet traffic
and contributed to 95% of the revenues from monetization (via
subscriptions, fees, or ads). As a result, video streaming services,
such as Netflix, need to continuously cope with rapid changes in
available capacity. The ability to measure available capacity in
real time leverages the different adaptive bitrate (ABR) compression
algorithms to ensure the best possible user experience. Measuring
aggregated capacity demand allows ISPs to be
ready for traffic spikes. For example, during the end-of-year holiday
season, the global demand for capacity has been shown to be 5-7 times
higher than during other seasons.  For end users, knowledge of their
capacity needs can help them select the best data plan given their
intended usage. In many cases, however, end users have more than
enough capacity, and adding more bandwidth will not improve their
experience -- after a point, it is no longer the limiting factor in
user experience. Finally, the ability to differentiate between the
"throughput" and the "goodput" can be helpful in identifying when the
network is saturated.</t>
          <t>In measuring network quality, latency is defined as the time it takes
a packet to traverse a network path from one end to the other. At the
time of this report, users in many places worldwide can enjoy Internet
access that has adequately high capacity and availability for their
current needs. For these users, latency improvements, rather than
bandwidth improvements, can lead to the most significant improvements
in QoE. The established latency metric is a
round-trip time (RTT), commonly measured in milliseconds. However,
users often find RTT values unintuitive since, unlike other
performance metrics, high RTT values indicate poor latency and users
typically understand higher scores to be better. To address this,
<xref target="I-D.cpaasch-ippm-responsiveness" format="default"/> and <xref target="Mathis2021" format="default"/> present an inverse metric, called
"Round-trips Per Minute" (RPM).</t>
          <t>There is an important distinction between "idle latency" and "latency
under working conditions". The former is measured when the network is
underused and reflects a best-case scenario. The latter is measured
when the network is under a typical workload. Until recently, typical
tools reported a network's idle latency, which can be misleading. For
example, data presented at the workshop shows that idle latencies can
be up to 25 times lower than the latency under typical working
loads. Because of this, it is essential to make a clear distinction
between the two when presenting latency to end users.</t>
          <t>Data shows that rapid changes in capacity affect
latency. <xref target="Foulkes2021" format="default"/> attempts to quantify how often a rapid change
in capacity can cause network connectivity to become "unstable" (i.e.,
having high latency with very little throughput). Such changes in
capacity can be caused by infrastructure failures but are much more
often caused by in-network phenomena, like changing traffic
engineering policies or rapid changes in cross-traffic.</t>
          <t>Data presented at the workshop shows that 36% of measured lines have
capacity metrics that vary by more than 10% throughout the day and
across multiple days. These differences are caused by many variables,
including local connectivity methods (Wi-Fi vs. Ethernet), competing
LAN traffic, device load/configuration, time of day, and local
loop/backhaul capacity. These factor variations make measuring
capacity using only an end-user device or other end-network
measurement difficult. A network router seeing aggregated traffic from
multiple devices provides a better vantage point for capacity
measurements. Such a test can account for the totality of local
traffic and perform an independent capacity test. However, various
factors might still limit the accuracy of such a test. Accurate
	  capacity measurement requires multiple samples.</t>
          <t>As users perceive the Internet through the lens of applications, it
may be difficult to correlate changes in capacity and latency with the
quality of the end-user experience. For example, web browsers rely on
cached page versions to shorten page load times and mitigate
connectivity losses. In addition, social networking applications often
rely on prefetching their "feed" items. These techniques make the
core in-network metrics less indicative of the users' experience and
necessitates collecting data from the end-user applications themselves.</t>
          <t>It is helpful to distinguish between applications that operate on a
"fixed latency budget" from those that have more tolerance to latency
variance. Cloud gaming serves as an example application that requires
a "fixed latency budget", as a sudden latency spike can decide the
"win/lose" ratio for a player. Companies that compete in the lucrative
cloud gaming market make significant infrastructure investments, such
as building entire data centers closer to their users. These data
centers highlight the economic benefit that lower numbers of latency
spikes outweigh the associated deployment costs. On the other hand,
applications that are more tolerant to latency spikes can continue to
operate reasonably well through short spikes. Yet, even those
applications can benefit from consistently low latency depending on
usage shifts. For example, Video-on-Demand (VOD) apps can work
reasonably well when the video is consumed linearly, but once the user
tries to "switch a channel" or to "skip ahead", the user experience
suffers unless the latency is sufficiently low.</t>
          <t>Finally, as applications continue to evolve, in-application metrics
are gaining in importance. For example, VOD applications can assess
the QoE by application-specific metrics, such as
whether the video player is able to use the highest possible
resolution, identifying when the video is smooth or freezing, or other
similar metrics. Application developers can then effectively use these
metrics to prioritize future work. All popular video platforms
(YouTube, Instagram, Netflix, and others) have developed frameworks to
collect and analyze VOD metrics at scale. One example is the Scuba
framework used by Meta <xref target="Scuba" format="default"/>.</t>
          <t>Unfortunately, in-application metrics can be challenging to use
for comparative research purposes. First, different applications
often use different metrics to measure the same phenomena. For
example, application A may measure the smoothness of video via "mean
time to rebuffer", while application B may rely on the "probability
of rebuffering per second" for the same purpose. A different
challenge with in-application metrics is that VOD is a significant source
of revenue for companies, such as YouTube, Facebook, and Netflix,
placing a proprietary incentive against exchanging the in-application
data. A final concern centers on the privacy issues resulting from
in-application metrics that accurately describe the activities and
preferences of an individual end user.</t>
        </section>
        <section anchor="availability-metrics" numbered="true" toc="default">
          <name>Availability Metrics</name>
          <t>Availability is simply defined as whether or not a packet can be sent
and then received by its intended recipient.  Availability is naively
thought to be the simplest to measure, but it is more complex when
considering that continual, instantaneous measurements would be needed
to detect the smallest of outages.  Also difficult is determining the
root cause of infallibility: was the user's line down, was something in
the middle of the network, or was it the service with which the user
was attempting to communicate?</t>
        </section>
        <section anchor="capacity-metrics" numbered="true" toc="default">
          <name>Capacity Metrics</name>
          <t>If the network capacity does not meet user demands, the network quality
will be impacted. Once the capacity meets the demands, increasing capacity
won't lead to further quality improvements.</t>
          <t>The actual network connection capacity is determined by the equipment and the
lines along the network path, and it varies throughout the day and across
multiple days. Studies involving DSL lines in North America indicate that over
30% of the DSL lines have capacity metrics that vary by more than 10%
throughout the day and across multiple days.</t>
          <t>Some factors that affect the actual capacity are:</t>
          <ol spacing="normal" type="1">
	    <li>Presence of a competing traffic, either in the LAN or in the WAN
environments. In the LAN setting, the competing traffic reflects the
multiple devices that share the Internet connection. In the WAN setting, the
competing traffic often originates from the unrelated network flows that
happen to share the same network path.</li>
            <li>Capabilities of the equipment along the path of the network connection,
including the data transfer rate and the amount of memory used for
buffering.</li>
            <li>Active traffic management measures, such as traffic shapers and policers
that are often used by the network providers.</li>
          </ol>
          <t>There are other factors that can negatively affect the actual line capacities.</t>
          <t>The user demands of the traffic follow the usage patterns and preferences of
the particular users. For example, large data transfers can use any available
capacity, while the media streaming applications require limited capacity to
function correctly. Videoconferencing applications typically need less
capacity than high-definition video streaming.</t>
        </section>
        <section anchor="latency-metrics" numbered="true" toc="default">
          <name>Latency Metrics</name>
          <t>End-to-end latency is the time that a particular packet takes to traverse the
network path from the user to their destination and back.  The end-to-end
latency comprises several components:</t>
           <ol spacing="normal" type="1">
	     <li>The propagation delay, which reflects the path distance and the individual
link technologies (e.g., fiber vs. satellite). The propagation doesn't depend
on the utilization of the network, to the extent that the network path
remains constant.</li>
            <li>The buffering delay, which reflects the time segments spent in the memory of
the network equipment that connect the individual network links, as well as
in the memory of the transmitting endpoint. The buffering delay depends on
the network utilization, as well as on the algorithms that govern the queued segments.</li>
            <li>The transport protocol delays, which reflect the time spent in
retransmission and reassembly, as well as the time spent when the transport
is "head-of-line blocked".</li>
            <li>Some of the workshop submissions that have explicitly called out the application
delay, which reflects the inefficiencies in the application layer.</li>
          </ol>
          <t>Typically, end-to-end latency is measured when the network is
idle. Results of such measurements mostly reflect the propagation
delay but not other kinds of delay. This report uses the term "idle
latency" to refer to results achieved under idle network conditions.</t>
          <t>Alternatively, if the latency is measured when the network is under
its typical working conditions, the results reflect multiple types of
delays. This report uses the term "working latency" to refer to such
results. Other sources use the term "latency under load" (LUL) as a
synonym.</t>
          <t>Data presented at the workshop reveals a substantial difference
between the idle latency and the working latency. Depending on the
traffic direction and the technology type, the working latency is
between 6 to 25 times higher than the idle latency:</t>
          <table align="center">
            <thead>
              <tr>
                <th align="left">Direction</th>
                <th align="left">Technology Type</th>
                <th align="left">Working Latency</th>
                <th align="left">Idle Latency</th>
                <th align="left">Working - Idle Difference</th>
                <th align="left">Working / Idle Ratio</th>
              </tr>
            </thead>
            <tbody>
              <tr>
                <td align="left">Downstream</td>
                <td align="left">FTTH</td>
                <td align="left">148</td>
                <td align="left">10</td>
                <td align="left">138</td>
                <td align="left">15</td>
              </tr>
              <tr>
                <td align="left">Downstream</td>
                <td align="left">Cable</td>
                <td align="left">103</td>
                <td align="left">13</td>
                <td align="left">90</td>
                <td align="left">8</td>
              </tr>
              <tr>
                <td align="left">Downstream</td>
                <td align="left">DSL</td>
                <td align="left">194</td>
                <td align="left">10</td>
                <td align="left">184</td>
                <td align="left">19</td>
              </tr>
              <tr>
                <td align="left">Upstream</td>
                <td align="left">FTTH</td>
                <td align="left">207</td>
                <td align="left">12</td>
                <td align="left">195</td>
                <td align="left">17</td>
              </tr>
              <tr>
                <td align="left">Upstream</td>
                <td align="left">Cable</td>
                <td align="left">176</td>
                <td align="left">27</td>
                <td align="left">149</td>
                <td align="left">6</td>
              </tr>
              <tr>
                <td align="left">Upstream</td>
                <td align="left">DSL</td>
                <td align="left">686</td>
                <td align="left">27</td>
                <td align="left">659</td>
                <td align="left">25</td>
              </tr>
            </tbody>
          </table>
          <t>While historically the tooling available for measuring latency focused
on measuring the idle latency, there is a trend in the industry to
start measuring the working latency as well,
e.g., Apple's <xref target="NetworkQuality" format="default"/>.</t>
</section>
        <section anchor="measurement-case-studies" numbered="true" toc="default">
          <name>Measurement Case Studies</name>
          <t>The participants have proposed several concrete methodologies for
measuring the network quality for the end users.</t>
          <t><xref target="I-D.cpaasch-ippm-responsiveness" format="default"/> introduced a methodology for measuring working latency
from the end-user vantage point. The suggested method incrementally
adds network flows between the user device and a server endpoint until
a bottleneck capacity is reached. From these measurements, a round-trip
latency is measured and reported to the end user. The authors
chose to report results with the RPM metric. The methodology had been
implemented in Apple's macOS Monterey.</t>
          <t><xref target="Mathis2021" format="default"/> applied the RPM metric to the results of more than
4 billion download tests that M-Lab performed from 2010-2021. During
this time frame, the M-Lab measurement platform underwent several
upgrades that allowed the research team to compare the effect of
different TCP congestion control algorithms (CCAs) on the measured
end-to-end latency. The study showed that the use of cubic CCA leads to
increased working latency, which is attributed to its use of larger
queues.</t>
          <t><xref target="Schlinker2019" format="default"/> presented a large-scale study that aimed to
establish a correlation between goodput and QoE on a
large social network. The authors performed the measurements at
multiple data centers from which video segments of set sizes were
streamed to a large number of end users. The authors used the goodput
and throughput metrics to determine whether particular paths were
congested.</t>
          <t><xref target="Reed2021" format="default"/> presented the analysis of working latency measurements collected as part of the Measuring Broadband America (MBA)
program by the Federal Communication Commission (FCC). The FCC does not include working latency in its yearly report
but does offer it in the raw data files. The authors used a
subset of the raw data to identify important differences in the
working latencies across different ISPs.</t>
          <t><xref target="MacMillian2021" format="default"/> presented analysis of working latency across
multiple service tiers. They found that, unsurprisingly, "premium"
tier users experienced lower working latency compared to a "value"
tier. The data demonstrated that working latency varies significantly
within each tier; one possible explanation is the difference in
equipment deployed in the homes.</t>
          <t>These studies have stressed the importance of measurement of working
latency. At the time of this report, many home router manufacturers
rely on hardware-accelerated routing that uses FIFO queues. Focusing
on measuring the working latency measurements on these devices and
making the consumer aware of the effect of choosing one manufacturer
vs. another can help improve the home router situation. The ideal
test would be able to identify the working latency and pinpoint
the source of the delay (home router, ISP, server side, or some network
node in between).</t>
          <t>Another source of high working latency comes from network routers
exposed to cross-traffic. As <xref target="Schlinker2019" format="default"/> indicated, these can
become saturated during the peak hours of the day. Systematic testing
of the working latency in routers under load can help improve both our
understanding of latency and the impact of deployed infrastructure.</t>
        </section>
        <section anchor="discussions-metrics-key-points" numbered="true" toc="default">
          <name>Metrics Key Points</name>
          <t>The metrics for network quality can be roughly grouped into the following:</t>
          <ol spacing="normal" type="1">
	    <li>Availability metrics, which indicate whether the user can access
the network at all.</li>
            <li>Capacity metrics, which indicate whether the actual line capacity is
sufficient to meet the user's demands.</li>
            <li>Latency metrics, which indicate if the user gets the data in a timely fashion.</li>
            <li>Higher-order metrics, which include both the network metrics, such as
inter-packet arrival time, and the application metrics, such as the mean
time between rebuffering for video streaming.</li>
          </ol>
          <t>The availability metrics can be seen as a derivative of either the capacity (zero
capacity leading to zero availability) or the latency (infinite latency
leading to zero availability).</t>
          <t>Key points from the presentations and discussions included the following:</t>
          <ol spacing="normal" type="1">
	    <li>Availability and capacity are "hygienic factors" -- unless an
application is capable of using extra capacity, end users will see
little benefit from using over-provisioned lines.</li>
            <li>Working latency has a stronger correlation with the user experience
than latency under an idle network load. Working latency can
exceed the idle latency by order of magnitude.</li>
            <li>The RPM metric is a stable metric, with positive values being
better, that may be more effective when communicating latency to
end users.</li>
            <li>The relationship between throughput and goodput can be effective in
finding the saturation points, both in client-side <xref target="I-D.cpaasch-ippm-responsiveness" format="default"/>
and server-side <xref target="Schlinker2019" format="default"/> settings.</li>
            <li>Working latency depends on the algorithm choice for addressing endpoint
congestion control and router queuing.</li>
          </ol>
          <t>Finally, it was commonly agreed to that the best metrics are those
that are actionable.</t>
        </section>
      </section>
      <section anchor="discussions-cross-layer" numbered="true" toc="default">
        <name>Cross-Layer Considerations</name>
        <t>In the cross-layer segment of the workshop, participants presented
material on and discussed how to accurately measure exactly where
problems occur.  Discussion centered especially on the differences
between physically wired and wireless connections and the difficulties
of accurately determining problem spots when multiple different types
of network segments are responsible for the quality.  As an example,
<xref target="Kerpez2021" format="default"/> showed that a limited bandwidth of 2.4 Ghz Wi-Fi bottlenecks the most frequently. In comparison, the wider bandwidth of
the 5 Ghz Wi-Fi has only bottlenecked in 20% of observations.</t>
        <t>The participants agreed that no single component of a network
connection has all the data required to measure the effects of the
network performance on the quality of the end-user experience.</t>
        <ul spacing="normal">
          <li>Applications that are running on the end-user devices have the best
insight into their respective performance but have limited
visibility into the behavior of the network itself and are unable
to act based on their limited perspective.</li>
          <li>ISPs have good insight into QoS
considerations but are not able to infer the effect of the QoS
metrics on the quality of end-user experiences.</li>
          <li>Content providers have good insight into the aggregated behavior of
the end users but lack the insight on what aspects of network
performance are leading indicators of user behavior.</li>
        </ul>
        <t>The workshop had identified the need for a standard and extensible way
to exchange network performance characteristics. Such an exchange
standard should address (at least) the following:</t>
        <ul spacing="normal">
          <li>A scalable way to capture the performance of multiple (potentially
thousands of) endpoints.</li>
          <li>The data exchange format should prevent data manipulation so that
the different participants won't be able to game the mechanisms.</li>
          <li>Preservation of end-user privacy. In particular, federated learning
approaches should be preferred so that no centralized entity has the
access to the whole picture.</li>
          <li>A transparent model for giving the different actors on a network
connection an incentive to share the performance data they collect.</li>
          <li>An accompanying set of tools to analyze the data.</li>
        </ul>
        <section anchor="separation-of-concerns" numbered="true" toc="default">
          <name>Separation of Concerns</name>
          <t>Commonly, there's a tight coupling between collecting performance
metrics, interpreting those metrics, and acting upon the
interpretation.  Unfortunately, such a model is not the best for
successfully exchanging cross-layer data, as:</t>
          <ul spacing="normal">
            <li>actors that are able to collect particular performance metrics
(e.g., the TCP RTT) do not necessarily have the context necessary for
a meaningful interpretation,</li>
            <li>the actors that have the context and the computational/storage
capacity to interpret metrics do not necessarily have the ability to
control the behavior of the network/application, and</li>
            <li>the actors that can control the behavior of networks and/or
applications typically do not have access to complete measurement
data.</li>
          </ul>
          <t>The participants agreed that it is important to separate the above
three aspects, so that:</t>
          <ul spacing="normal">
            <li>the different actors that have the data, but not the ability to
interpret and/or act upon it, should publish their measured data and</li>
            <li>the actors that have the expertise in interpreting and synthesizing
performance data should publish the results of their interpretations.</li>
          </ul>
        </section>
        <section anchor="security-and-privacy-considerations" numbered="true" toc="default">
          <name>Security and Privacy Considerations</name>
          <t>Preserving the privacy of Internet end users is a difficult
requirement to meet when addressing this problem space. There is an
intrinsic trade-off between collecting more data about user
activities and infringing on their privacy while doing so.
Participants agreed that observability across multiple layers is
necessary for an accurate measurement of the network quality, but
doing so in a way that minimizes privacy leakage is an open question.</t>
        </section>
        <section anchor="metric-measurement-considerations" numbered="true" toc="default">
          <name>Metric Measurement Considerations</name>
          <ul spacing="normal">
            <li>
              <t>The following TCP protocol metrics have been found to be effective
and are available for passive measurement:
              </t>
              <ul spacing="normal">
                <li>TCP connection latency measured using selective acknowledgment (SACK) or acknowledgment (ACK) timing, as well as
the timing between TCP retransmission events, are good proxies for
end-to-end RTT measurements.</li>
                <li>On the Linux platform, the tcp_info structure is the de facto
standard for an application to inspect the performance of
kernel-space networking. However, there is no equivalent
de facto standard for user-space networking.</li>
              </ul>
            </li>
            <li>
              <t>The QUIC and MASQUE protocols make passive performance measurements
more challenging.
              </t>
              <ul spacing="normal">
                <li>An approach that uses federated measurement/hierarchical
aggregation may be more valuable for these protocols.</li>
                <li>The QLOG format seems to be the most mature candidate for such
an exchange.</li>
              </ul>
            </li>
          </ul>
        </section>
        <section anchor="discussions-cross-observability" numbered="true" toc="default">
          <name>Towards Improving Future Cross-Layer Observability</name>
          <t>The ownership of the Internet is spread across multiple administrative
domains, making measurement of end-to-end performance data
difficult. Furthermore, the immense scale of the Internet makes
aggregation and analysis of this difficult. <xref target="Marx2021" format="default"/> presented a
simple logging format that could potentially be used to collect and
aggregate data from different layers.</t>
          <t>Another aspect of the cross-layer collaboration hampering measurement is
that the majority of current algorithms do not explicitly provide
performance data that can be used in cross-layer analysis. The IETF
community could be more diligent in identifying each protocol's key
performance indicators and exposing them as part of the protocol
specification.</t>
          <t>Despite all these challenges, it should still be possible to perform
limited-scope studies in order to have a better understanding of how
user quality is affected by the interaction of the different
components that constitute the Internet. Furthermore, recent
development of federated learning algorithms suggests that it might be
possible to perform cross-layer performance measurements while
preserving user privacy.</t>
        </section>
        <section anchor="discussions-cross-layer-hw-tp" numbered="true" toc="default">
          <name>Efficient Collaboration between Hardware and Transport Protocols</name>
          <t>With the advent of the low latency, low loss, and scalable throughput
(L4S) congestion notification and control, there is an even higher
need for the transport protocols and the underlying hardware to work
in unison.</t>
          <t>At the time of the workshop, the typical home router uses a single
FIFO queue that is large enough to allow amortizing the lower-layer header
overhead across multiple transport PDUs. These designs worked well
with the cubic congestion control algorithm, yet the newer generation
of algorithms can operate on much smaller queues. To fully support latencies
less than 1 ms, the home router needs to work efficiently on sequential
transmissions of just a few segments vs. being optimized for large
packet bursts.</t>
          <t>Another design trait common in home routers is the use of packet
aggregation to further amortize the overhead added by the lower-layer
headers.  Specifically, multiple IP datagrams are combined into a
single, large transfer frame. However, this aggregation can add up to
10 ms to the packet sojourn delay.</t>
          <t>Following the famous "you can't improve what you don't measure" adage,
it is important to expose these aggregation delays in a way that would
allow identifying the source of the bottlenecks and making hardware
more suitable for the next generation of transport protocols.</t>
        </section>
        <section anchor="cross-layer-keypoints" numbered="true" toc="default">
          <name>Cross-Layer Key Points</name>
          <ul spacing="normal">
            <li>Significant differences exist in the characteristics of metrics to be measured and the required optimizations needed in wireless vs. wired
networks.</li>
            <li>Identification of an issue's root cause is hampered by the
challenges in measuring multi-segment network paths.</li>
            <li>No single component of a network connection has all the data
required to measure the effects of the complete network performance
on the quality of the end-user experience.</li>
            <li>Actionable results require both proper collection and interpretation.</li>
            <li>Coordination among network providers is important to successfully
improve the measurement of end-user experiences.</li>
            <li>Simultaneously providing accurate measurements while preserving
end-user privacy is challenging.</li>
            <li>Passive measurements from protocol implementations may provide
beneficial data.</li>
          </ul>
        </section>
      </section>
      <section anchor="synthesis" numbered="true" toc="default">
        <name>Synthesis</name>
        <t>Finally, in the synthesis section of the workshop, the presentations
and discussions concentrated on the next steps likely needed to make
forward progress. Of particular concern is how to bring forward
measurements that can make sense to end users trying to select
between various networking subscription options.</t>
        <section anchor="measurement-and-metrics-considerations" numbered="true" toc="default">
          <name>Measurement and Metrics Considerations</name>
          <t>One important consideration is how decisions can be made and what actions
can be taken based on collected metrics.  Measurements must be integrated
with applications in order to get true application views of
congestion, as measurements over different infrastructure or via other
applications may return incorrect results.  Congestion itself can be a
temporary problem, and mitigation strategies may need to be different
depending on whether it is expected to be a short-term or long-term
phenomenon.  A significant challenge exists in measuring short-term
problems, driving the need for continuous measurements to ensure
critical moments and long-term trends are captured.  For short-term
problems, workshop participants debated whether an issue that goes
away is indeed a problem or is a sign that a network is properly
adapting and self-recovering.</t>
          <t>Important consideration must be taken when constructing metrics in
order to understand the results.  Measurements can also be affected by
individual packet characteristics -- differently sized packets typically have a
linear relationship with their delay. With this in mind,
measurements can be divided into a delay based on geographical
distances, a packet-size serialization delay, and a variable (noise)
delay.  Each of these three sub-component delays can be different and
individually measured across each segment in a multi-hop path.
Variable delay can also be significantly impacted by external factors,
such as bufferbloat, routing changes, network load sharing, and other
local or remote changes in performance.  Network measurements,
especially load-specific tests, must also be run long enough to ensure
that any problems associated with buffering, queuing, etc. are captured.
Measurement technologies should also distinguish between upstream and
downstream measurements, as well as measure the difference between
end-to-end paths and sub-path measurements.</t>
        </section>
        <section anchor="end-user-metrics-presentation" numbered="true" toc="default">
          <name>End-User Metrics Presentation</name>
          <t>Determining end-user needs requires informative measurements and
metrics.  How do we provide the users with the service they need or
want? Is it possible for users to even voice their desires
effectively?  Only high-level, simplistic answers like "reliability",
"capacity", and "service bundling" are typical answers given in
end-user surveys.  Technical requirements that operators can consume,
like "low-latency" and "congestion avoidance", are not terms known to
and used by end users.</t>
          <t>Example metrics useful to end users might include the number of users
supported by a service and the number of applications or streams that
a network can support.  An example solution to combat networking
issues include incentive-based traffic management strategies (e.g., an
application requesting lower latency may also mean accepting lower
bandwidth).  User-perceived latency must be considered, not just
network latency -- user experience in-application to in-server
latency and network-to-network measurements may only be studying the
lowest-level latency.  Thus, picking the right protocol to use in a
measurement is critical in order to match user experience (for
example, users do not transmit data over ICMP, even though it is a
common measurement tool).</t>
          <t>In-application measurements should consider how to measure different
types of applications, such as video streaming, file sharing,
multi-user gaming, and real-time voice communications.  It may be that
asking users for what trade-offs they are willing to accept would be a
helpful approach: would they rather have a network with low latency
or a network with higher bandwidth?  Gamers may make different
decisions than home office users or content producers, for example.</t>
          <t>Furthermore, how can users make these trade-offs in a fair manner that
does not impact other users? There is a tension between solutions in
this space vs. the cost associated with solving these problems, as well as
which customers are willing to front these improvement costs.</t>
          <t>Challenges in providing higher-priority traffic to users centers
around the ability for networks to be willing to listen to client
requests for higher incentives, even though commercial interests may
not flow to them without a cost incentive.  Shared mediums in general
are subject to oversubscribing, such that the number of users a network
can support is either accurate on an underutilized network or may
assume an average bandwidth or other usage metric that fails to be
accurate during utilization spikes.  Individual metrics are also
affected by in-home devices from cheap routers to microwaves and by
(multi-)user behaviors during tests.  Thus, a single metric alone or a
single reading without context may not be useful in assisting a user
or operator to determine where the problem source actually is.</t>
          <t>User comprehension of a network remains a challenging problem.
Multiple workshop participants argued for a single number (potentially
calculated with a weighted aggregation formula) or a small number of
measurements per expected usage (e.g., a "gaming" score vs. a "content
producer" score).  Many agreed that some users may instead prefer to
consume simplified or color-coded ratings (e.g., good/better/best,
red/yellow/green, or bronze/gold/platinum).</t>
        </section>
        <section anchor="synthesis-key-points" numbered="true" toc="default">
          <name>Synthesis Key Points</name>
          <ul spacing="normal">
            <li>
              <t>Some proposed metrics:</t>
              <ul spacing="normal">
                <li>Round-trips Per Minute (RPM)</li>
                <li>users per network</li>
                <li>latency</li>
                <li>99% latency and bandwidth</li>
              </ul>
            </li>
            <li>Median and mean measurements are distractions from the real problems.</li>
            <li>Shared network usage greatly affects quality.</li>
            <li>Long measurements are needed to capture all facets of potential
network bottlenecks.</li>
            <li>Better-funded research in all these areas is needed for progress.</li>
            <li>End users will best understand a simplified score or ranking system.</li>
          </ul>
        </section>
      </section>
    </section>
    <section anchor="conclusions" numbered="true" toc="default">
      <name>Conclusions</name>
 
      <t>During the final hour of the three-day workshop, statements that the group deemed to be summary statements were gathered. Later, any statements that were in contention were discarded (listed further below for completeness).
For this document, the authors took the original list
and divided it into rough categories, applied some suggested edits
discussed on the mailing list, and further edited for clarity and to
provide context.</t>
      <section anchor="general-statements" numbered="true" toc="default">
        <name>General Statements</name>
        <ol spacing="normal" type="1">
	  <li>Bandwidth is necessary but not alone sufficient.</li>
          <li>In many cases, Internet users don't need more bandwidth but rather
need "better bandwidth", i.e., they need other improvements to
their connectivity.</li>
          <li>We need both active and passive measurements -- passive measurements
can provide historical debugging.</li>
          <li>We need passive measurements to be continuous, archivable, and
queriable, including reliability/connectivity measurements.</li>
          <li>A really meaningful metric for users is whether their application
will work properly or fail because of a lack of a network with
sufficient characteristics.</li>
          <li>A useful metric for goodness must actually incentivize goodness --
good metrics should be actionable to help drive industries towards
improvement.</li>
          <li>A lower-latency Internet, however achieved, would benefit all end
users.</li>
        </ol>
      </section>
      <section anchor="specific-statements-about-detailed-protocolstechniques" numbered="true" toc="default">
        <name>Specific Statements about Detailed Protocols/Techniques</name>
        <ol spacing="normal" type="1">
	  <li>Round-trips Per Minute (RPM) is a useful, consumable metric.</li>
          <li>We need a usable tool that fills the current gap between network
reachability, latency, and speed tests.</li>
          <li>End users that want to be involved in QoS decisions should be able
to voice their needs and desires.</li>
          <li>Applications are needed that can perform and report good quality
measurements in order to identify insufficient points in
network access.</li>
          <li>Research done by regulators indicate that users/consumers prefer
a simple metric per application, which frequently resolves to
whether the application will work properly or not.</li>
          <li>New measurements and QoS or QoE techniques should not rely only or
depend on reading TCP headers.</li>
          <li>It is clear from developers of interactive applications and from
network operators that lower latency is a strong factor in user
QoE.  However, metrics are lacking to support this statement
directly.</li>
        </ol>
      </section>
      <section anchor="problem-statements-and-concerns" numbered="true" toc="default">
        <name>Problem Statements and Concerns</name>
        <ol spacing="normal" type="1">
	  <li>Latency mean and medians are distractions from better measurements.</li>
          <li>It is frustrating to only measure network services without
simultaneously improving those services.</li>
          <li>Stakeholder incentives aren't aligned for easy wins in this space.
Incentives are needed to motivate improvements in public network
access.  Measurements may be one step towards driving competitive
market incentives.</li>
          <li>For future-proof networking, it is important to measure the
ecological impact of material and energy usage.</li>
          <li>We do not have incontrovertible evidence that any one metric
(e.g., latency or speed) is more important than others to persuade
device vendors to concentrate on any one optimization.</li>
        </ol>
      </section>
      <section anchor="no-consensus-reached-statements" numbered="true" toc="default">
        <name>No-Consensus-Reached Statements</name>
        <t>Additional statements were discussed and recorded that did not have consensus of the
group at the time, but they are listed here for completeness:</t>
        <ol spacing="normal" type="1">
	  <li>We do not have incontrovertible evidence that bufferbloat is a
prevalent problem.</li>
          <li>
            <t>The measurement needs to support reporting localization in order to
find problems.  Specifically:</t>
            <ul spacing="normal">
              <li>Detecting a problem is not sufficient if you can't find the location.</li>
              <li>Need more than just English -- different localization concerns.</li>
            </ul>
          </li>
          <li>Stakeholder incentives aren't aligned for easy wins in this space.</li>
        </ol>
      </section>
    </section>
    <section anchor="follow-on-work" numbered="true" toc="default">
      <name>Follow-On Work</name>
      <t>There was discussion during the workshop about where future work
should be performed.  The group agreed that some work could be done
more immediately within existing IETF working groups (e.g., IPPM,
DetNet, and RAW), while other longer-term research may be needed in
IRTF groups.</t>
    </section>
    <section anchor="iana-considerations" numbered="true" toc="default">
      <name>IANA Considerations</name>
      <t>This document has no IANA actions.</t>
    </section>
    <section anchor="security-considerations" numbered="true" toc="default">
      <name>Security Considerations</name>
      <t>A few security-relevant topics were discussed at the workshop,
including but not limited to:</t>
      <ul spacing="normal">
        <li>what prioritization techniques can work without invading the privacy
of the communicating parties and</li>
        <li>how oversubscribed networks can essentially be viewed as a DDoS
attack.</li>
      </ul>
    </section>
  </middle>
  <back>
    <displayreference target="I-D.morton-ippm-pipe-dream" to="Morton2021"/>
    <displayreference target="I-D.cpaasch-ippm-responsiveness" to="Paasch2021"/>

    <references>
      
      <name>Informative References</name>

      <reference anchor="FCC_MBA" target="https://www.fcc.gov/general/measuring-broadband-america">
        <front>
          <title>Measuring Broadband America</title>
          <author>
            <organization>FCC</organization>
          </author>
        </front>
      </reference>

      <reference anchor="FCC_MBA_methodology" target="https://www.fcc.gov/general/measuring-broadband-america-open-methodology">
        <front>
          <title>Measuring Broadband America - Open Methodology</title>
          <author>
            <organization>FCC</organization>
          </author>
        </front>
      </reference>

      <reference anchor="Scuba" target="https://research.facebook.com/publications/scuba-diving-into-data-at-facebook/">
        <front>
          <title>Scuba: Diving into Data at Facebook</title>
          <author initials="" surname="Abraham, L. et al."/>
        </front>
      </reference>

      <reference anchor="WORKSHOP" target="https://www.iab.org/activities/workshops/network-quality">
        <front>
          <title>IAB Workshop: Measuring Network Quality for End-Users, 2021</title>
          <author>
            <organization>IAB</organization>
          </author>
          <date year="2021" month="September"/>
        </front>
      </reference>

      <reference anchor="Cheshire2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/draft-cheshire-internet-is-shared-00b.pdf">
        <front>
          <title>The Internet is a Shared Network</title>
          <author initials="S." surname="Cheshire">
            <organization/>
          </author>
          <date year="2021" month="August"/>
        </front>
      </reference>

      <reference anchor="Iyengar2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/The-Internet-Exists-In-Its-Use.pdf">
        <front>
          <title>The Internet Exists In Its Use</title>
          <author initials="J." surname="Iyengar">
            <organization/>
          </author>
	  <date year="2021" month="August"/>
        </front>
      </reference>

      <reference anchor="Stein2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/QoS-futility.pdf">
        <front>
          <title>The Futility of QoS</title>
          <author initials="Y." surname="Stein">
            <organization/>
          </author>
	  <date year="2021" month="August"/>
        </front>
      </reference>

      <reference anchor="Casas2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/net_quality_internet_qoe_CASAS.pdf">
        <front>
          <title>10 Years of Internet-QoE Measurements Video, Cloud, Conferencing, Web and Apps. What do we need from the Network Side?</title>
          <author initials="P." surname="Casas">
            <organization/>
          </author>
          <date year="2021" month="August"/>
        </front>
      </reference>

      <reference anchor="Pardue2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Lower-layer-performance-is-not-indicative-of-upper-layer-success-20210906-00-1.pdf">
        <front>
          <title>Lower-layer performance is not indicative of upper-layer success</title>
          <author initials="L." surname="Pardue">
            <organization/>
          </author>
          <author initials="S." surname="Tellakula">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Aldabbagh2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/2021-09-07-Aldabbagh-Ofcom-presentationt-to-IAB-1v00-1.pdf">
        <front>
          <title>Regulatory perspective on measuring network quality for end-users</title>
          <author initials="A." surname="Aldabbagh">
            <organization/>
          </author>
          <date year="2021" month="September"/>
        </front>
      </reference>

      <reference anchor="Welzl2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/iab-longtermstats_cameraready.docx-1.pdf">
        <front>
          <title>A Case for Long-Term Statistics</title>
          <author initials="M." surname="Welzl">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Fabini2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Fabini-IAB-NetworkQuality.txt">
        <front>
          <title>Network Quality from an End User Perspective</title>
          <author initials="J." surname="Fabini">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Mathis2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Preliminary-Longitudinal-Study-of-Internet-Responsiveness-1.pdf">
        <front>
          <title>Preliminary Longitudinal Study of Internet Responsiveness</title>
          <author initials="M." surname="Mathis">
            <organization/>
          </author>
          <date year="2021" month="August"/>
        </front>
      </reference>

      <reference anchor="Schlinker2019" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Internet-Performance-from-Facebooks-Edge.pdf">
        <front>
          <title>Internet Performance from Facebook's Edge</title>
          <author initials="B." surname="Schlinker">
            <organization/>
          </author>
          <author initials="I." surname="Cunha">
            <organization/>
          </author>
          <author initials="Y." surname="Chiu">
            <organization/>
          </author>
          <author initials="S." surname="Sundaresan">
            <organization/>
          </author>
          <author initials="E." surname="Katz-Basset">
            <organization/>
          </author>
          <date year="2019" month="February"/>
        </front>
      </reference>

      <reference anchor="Foulkes2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/IAB_Metrics_helpful_in_assessing_Internet_Quality.pdf">
        <front>
          <title>Metrics helpful in assessing Internet Quality</title>
          <author initials="J." surname="Foulkes">
            <organization/>
          </author>
          <date year="2021" month="September"/>
        </front>
      </reference>

      <reference anchor="Sivaraman2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/CanopusPositionPaperCameraReady.pdf">
        <front>
          <title>Measuring Network Experience Meaningfully, Accurately, and Scalably</title>
          <author initials="V." surname="Sivaraman">
            <organization/>
          </author>
          <author initials="S." surname="Madanapalli">
            <organization/>
          </author>
          <author initials="H." surname="Kumar">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Reed2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Camera_Ready_-Measuring-ISP-Performance-in-Broadband-America.pdf">
        <front>
          <title>Measuring ISP Performance in Broadband America: A Study of Latency Under Load</title>
          <author initials="D.P." surname="Reed">
            <organization/>
          </author>
          <author initials="L." surname="Perigo">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="MacMillian2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/2021_nqw_lul.pdf">
        <front>
          <title>Beyond Speed Test: Measuring Latency Under Load Across Different Speed Tiers</title>
          <author initials="K." surname="MacMillian">
            <organization/>
          </author>
          <author initials="N." surname="Feamster">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Mirsky2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/IAB-worshop-Error-performance-measurement-in-packet-switched-networks.pdf">
        <front>
          <title>The error performance metric in a packet-switched network</title>
          <author initials="G." surname="Mirsky">
            <organization/>
          </author>
          <author initials="X." surname="Min">
            <organization/>
          </author>
          <author initials="G." surname="Mishra">
            <organization/>
          </author>
          <author initials="L." surname="Han">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Dion2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Nokia-IAB-Measuring-Network-Quality-Improving-and-focusing-on-latency-.pdf">
        <front>
          <title>Focusing on latency, not throughput, to provide a better internet experience and network quality</title>
          <author initials="G." surname="Dion">
            <organization/>
          </author>
	  <author initials="K." surname="De Schepper">
	    <organization/>
	  </author>
	  <author initials="O." surname="Tilmans">
	    <organization/>
	  </author>
          <date year="2021" month="August"/>
        </front>
      </reference>

      <reference anchor="Balasubramanian2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/transportstatsquality.pdf">
        <front>
          <title>Transport Layer Statistics for Network Quality</title>
          <author initials="P." surname="Balasubramanian">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Arkko2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/iab-position-paper-observability.pdf">
        <front>
          <title>Observability is needed to improve network quality</title>
          <author initials="J." surname="Arkko">
            <organization/>
          </author>
          <author initials="M." surname="Kühlewind">
            <organization/>
          </author>
          <date year="2021" month="August"/>
        </front>
      </reference>

      <reference anchor="Marx2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/MergeThoseMetrics_Marx_Jul2021.pdf">
        <front>
          <title>Merge Those Metrics: Towards Holistic (Protocol) Logging</title>
          <author initials="R." surname="Marx">
            <organization/>
          </author>
          <author initials="J." surname="Herbots">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Ghai2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/xfinity-wifi-ietf-iab-v2-1.pdf">
        <front>
          <title>Using TCP Connect Latency for measuring CX and Network Optimization</title>
          <author initials="R." surname="Ghai">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="DeSchepper2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Nokia-IAB-Measuring-Network-Quality-Low-Latency-measurement-workshop-20210802.pdf">
        <front>
          <title>Challenges and opportunities of hardware support for Low Queuing Latency without Packet Loss</title>
          <author initials="K." surname="De Schepper">
            <organization/>
          </author>
          <author initials="O." surname="Tilmans">
            <organization/>
          </author>
          <author initials="G." surname="Dion">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Michel2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/camera_ready_Packet_delivery_time_as_a_tie_breaker_for_assessing_Wi_Fi_access_points.pdf">
        <front>
          <title>Packet delivery time as a tie-breaker for assessing Wi-Fi access points</title>
          <author initials="F." surname="Michel">
            <organization/>
          </author>
          <author initials="O." surname="Bonaventure">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Kerpez2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Wi-Fi-Report-ASSIA.pdf">
        <front>
          <title>Wi-Fi and Broadband Data</title>
          <author initials="J." surname="Shafiei">
            <organization/>
          </author>
          <author initials="K." surname="Kerpez">
            <organization/>
          </author>
          <author initials="J." surname="Cioffi">
            <organization/>
          </author>
          <author initials="P." surname="Chow">
            <organization/>
          </author>
          <author initials="D." surname="Bousaber">
            <organization/>
          </author>
          <date year="2021" month="September"/>
        </front>
      </reference>

      <reference anchor="Liubogoshchev2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Cross-layer-Cooperation-for-Better-Network-Service-2.pdf">
        <front>
          <title>Cross-layer Cooperation for Better Network Service</title>
          <author initials="M." surname="Liubogoshchev">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Laki2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/11/CamRdy-IAB_user_meas_WS_Nadas_et_al_IncentiveBasedTMwQoS.pdf">
        <front>
          <title>Incentive-Based Traffic Management and QoS Measurements</title>
          <author initials="S." surname="Nadas">
            <organization/>
          </author>
          <author initials="B." surname="Varga">
            <organization/>
          </author>
          <author initials="L.M." surname="Contreras">
            <organization/>
          </author>
          <author initials="S." surname="Laki">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Sengupta2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Camera_Ready__Fine-Grained_RTT_Monitoring_Inside_the_Network.pdf">
        <front>
          <title>Fine-Grained RTT Monitoring Inside the Network</title>
          <author initials="S." surname="Sengupta">
            <organization/>
          </author>
          <author initials="H." surname="Kim">
            <organization/>
          </author>
          <author initials="J." surname="Rexford">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <xi:include href="https://datatracker.ietf.org/doc/bibxml3/reference.I-D.morton-ippm-pipe-dream.xml"/>
  
      <reference anchor="Kilkki2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Kilkki-In-Search-of-Lost-QoS.pdf">
        <front>
          <title>In Search of Lost QoS</title>
          <author initials="K." surname="Kilkki">
            <organization/>
          </author>
          <author initials="B." surname="Finley">
            <organization/>
          </author>
          <date year="2021" month="February"/>
        </front>
      </reference>

      <reference anchor="Davies2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/PNSol-et-al-Submission-to-Measuring-Network-Quality-for-End-Users-1.pdf">
        <front>
          <title>Measuring Network Impact on Application Outcomes Using Quality Attenuation</title>
          <author initials="N." surname="Davies">
            <organization/>
          </author>
          <author initials="P." surname="Thompson">
            <organization/>
          </author>
          <date year="2021" month="September"/>
        </front>
      </reference>

      <reference anchor="Zhang2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/User_Perceived_Latency-1.pdf">
        <front>
          <title>User-Perceived Latency to Measure CCAs</title>
          <author initials="M." surname="Zhang">
            <organization/>
          </author>
          <author initials="V." surname="Goel">
            <organization/>
          </author>
          <author initials="L." surname="Xu">
            <organization/>
          </author>
          <date year="2021" month="September"/>
        </front>
      </reference>

  <xi:include href="https://datatracker.ietf.org/doc/bibxml3/reference.I-D.cpaasch-ippm-responsiveness.xml"/>
      
      <reference anchor="Briscoe2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/single-delay-metric-1.pdf">
        <front>
          <title>A Single Common Metric to Characterize Varying Packet Delay</title>
          <author initials="B." surname="Briscoe">
            <organization/>
          </author>
          <author initials="G." surname="White">
            <organization/>
          </author>
          <author initials="V." surname="Goel">
            <organization/>
          </author>
          <author initials="K." surname="De Schepper">
            <organization/>
          </author>
          <date year="2021" month="September"/>
        </front>
      </reference>

      <reference anchor="McIntyre2021" target="https://www.iab.org/wp-content/IAB-uploads/2021/09/Internet-Score-2.pdf">
        <front>
          <title>An end-user approach to an Internet Score</title>
          <author initials="C." surname="Paasch">
            <organization/>
          </author>
          <author initials="K." surname="McIntyre">
            <organization/>
          </author>
          <author initials="O." surname="Shapira">
            <organization/>
          </author>
          <author initials="R." surname="Meyer">
            <organization/>
          </author>
          <author initials="S." surname="Cheshire">
            <organization/>
          </author>
          <date year="2021" month="September"/>
        </front>
      </reference>

      <reference anchor="Speedtest" target="https://www.speedtest.net">
        <front>
          <title>Speedtest</title>
          <author>
            <organization>Ookla</organization>
          </author>
        </front>
      </reference>
      
      <reference anchor="NetworkQuality" target="https://support.apple.com/en-gb/HT212313">
        <front>
          <title>Network Quality</title>
          <author>
            <organization>Apple</organization>
          </author>
        </front>
      </reference>

      <reference anchor="SamKnows" target="https://www.samknows.com/">
        <front>
          <title>SamKnows</title>
          <author>
            <organization/>
          </author>
        </front>
      </reference>
    </references>

  <section anchor="program-committee" numbered="true" toc="default">
        <name>Program Committee</name>
        <t>The program committee consisted of:</t>
	<ul empty="true" spacing="compact">
	  <li><t><contact fullname="Jari Arkko"/></t></li>
	  <li><t><contact fullname="Olivier Bonaventure"/></t></li>
	  <li><t><contact fullname="Vint Cerf"/></t></li>
	  <li><t><contact fullname="Stuart Cheshire"/></t></li>
	  <li><t><contact fullname="Sam Crowford"/></t></li>
	  <li><t><contact fullname="Nick Feamster"/></t></li>
	  <li><t><contact fullname="Jim Gettys"/></t></li>
	  <li><t><contact fullname="Toke Hoiland-Jorgensen"/></t></li>
	  <li><t><contact fullname="Geoff Huston"/></t></li>
	  <li><t><contact fullname="Cullen Jennings"/></t></li>
	  <li><t><contact fullname="Katarzyna Kosek-Szott"/></t></li>
	  <li><t><contact fullname="Mirja Kühlewind"/></t></li>
	  <li><t><contact fullname="Jason Livingood"/></t></li>
	  <li><t><contact fullname="Matt Mathis"/></t></li>
	  <li><t><contact fullname="Randall Meyer"/></t></li>
	  <li><t><contact fullname="Kathleen Nichols"/></t></li>
	  <li><t><contact fullname="Christoph Paasch"/></t></li>
	  <li><t><contact fullname="Tommy Pauly"/></t></li>
	  <li><t><contact fullname="Greg White"/></t></li>
	  <li><t><contact fullname="Keith Winstein"/></t></li>
	</ul>
      </section>

        <section anchor="workshop-chairs" numbered="true" toc="default">
        <name>Workshop Chairs</name>
        <t>The workshop chairs consisted of:</t>
	<ul empty="true" spacing="compact">
	  <li><t><contact fullname="Wes Hardaker"/></t></li>
	  <li><t><contact fullname="Evgeny Khorov"/></t></li>
	  <li><t><contact fullname="Omer Shapira"/></t></li>
	</ul>
      </section>
    
    <section anchor="participants-list" numbered="true" toc="default">
      <name>Workshop Participants</name>
      <t>The following is a list of participants who attended the workshop over a remote connection:</t>
      <ul empty="true" spacing="compact">
	<li><t><contact fullname="Ahmed Aldabbagh"/></t></li>
	<li><t><contact fullname="Jari Arkko"/></t></li>
	<li><t><contact fullname="Praveen Balasubramanian"/></t></li>
	<li><t><contact fullname="Olivier Bonaventure"/></t></li>
	<li><t><contact fullname="Djamel Bousaber"/></t></li>
	<li><t><contact fullname="Bob Briscoe"/></t></li>
	<li><t><contact fullname="Rich Brown"/></t></li>
	<li><t><contact fullname="Anna Brunstrom"/></t></li>
	<li><t><contact fullname="Pedro Casas"/></t></li>
	<li><t><contact fullname="Vint Cerf"/></t></li>
	<li><t><contact fullname="Stuart Cheshire"/></t></li>
	<li><t><contact fullname="Kenjiro Cho"/></t></li>
	<li><t><contact fullname="Steve Christianson"/></t></li>
	<li><t><contact fullname="John Cioffi"/></t></li>
	<li><t><contact fullname="Alexander Clemm"/></t></li>
	<li><t><contact fullname="Luis M. Contreras"/></t></li>
	<li><t><contact fullname="Sam Crawford"/></t></li>
	<li><t><contact fullname="Neil Davies"/></t></li>
	<li><t><contact fullname="Gino Dion"/></t></li>
	<li><t><contact fullname="Toerless Eckert"/></t></li>
	<li><t><contact fullname="Lars Eggert"/></t></li>
	<li><t><contact fullname="Joachim Fabini"/></t></li>
	<li><t><contact fullname="Gorry Fairhurst"/></t></li>
	<li><t><contact fullname="Nick Feamster"/></t></li>
	<li><t><contact fullname="Mat Ford"/></t></li>
	<li><t><contact fullname="Jonathan Foulkes"/></t></li>
	<li><t><contact fullname="Jim Gettys"/></t></li>
	<li><t><contact fullname="Rajat Ghai"/></t></li>
	<li><t><contact fullname="Vidhi Goel"/></t></li>
	<li><t><contact fullname="Wes Hardaker"/></t></li>
	<li><t><contact fullname="Joris Herbots"/></t></li>
	<li><t><contact fullname="Geoff Huston"/></t></li>
	<li><t><contact fullname="Toke Høiland-Jørgensen"/></t></li>
	<li><t><contact fullname="Jana Iyengar"/></t></li>
	<li><t><contact fullname="Cullen Jennings"/></t></li>
	<li><t><contact fullname="Ken Kerpez"/></t></li>
	<li><t><contact fullname="Evgeny Khorov"/></t></li>
	<li><t><contact fullname="Kalevi Kilkki"/></t></li>
	<li><t><contact fullname="Joon Kim"/></t></li>
	<li><t><contact fullname="Zhenbin Li"/></t></li>
	<li><t><contact fullname="Mikhail Liubogoshchev"/></t></li>
	<li><t><contact fullname="Jason Livingood"/></t></li>
	<li><t><contact fullname="Kyle MacMillan"/></t></li>
	<li><t><contact fullname="Sharat Madanapalli"/></t></li>
	<li><t><contact fullname="Vesna Manojlovic"/></t></li>
	<li><t><contact fullname="Robin Marx"/></t></li>
	<li><t><contact fullname="Matt Mathis"/></t></li>
	<li><t><contact fullname="Jared Mauch"/></t></li>
	<li><t><contact fullname="Kristen McIntyre"/></t></li>
	<li><t><contact fullname="Randall Meyer"/></t></li>
	<li><t><contact fullname="François Michel"/></t></li>
	<li><t><contact fullname="Greg Mirsky"/></t></li>
	<li><t><contact fullname="Cindy Morgan"/></t></li>
	<li><t><contact fullname="Al Morton"/></t></li>
	<li><t><contact fullname="Szilveszter Nadas"/></t></li>
	<li><t><contact fullname="Kathleen Nichols"/></t></li>
	<li><t><contact fullname="Lai Yi Ohlsen"/></t></li>
	<li><t><contact fullname="Christoph Paasch"/></t></li>
	<li><t><contact fullname="Lucas Pardue"/></t></li>
	<li><t><contact fullname="Tommy Pauly"/></t></li>
	<li><t><contact fullname="Levi Perigo"/></t></li>
	<li><t><contact fullname="David Reed"/></t></li>
	<li><t><contact fullname="Alvaro Retana"/></t></li>
	<li><t><contact fullname="Roberto"/></t></li>
	<li><t><contact fullname="Koen De Schepper"/></t></li>
	<li><t><contact fullname="David Schinazi"/></t></li>
	<li><t><contact fullname="Brandon Schlinker"/></t></li>
	<li><t><contact fullname="Eve Schooler"/></t></li>
	<li><t><contact fullname="Satadal Sengupta"/></t></li>
	<li><t><contact fullname="Jinous Shafiei"/></t></li>
	<li><t><contact fullname="Shapelez"/></t></li>
	<li><t><contact fullname="Omer Shapira"/></t></li>
	<li><t><contact fullname="Dan Siemon"/></t></li>
	<li><t><contact fullname="Vijay Sivaraman"/></t></li>
	<li><t><contact fullname="Karthik Sundaresan"/></t></li>
	<li><t><contact fullname="Dave Taht"/></t></li>
	<li><t><contact fullname="Rick Taylor"/></t></li>
	<li><t><contact fullname="Bjørn Ivar Teigen"/></t></li>
	<li><t><contact fullname="Nicolas Tessares"/></t></li>
	<li><t><contact fullname="Peter Thompson"/></t></li>
	<li><t><contact fullname="Balazs Varga"/></t></li>
	<li><t><contact fullname="Bren Tully Walsh"/></t></li>
	<li><t><contact fullname="Michael Welzl"/></t></li>
	<li><t><contact fullname="Greg White"/></t></li>
	<li><t><contact fullname="Russ White"/></t></li>
	<li><t><contact fullname="Keith Winstein"/></t></li>
	<li><t><contact fullname="Lisong Xu"/></t></li>
	<li><t><contact fullname="Jiankang Yao"/></t></li>
	<li><t><contact fullname="Gavin Young"/></t></li>
	<li><t><contact fullname="Mingrui Zhang"/></t></li>
      </ul>
    </section>

    <section anchor="iab-members-at-the-time-of-approval" numbered="false" toc="default">
      <name>IAB Members at the Time of Approval</name>
      <t>Internet Architecture Board members at the time this document was
      approved for publication were:</t>
      <ul empty="true" spacing="compact">
	<li><t><contact fullname="Jari Arkko"/></t></li>
	<li><t><contact fullname="Deborah Brungard"/></t></li>
	<li><t><contact fullname="Lars Eggert"/></t></li>
	<li><t><contact fullname="Wes Hardaker"/></t></li>
	<li><t><contact fullname="Cullen Jennings"/></t></li>
	<li><t><contact fullname="Mallory Knodel"/></t></li>
	<li><t><contact fullname="Mirja Kühlewind"/></t></li>
	<li><t><contact fullname="Zhenbin Li"/></t></li>
	<li><t><contact fullname="Tommy Pauly"/></t></li>
	<li><t><contact fullname="David Schinazi"/></t></li>
	<li><t><contact fullname="Russ White"/></t></li>
	<li><t><contact fullname="Qin Wu"/></t></li>
	<li><t><contact fullname="Jiankang Yao"/></t></li>
      </ul>
    </section>
    <section anchor="acknowledgments" numbered="false" toc="default">
      <name>Acknowledgments</name>
      <t>The authors would like to thank the workshop participants, the members
of the IAB, and the program committee for creating and participating
      in many interesting discussions.</t>
    </section>
      <section anchor="draft-contributors" numbered="false" toc="default">
        <name>Contributors</name>
        <t>Thank you to the people that contributed edits to this document:</t>
	<ul empty="true" spacing="compact">
	  <li><t><contact fullname="Erik Auerswald"/></t></li>
	  <li><t><contact fullname="Simon Leinen"/></t></li>
	  <li><t><contact fullname="Brian Trammell"/></t></li>
	</ul>
      </section>
  </back>
</rfc>