Skip to content

Commit

Permalink
strip theory out of README
Browse files Browse the repository at this point in the history
  • Loading branch information
joelb123 committed Jan 31, 2024
1 parent 2f44f81 commit f10c9a5
Showing 1 changed file with 6 additions and 101 deletions.
107 changes: 6 additions & 101 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,102 +199,6 @@ how the fish are biting. Then you back off the number
of lines to the number that you and your crew can handle
at a time that day.

### Quantifying rates and times

The downloading process lives simultaneously in two spaces,
rates and times. In _rate space_, the observables are the
overall bandwidth (in units of Mbit/s) of all downloads
(obtainable from network interface statistics, which may
include other traffic) and the overall queue depths at launch
and at completion. In _time space_ the observables are the
download times and sizes of individual files and the
queue depths at launch and completion of the individual
server used for the transfer.

Packets are always transmitted at line speed of the specific
connection. The effective rate is determined by the slowest
connection, typically the downstream WAN connection. Starting
transfers to a particular server requires a certain amount of
2-way exchanges for handshaking. There will also be latencies
for acknowledging receipt of packets received, depending on
the transfer protocol in use. The first transfer from a
particular server occupies only a few percent of the effective
bandwith at best. Packets from each subsequent concurrent
transfer from a server either fit into the latency periods
of other transfers or have to wait for packets to be transferred.
The total bit rate thus increases nearly linearly versus
the number of transfers for the first few concurrent transfers,
then falls off linear as more transfers interfere with each
other, then decays exponentially to zero whenever the line is
saturated with requests. The sweet spot is on the
expoentially-decaying section at the queue depth where
the decay is largest. (This may be shown analytically for
exponenential decay where the step size in queue depth
is small enough to be considered continuous.)

Ignoring the effects of finite packet size and treating the
networking components shared among each connection as the main
limitation to transfer rates, we can write the _Equation of Time_
for the time required to receive file $i$ from server $j$ as
approximately given by

$`
\begin{equation}
t_{i} = F_i - I_i \approx L_j +
(c_{\rm ack} L_j + 1 /B_{\rm eff}) S_i +
H_{ij}(i, D_j, D_{{\rm crit}_j})
\end{equation}
`$

where

- $F_i$ is the finish time of the transfer,
- $I_i$ is the initial time of the transfer,
- $L_j$ is the server-dependent service latency, more-or-less
the same as the value one gets from the _ping_ command,
- $c_{\rm ack}$ is a value reflecting the number of service latencies
required and the number of bytes transferred per acknowledgement,
and since it is nearly constant given the HTTP and network protocols it
is the part of the slope expression that is fit and not measured,
- $B_{\rm eff}$ is the effective download bit rate of your WAN connection,
which can be measured through network interface statistics if the
transfer is long enough to reach saturation,
- $S_i$ is the size of file $i$,
- $H_{ij}$ is the file- and server-dependent
[Head-Of-Line Latency](https://en.wikipedia.org/wiki/Head-of-line_blocking)
that reflects waiting to get an active slot when the
server queue depth $D_j$ is above some server-dependent
critical value $D_{{\rm crit}_j}$.

If your downloading process is the only one accessing the server,
the Head-Of-Line latency can be quantified via the relation

$`
\begin{equation}
H_{ij} =
\left\{ \begin{array}{ll}
0, & D_j < D_{{\rm crit}_j} \cr
I_i - F_{i^{\prime}-D_j+D_{{\rm crit}_j}-1},
& D_j \ge D_{{\rm crit}_j} \cr
\end{array} \right.
\end{equation}
`$

where the prime in the subscript represents a re-indexing of
entries in order of end times rather than start times. If
other users are accessing the server at the same time, this
expression becomes indeterminate, but the important thing
to note is that it has an expectation value of a multiple
of the modal file service time, $n\tilde{t}_{\rm file}$

At queue depths small enough that no time is spent waiting to
get to the head of the line, the file transfer time is linear
in file size with the intercept given by the service
latency $L_j$ and slope governed by an expression whose only
unknown is a near-constant related to acknowledgements. As queue
depth increases, transfer times are dominated by $H_{ij}$, the
time spent waiting to get to the head of the queue.

### Adaptilastic Queueing

_Flardl_ implements a method I call "adaptilastic"
Expand Down Expand Up @@ -353,19 +257,20 @@ where

After waiting an exponentially-distributed stochastic period
given by the applicable value for $k_j$, testing is done
against three queue depth limits:
against four limits calculated by the methods in [theory]:

- $D_{{\rm max}_j}$ the maximum per-server queue depth
which is an input parameter, revised downward if any
queue requests are rejected (default 100),
- $D_{\rm sat}$ the total queue depth at which the download
bit rate saturates or exceeds the maximum bit rate,
- $D_{{\rm crit}_j}$ the critical per-server queue depth,
calculated each session when updated information is available.
calculated each session when updated information is available,
- $B_{\rm max}$ the maximum bandwidth allowed.

If any of the three queue depth limits is exceeded, a second
stochastic wait period at the inverse of the current per-server
rate $k_j$ is added.
If any of the limits are exceeded, a stochastic wait period
at the inverse of the current per-server rate $k_j$ is added
until the limit is no longer exceeded.

After enough files have come back from a server or set of
servers (a configurable parameter $N_{\rm min}$), _flardl_
Expand Down

0 comments on commit f10c9a5

Please sign in to comment.