docs updated

README.md

@@ -96,33 +96,35 @@ estimate and fits to log-normal and normal distributions.
 
 Queueing algorithms that rely upon per-file rates as the
 pricipal control mechanism implicitly assume that queue
-statistic can be approximated with a normal distribution.
+statistics can be approximated with a normal distribution.
 In making that assumption, they largely ignore the effects
-of big files on overall download statistics. Such software
-inevitably encounters big problems because **mean values
+of big files on overall download statistics. Such algorithms
+inevitably encounter problems because **mean values
 are neither stable nor characteristic of the distribution**.
-As can be seen in the fits above, the mean and standard
-distribution of samples drawn from a
-long-tail distribution tend to grow with increasing sample
-size. In the example shown in the figure above, a fit of
+For example, as can be seen in the fits above, the mean and
+standard distribution of samples drawn from a long-tail
+distribution tend to grow with increasing sample size.
+In the example shown in the figure above, a fit of
 a normal distribution to a sample of 5% of the data (dashed
 line) gives a markedly-lower mean and standard deviation than
 the fit to all points (dotted line) and both fits are poor.
 The reason why the mean tend to grow larger with more files is
 because the more files sampled, the higher the likelihood that
-one of them will be huge. Algorithms that employ average
-per-file rates or times as the primary means of control will
-launch requests too slowly most of the time while letting
-queues run too deep when big downloads are encountered.
-While the _mean_ per-file download time isn't a good statistic,
-the _modal_ per-file download time $\tilde{t}_{\rm file}$ is better,
+one of them will be huge enough to dominate the average values.
+Algorithms that employ average per-file rates or times as the
+primary means of control will launch requests too slowly most
+of the time while letting queues run too deep when big downloads
+are encountered. While the _mean_ per-file download time isn't a
+good statistic, **_modal_ per-file file statistics will be
+consistent** (e.g., the modal per-file download time
+$\tilde{t}_{\rm file}$, where the tilde indicates a modal value),
 at least on timescales over which network and server performance
-are consistent. You are not guaranteed to be the only user of
-either your WAN connection nor of the server, and sharing those
-resources impact download statistics in different ways, especially
-if multiple servers are involved.
+are consistent. You are not guaranteed to be the only user of either
+your WAN connection nor of the server, and sharing those resources
+impact download statistics in different ways, especially if multiple
+servers are involved.
 
-Ignore the effects of finite packet size and treating the
+Ignoring the effects of finite packet size and treating the
 networking components shared among each connection as the main
 limitation to transfer rates, we can write the _Equation of Time_
 for the time required to receive file $i$ from server $j$ as
@@ -144,12 +146,12 @@ where
   the same as the value one gets from the _ping_ command,
 - $c_{\rm ack}$ is a value reflecting the number of service latencies
   required and the number of bytes transferred per acknowledgement,
-  and while nearly constant given the HTTP and network protocols it
+  and since it is nearly constant given the HTTP and network protocols it
   is the part of the slope expression that is fit and not measured,
-- $S_i$ is the size of file $i$,
-- $B_{\rm lim}$ is the limiting download bit rate across all servers,
+- $B_{\rm eff}$ is the effective download bit rate of your WAN connection,
   which can be measured through network interface statistics if the
   transfer is long enough to reach saturation,
+- - $S_i$ is the size of file $i$,
 - $H_{ij}$ is the file- and server-dependent
   [Head-Of-Line Latency](https://en.wikipedia.org/wiki/Head-of-line_blocking)
   that reflects waiting to get an active slot when the
@@ -173,30 +175,32 @@ $`
 where the prime in the subscript represents a re-indexing of
 entries in order of end times rather than start times. If
 other users are accessing the server at the same time, this
-expression becomes indeterminate, but with an expectation
-value of a multiple of the most-common file service time.
+expression becomes indeterminate, but the important thing
+to note is that it has an expectation value of a multiple
+of the modal file service time, $n\tilde{t}_{\rm file}$
 
 At queue depths small enough that no time is spent waiting to
 get to the head of the line, the file transfer time is linear
 in file size with the intercept given by the service
-latency and slope governed by an expression whose only unknown is
-a near-constant related to acknowledgements. As queue depth
-increases, transfer times are dominated by time spent waiting
-to get to the head of the queue.
+latency $L_j$ and slope governed by an expression whose only
+unknown is a near-constant related to acknowledgements. As queue
+depth increases, transfer times are dominated by $H_{ij}$, the
+time spent waiting to get to the head of the queue.
 
 ### Queue Depth is Toxic
 
 At first glance, running at high queue depths seems attractive.
-One of the simplest queueing algorithm would to simply put every
+One of the simplest queueing algorithms would to simply put every
 job in a queue at startup and let the server(s) handle requests
-in parallel and serial as best they can. But such non-adaptive
-non-elastic algorithms give poor real-world performance or multiple
-reasons. First, if there is more than one server queue, differing
-file sizes and tramsfer rates will result in the queueing equivalent
-of
+in parallel up to their individual critical queue depths, then
+serial as best they can. But such non-adaptive non-elastic algorithms
+give poor real-world performance or multiple reasons. First, if
+there is more than one server queue, differing file sizes and
+tramsfer rates will result in the queueing equivalent of
 [Amdahl's law](https://en.wikipedia.org/wiki/Amdahl%27s_law),
 an "overhang" where one server still has many files queued up to
-serve while others have completed all requests.
+serve while others have completed all requests. The server with
+the overhang is not guaranteed to be the fastest server, either.
 
 Moreover, if a server decides you are abusing its queue policies,
 it may take action that hurts your current and future downloads.
@@ -207,18 +211,19 @@ time is the main hallmark of a DOS attack. The minimal response
 to a DOS event causes the server to dump your latest requests,
 a minor nuisance. Worse is if the server responds by severely
 throttling further requests from your IP address for hours
-or sometime days. Worst of all, your IP address can get the "death
-penalty" and be put on a permanent blacklist that may require manual
-intervention for removal. Blacklisting might not even be your personal
-fault, but a collective problem. I have seen a practical class of 20
-students brought to a complete halt by a server's 24-hour blacklisting
-of the institution's public IP address. Until methods are developed
-for servers to publish their "play-friendly" values and whitelist
-known-friendly servers, the highest priority for downloading
+or sometime days. But in the worst case, your IP address can get the
+"death penalty" of being put on a permanent blacklist for throttling
+or blockage that may require manual intervention by someone in control
+of the server to get removed from. Blacklisting might not even be your
+personal fault, but a collective problem. I have seen a practical class
+of 20 students brought to a complete halt by a 24-hour blacklisting
+of the institution's public IP address by a government site. Until methods
+are developed for servers to publish their "play-friendly" values and
+whitelist known-friendly clients, the highest priority for downloading
 algorithms must be to **avoid blacklisting by a server by minimizing
-queue depth**. However, the absolute minimum queue depth is
+queue depth**. At the other extreme, the absolute minimum queue depth is
 retreating back to synchronous downloading. How can we balance
-the competing demands of speed and avoiding blacklisting?
+the competing demands of speed while avoiding blacklisting?
 
 ### A Fishing Metaphor
 
@@ -240,18 +245,18 @@ you can set out as many lines as you want (up to some limit)
 and fish in parallel. At first, you catch small bony fishes
 that are the ocean-going equivalent of crappies. But
 eventually you hook a small shark. Not does it take a lot of
-your time and attention to reel in a shark, but a landing
+your time and attention to reel in the shark, but landing
 a single shark totally skews the average weight of your catch.
-If you catch a small shark, then if you fish for long enough
-you will probably catch a big shark. Maybe you might even
-hook a small whale. But you and your crew can only effecively
-reel in so many hooked lines at once. Putting out more lines than
-that effective limit of hooked plus waiting-to-be-hooked
+If you fish in the ocean for long enough you will probably catch
+a big shark that weighs hundreds of times more than crappies.
+Maybe you might even hook a small whale. But you and your crew
+can only effecively reel in so many hooked lines at once. Putting
+out more lines than that effective limit of hooked plus waiting-to-be-hooked
 lines only results in fishes waiting on the line, when they
 may break the line or get partly eaten before you can reel
 them in. Our theory of fishing says to put out lines
 at a high-side estimate of the most probable rate of catching
-fish until you reach the maximum number of lines the boat
+modal-sized fishes until you reach the maximum number of lines the boat
 allows or until you catch enough fish to be able to estimate
 how the fish are biting. Then you back off the number
 of lines to the number that you and your crew can handle
@@ -311,42 +316,38 @@ where
 - $I_{\rm first}$ is the initiation time for the first
   transfer to arrive,
 - and $\tilde{\tau_j}$ is the modal file transfer rate
-  for the current session,
+  for the current session.
 
 After waiting an exponentially-distributed stochastic period
 given by the applicable value for $k_j$, testing is done
-against three queue depth limits
+against three queue depth limits:
 
-- $D_{{\rm max}_j}$, the maximum per-server queue depth
-  which is an input parameter (by default 100) that is
-  updated if any queue requests are rejected.
-- $D_{\rm sat}$, the total queue depth at which the download
-  bit rate saturates or exceeds the maximum bit rate.
-- $D_{{\rm crit}_j}$, the critical per-server queue depth.
-  calculated each session when updated information
+- $D_{{\rm max}_j}$ the maximum per-server queue depth
+  which is an input parameter, revised downward if any
+  queue requests are rejected (default 100),
+- $D_{\rm sat}$ the total queue depth at which the download
+  bit rate saturates or exceeds the maximum bit rate,
+- $D_{{\rm crit}_j}$ the critical per-server queue depth,
+  calculated each session when updated information is available.
 
 If any of the three queue depth limits is exceeded, a second
-stochastic wait period is added.
-
-At low queue
-depths, one can fit to this expression to estimate the server
-latency $L_j$, the limiting download bit rate at saturation
-$B_{\rm lim}$, and the total queue depth at saturation
-$D_{\rm sat}$. If the server queue depth $D_j$ is run up high
-enough during the initial latency period before files are returned,
-one can estimate the critical queue depth $D_{{\rm crit}_j}$ by
-noting where the deviations approach a multiple of the modal
-transfer time. This estimate of critical queue depth reflects
-both server policy and overall server load at time of request.
-
-As transfers are completed, _flardl_ estimates the queue
-depth at which saturation was achieved (totalled over all
-servers), and updates its estimate of $B_{\rm eff}$ over
-all servers at saturation from the network interface
-statistics and the critical queue depth and modal
-per-file return rate on a per-server basis. These values
-form the bases for launching the remaining requests. The servers
-with higher modal service rates (i.e., rates of serving
+stochastic wait period at the inverse of the current per-server
+rate $k_j$ is added.
+
+After enough files have come back from a server or set of
+servers (a configurable parameter $N_{\rm min}$), _flardl_
+fits the curve of observed network bandwidth versus queue
+depth to obtain the effective download bit rate at saturation
+$B_{\rm eff} and the total queue depth at saturation
+$D*{\rm sat}$. Then, per-server, \_flardl* fits the curves
+of service times versus file sized to the Equation of Time
+to estimate server latencies $L_j$ and if the server queue
+depth $D_j$ is run up high enough the critical queue depths
+$D_{{\rm crit}_j}$. This estimates reflects local
+network conditions, server policy, and overall server
+load at time of request, so they are both adaptive and elastic.
+These values form the bases for launching the remaining requests .
+Servers with higher modal service rates (i.e., rates of serving
 crappies) will spend less time waiting and thus stand a better
 chance at nabbing an open queue slot, without penalizing servers
 that happen to draw a big downloads (whales).
@@ -356,13 +357,20 @@ that happen to draw a big downloads (whales).
 The adapilastic algorithm assumes that file sizes are randomly-ordered
 in the list. But what if we know the file sizes beforehand? The server
 that draws the biggest file is most likely to finish last, so it's
-important for that file to be started on the fastest server as soon
-as one has current information about which server is indeed
-fastest (i.e., by reaching the _arriving_ state). One simple way
-of making downloads more optimal when file sizes are known is to sort
-the incoming list of download by size, downloading in order of ascending
-size until the server receives the first file, then switch to downloading
-in order of descending size.
+important for that file to be started on the lowest-latency server as
+soon as one has current information about which server is indeed
+fastest (i.e., by reaching the _arriving_ state). The way that _flardl_
+optimizes downloads when provided a dictionary of relative file sizes
+is to sort the incoming list of downloads into two lists. The first
+list is sorted into tiers of files with the size of the tier equal to
+the number of servers $N_{\rm serv}$ in use. The 0th tier is the
+smallest $N_{\rm serv}$ files, the next tier is the largest $N_{\rn serv}$
+files below a cutoff file size (default of 2x the modal file size), and
+alternating between the smallest and biggest crappies out to $N_{\min}$
+files. The second list is all other files, sorted in order of descending
+file size. _Flardl_ waits until it enters the _updated_ state, with all
+files from the first list returned before drawing from the second list
+so that the fastest server will get the job of sending the biggest file.
 
 ## Requirements
 
@@ -388,8 +396,7 @@ files. See test examples for usage.
 
 ## Contributing
 
-Contributions are very welcome.
-To learn more, see the [Contributor Guide].
+Contributions are very welcome. To learn more, see the [Contributor Guide].
 
 ## License
 
@@ -398,8 +405,8 @@ _flardl_ is free and open source software.
 
 ## Issues
 
-If you encounter any problems,
-please [file an issue] along with a detailed description.
+If you encounter any problems, please [file an issue] along with a
+detailed description.
 
 ## Credits