Merge pull request #1059 from CesiumGS/doc-tile-selection-algorithm

Tile selection algorithm documentation, round 2

Cesium3DTilesSelection/include/Cesium3DTilesSelection/Tileset.h

@@ -496,14 +496,14 @@ class CESIUM3DTILESSELECTION_API Tileset final {
 
     /**
      * @brief Medium priority tiles that are needed to render the current view
-     * the appropriate level-of-detail.
+     * at the appropriate level-of-detail.
      */
     Normal = 1,
 
     /**
-     * @brief High priority tiles that are causing extra detail to be rendered
-     * in the scene, potentially creating a performance problem and aliasing
-     * artifacts.
+     * @brief High priority tiles whose absence is causing extra detail to be
+     * rendered in the scene, potentially creating a performance problem and
+     * aliasing artifacts.
      */
     Urgent = 2
   };

doc/diagrams/tile-load-states.mmd

@@ -0,0 +1,18 @@
+stateDiagram-v2
+  unloaded : Unloaded
+  contentLoading : Content Loading
+  contentLoaded : Content Loaded
+  done : Done
+  failed : Failed
+  failedTemp : Failed Temporarily
+  unloading : Unloading
+  [*] --> unloaded
+  unloaded --> contentLoading: Start Async Loading
+  contentLoading --> contentLoaded: Done Async Loading
+  contentLoading --> failed: Load Failed
+  contentLoading --> failedTemp: Retry Later
+  failedTemp --> contentLoading: Start Async Loading
+  contentLoaded --> done: Main Thread Loading
+  contentLoaded --> unloaded: Unload Tile Content
+  done --> unloading: Start Unloading
+  unloading --> unloaded: Done Unloading

doc/topics/developer.md

@@ -1,6 +1,13 @@
 # Developer Resources {#developer-resources}
 
+## Getting Started
+
 * \subpage developer-setup
-* \subpage multithreading
 * \subpage style-guide
-* \subpage contributing
+* \subpage contributing
+
+## Architecture of Cesium Native
+
+* \subpage multithreading
+* \subpage selection-algorithm-details
+* \subpage rendering-3d-tiles

doc/topics/selection-algorithm-details.md

@@ -109,7 +109,7 @@ When these two conditions are met for a tile's descendants, then we should not r
 We decide whether a _Kick_ is necessary by first traversing the tile's children normally, as in a _REFINE_. Calling `_visitTileIfNeeded` on each child tile returns a `TraversalDetails` instance with three fields:
 
 * `allAreRenderable`: `true` if every selected tile is renderable. `false` if even one selected tile is not yet renderable.
-* `anyWereRenderedLastFrame`: `true` if any selected tile was rendered last frame. `false` is none of the selected tiles were rendered last frame.
+* `anyWereRenderedLastFrame`: `true` if any selected tile was rendered last frame. `false` if none of the selected tiles were rendered last frame.
 * `notYetRenderableCount`: The number of selected tiles for which `isRenderable` is false.
 
 The `TraversalDetails` for all children are combined in the intuitive way: `allAreRenderable` values are combined with a boolean AND, `anyWereRenderedLastFrame` values are combined with a boolean OR, and `notYetRenderableCount` values are combined by summing.
@@ -139,13 +139,118 @@ The `loadingDescendantLimit` works like a heuristic for deciding when intermedia
 
 Once the current tile finishes loading and is rendered, only then will the tiles deeper in the subtree be given an opportunity to load and render. This ensures that the user sees the model sooner, at the cost of loading more tiles. The idea is to strike a tunable balance between user feedback and loading efficiency.
 
+## Tile Content Loading
+
+It is important to understand the distinction between a [Tile](@ref Cesium3DTilesSelection::Tile) and its _content_. In the 3D Tiles specification, a [Tile](https://github.com/CesiumGS/3d-tiles/blob/main/specification/PropertiesReference_3dtiles.adoc#tile) is a node in the tileset's bounding-volume hierarchy (BVH), and has a bounding volume, a transform, a geometric error value, and more. Tiles are arranged in a tree, so every `Tile` has a parent `Tile` (except the root) and zero or more children. Tiles also have a `content.uri` property which points to the externally-stored _content_ for the tile. This external content is usually some sort of renderable object, such as a glTF model or a legacy container format like [b3dm](https://github.com/CesiumGS/3d-tiles/blob/main/specification/TileFormats/Batched3DModel/README.adoc#tileformats-batched3dmodel-batched-3d-model). It can also be a further subtree of the BVH rooted at this node, which is known as an [external tileset](https://github.com/CesiumGS/3d-tiles/tree/main/specification#core-external-tilesets).
+
+Because _content_ is by far the largest part of a tile, and the most time-consuming to load, Cesium Native aims to only load it when when it is needed. The loading happens through a small state machine maintained in the `_loadState` property of each tile. The possible load states are captured in the [TileLoadState](@ref Cesium3DTilesSelection::TileLoadState) enumeration.
+
+@mermaid{tile-load-states}
+
+A `Tile` starts in the `Unloaded` state. A `Tile` in this state is added to the `_workerThreadLoadQueue` the first time it is visited during the selection algorithm. Any tiles that remain in this queue at the end of the selection (that is, that are not [kicked](#kicking)) are prioritized for loading as described in the [next section](#load-prioritization).
+
+The number of tiles that may load simultaneously is controlled by the [TilesetOptions::maximumSimultaneousTileLoads](\ref Cesium3DTilesSelection::TilesetOptions::maximumSimultaneousTileLoads) property. We can picture tile loading as a swimming pool with `maximumSimultaneousTileLoads` swim lanes. The highest priority tiles jump in the pool, each in their own lane, and race for the other end. When a tile reaches the other side (finishes asynchronous loading), the next highest priority tile can jump in the pool in that now-unoccupied lane and start swimming. Multiple tiles can never share a swim lane.
+
+The swim across the pool includes all of the steps of the tile loading process that do not need to happen on the main thread, including:
+
+* Initiating an HTTP request for the tile content and receiving the response.
+* Parsing the content from the response, such as parsing the glTF or external tileset JSON.
+* Decoding compressed geometry and textures.
+* Generating extra data needed for rendering, such as normals and raster overlay texture coordinates.
+
+When a tile jumps in the pool, its `TileLoadState` is changed to `ContentLoading`. When it reaches the other end, the state is changed to `ContentLoaded`.
+
+The next time the selection algorithm runs, and it sees a tile in the `ContentLoaded` state, the tile is added to the `_mainThreadLoadQueue`. Just like with the `_workerThreadLoadQueue`, tiles may be kicked from this queue as the selection algorithm proceeds, and those that remain are ranked by priority. This time, though, the loading happens synchronously, on the same thread that is running the selection algorithm. To avoid monopolizing too much main thread time, which could have a severe impact on interactivity, the highest priority tiles from this queue are processed only until the [TilesetOptions::mainThreadLoadingTimeLimit](\ref Cesium3DTilesSelection::TilesetOptions::mainThreadLoadingTimeLimit) has elapsed. Additional tiles will need to wait until the next frame.
+
+The primary task that is completed during main thread loading is to call [IPrepareRendererResources::prepareInMainThread](\ref Cesium3DTilesSelection::IPrepareRendererResources::prepareInMainThread). See [Rendering 3D Tiles](#rendering-3d-tiles) for details.
+
+Once a tile has completed its main-thread loading, it enters the `Done` state and is ready to be rendered.
+
+If something goes wrong during the `ContentLoading` state, such as an HTTP error because the tile's content URL is invalid or the server is down, the tile enters the `Failed` state. Failed tiles will usualy show up as missing data or "holes" in the model. Content loads can also fail temporarily, such as when an access token expires and needs to be refreshed. Such tiles will transition to the `FailedTemporarily` state, and the async loading process will be restarted the next time the tile is needed.
+
+Finally, there is the `Unloading` state. When a tile is no longer needed, it usually transitions directly and synchronously from the `ContentLoaded` or `Done` state to the `Unloaded` state. However, if the tile being unloaded happens at the same time to be the source of an active raster overlay "upsampling" operation, then unloading its content would lead to undefined behavior. Instead, we only unload its renderer resources (by calling [IPrepareRendererResources::free](\ref Cesium3DTilesSelection::IPrepareRendererResources::free)) but do _not_ delete its content. We put it in the `Unloading` state to mark that this has been done, and transition it to the `Unloaded` state when it is safe to do so. For more details, see the [pull request](https://github.com/CesiumGS/cesium-native/pull/554) where this mechanism was added.
+
+## Load Prioritization {#load-prioritization}
+
+Tiles that need to be loaded are prioritized so that the most important tiles are loaded first. Load priority consists of a priority _group_ plus a priority _value_ within that group. The group is chosen according to the reason that this tile is needed by the selection algorithm. The groups are as follows:
+
+* `Preload` - Low priority tiles that aren't needed right now, but are being preloaded for the future.
+* `Normal` - Medium priority tiles that are needed to render the current view at the appropriate level-of-detail.
+* `Urgent` - High priority tiles whose absence is causing extra detail to be rendered in the scene, potentially creating a performance problem and aliasing artifacts.
+
+Within the group, a tile with a lower priority value will be loaded before a tile with a higher priority value. The priority value is computed as follows:
+
+```
+(1.0 - dot(tileDirection, cameraDirection)) * distance
+```
+
+Where `distance` is the distance from the camera to the closest point on the tile, `tileDirection` is the unit vector from the camera to the center of the tile's bounding volume, and `cameraDirection` is the look direction of of the camera. The idea is that tiles that are near the center of the screen and closer to the camera are loaded first.
+
+## Forbid Holes {#forbid-holes}
+
+With the default settings, the tile selection algorithm prioritizes getting the needed detail to the screen as quickly as possible. The downside of this approach is that it can sometimes lead to "holes" - or parts of the model that are visibly missing - when the camera moves. This happens when the camera movement reveals a part of the model that wasn't previously visible, and the tiles necessary to show that part of the model are not yet loaded.
+
+This default behavior can be changed by setting [TilesetOptions::forbidHoles](\ref Cesium3DTilesSelection::TilesetOptions::forbidHoles) to `true`. When holes are forbidden, loading will take longer, because some extra tiles will need to be loaded in order to fill the potential holes, and camera movement may still reveal areas of lower detail, but it will never reveal a part of the model that is missing entirely. This may offer a better user experience for many applications.
+
+_Forbid Holes_ mode operates via a small change in `_visitTileIfNeeded`. Normally, when a tile is culled, we either don't load it at all, or we load with it with the lower `Preload` [priority](#load-prioritization). But when holes are forbidden, a culled tile is instead loaded with `Normal` priority, and it is also represented in the `TraversalDetails` returned from the method. This means that the subtree containing this tile will be [kicked](#kicking) if this tile is not yet loaded. This guarantees the subtree will not be rendered until this tile is loaded, which in turn guarantees that if the camera is moved, so that this tile suddenly becomes visible, it will be possible to render it immediately. There will not be a hole.
+
+## Unconditionally-Refined Tiles
+
+A tile that is "unconditionally refined" will always be _REFINED_, it will never be _RENDERED_. We can think of such a tile as having infinite geometric and screen-space error. Unconditionally-refined tiles are used in the following situations:
+
+1. The `_pRootTile` held by the `TilesetContentManager`. This is root tile of the entire bounding-volume hierarchy. For a standard 3D Tiles `tileset.json` tileset, the root tile defined in the `tileset.json` is the single child of this `_pRootTile`.
+2. Tiles whose "content" is an [external tileset](https://github.com/CesiumGS/3d-tiles/tree/main/specification#core-external-tilesets).
+3. Tiles that have a geometric error that is higher than their parent's.
+
+In all three cases, the tile flag is set with a call to [Tile::setUnconditionallyRefine](\ref Cesium3DTilesSelection::Tile::setUnconditionallyRefine).
+
+Consider a tileset which has a root tile with some renderable content, and four children. Three of the children have normal renderable content as well, but the forth is an external tileset. This means that it provides more nodes for the bounding-volume hierarchy, but it does not have any renderable content itself. Even once that fourth tile is done loading, we can't render it. If we did, we would end up creating a visible hole in the tileset that would not be filled until the children of the tile finished loading as well.
+
+Normally, a tile, no matter how large its screen-space error, can be rendered in some cases, such as when other tiles are not loaded yet. While we can conceptually think of unconditionally-refined tiles as simply having very large error, the handling of them within the selection algorithm goes a bit deeper, in order to ensure they are never rendered.
+
+To start, [Tile::isRenderable](\ref Cesium3DTilesSelection::Tile::isRenderable) will return false for an unconditionally-refined tile. This will ensure the tile is [Kicked](#kicking) and thus not rendered.
+
+> [!note]
+> There is one exception where `isRenderable` will return true for an unconditionally-refined tile: when that tile also does not have any children, and never will. It would be an unusual - perhaps buggy! - tileset that has such a situation, but when it does occur, we must allow the tile to render so that its sibilings, if any, may render.
+
+Also, when the children of an unconditionally-refined tile are kicked out of the render list, those tiles will _not_ also be kicked out of the load queue, even if the [Loading Descendant Limit](#loading-descendant-limit) is exceeded. Because the unconditionally-refined tile will never become renderable, failing to load its children would result in the entire subtree never becoming renderable.
+
+Finally, when [Forbid Holes](#forbid-holes) is enabled, `_visitTileIfNeeded` will always visit unconditionally-refined tiles, even if they're culled. This is necessary because the non-renderable, unconditionally-refined tile would otherwise block renderable siblings from rendering, too. By visiting the unconditionally-refined tile, we allow its children to load, and thereby allow the subtree to become renderable.
+
+## TilesetContentLoader {#tileset-content-loader}
+
+The process of loading content and children for a tile is delegated to a pluggable interface called [TilesetContentLoader](\ref Cesium3DTilesSelection::TilesetContentLoader). This means that the Cesium Native 3D Tiles selection algorithm is not limited to 3D Tiles. Anything that can be portrayed in Cesium Native's 3D Tiles and glTF object model can be loaded, selected, and rendered by Cesium Native. Every `Tile` is associated with a loader, and that loader is used to load that `Tile`'s content. Child `Tiles` may use the same or a different loader.
+
+The following `TilesetContentLoader` types are currently provided:
+
+* `TilesetJsonLoader` - The standard loader for explicit 3D Tiles based on `tileset.json`. Individual tile content is loaded via [GltfConverters](\ref Cesium3DTilesContent::GltfConverters).
+* `CesiumIonTilesetLoader` - Loads a 3D Tiles asset or `layer.json` / `quantized-mesh-1.0` terrain asset from [Cesium ion](https://cesium.com/platform/cesium-ion/), by delegating to one of the other loaders as appropriate. Automatically handles refreshing the token when it expires.
+* `LayerJsonTerrainLoader` - Loads terrain described by a `layer.json` and individual terrain tiles in `quantized-mesh-1.0` format.
+* `ImplicitQuadtreeLoader` - Loads a 3D Tiles 1.1 implicit quadtree.
+* `ImplicitOctreeLoader` - Loads a 3D Tiles 1.1 implicit octree.
+* `EllipsoidTilesetLoader` - Generates tiles on-the-fly by tessellating an ellipsoid, such as the WGS84 ellipsoid. Does not load any data from the disk or network.
+
+Other loaders can be added by users of the library.
+
+## Implicit Tilesets {#implicit-tilesets}
+
+The tile selection algorithm selects tiles from an explicit representation of the bounding-volume hierarchy. Every tile in the tileset is represented as a [Tile](\ref Cesium3DTilesSelection::Tile) instance. Starting with 3D Tiles 1.1, the bounding-volume hierarchy may instead be defined _implicitly_ using [Implicit Tiling](https://github.com/CesiumGS/3d-tiles/tree/main/specification#core-implicit-tiling). This is much more efficient representation when the bounding-volume hierarchy has a uniform subdivision structure.
+
+> [!note]
+> The older `layer.json` / [quantized-mesh-1.0](https://github.com/CesiumGS/quantized-mesh) terrain format also uses a form of implicit tiling.
+
+Cesium Native supports implicit tiling by lazily transforming the implicit representation into an explicit one as individual tiles are needed. This happens in the `TilesetContentManager::createLatentChildrenIfNecessary` method, called for each tile near the top of `_visitTileIfNeeded`. This method attempts to create explicit tile instances for the implicitly-defined children of the current tile by invoking [TilesetContentLoader::createTileChildren](\ref Cesium3DTilesSelection::TilesetContentLoader::createTileChildren). Thus, the `TilesetContentLoader` interface is not only responsible for loading tile content, it is also responsible for creating additional `Tile` instances in the bounding-volume hierarchy as needed.
+
+Implicit [loaders](#tileset-content-loader), such as `ImplicitQuadtreeLoader`, `ImplicitOctreeLoader`, and `LayerJsonTerrainLoader`, implement this method by determining in their own way whether this tile has any children, and creating them if it does. In some cases, extra asynchronous work, like downloading subtree availability files, may be necessary to determine if children exist. In that case, the `createTileChildren` will return [TileLoadResultState::RetryLater](\ref Cesium3DTilesSelection::TileLoadResultState::RetryLater) to signal that children may exist, but they can't be created yet. The selection algorithm will try again next frame if the tile's children are still needed.
+
+Currently, a `Tile` instance, once created, will not be destroyed until the entire [Tileset](\ref Cesium3DTilesSelection::Tileset) is destroyed. This is true for `Tile` instances created explicitly from `tileset.json` as well as `Tile` instances created lazily by the implicit loaders. This is convenient because we don't need to worry about a `Tile` instance vanishing unexpectedly, but it can cause a slow increase in memory usage over time.
+
+> [!note]
+> The above refers to `Tile` instances, _not_ their content. Content is unloaded when it is no longer needed. This is important because content is by far the largest portion of a tile.
+
 ## Additional Topics Not Yet Covered {#additional-topics}
 
 Here are some additional selection algorithm topics that are not yet covered here, but should be in the future:
 
-* Load prioritization
-* Forbid Holes <!--! \anchor forbid-holes !-->
-* Unconditionally-Refined Tiles
 * Occlusion Culling
-* External Tilesets and Implicit Tiles
 * Additive Refinement