From a8d5dc9ae92d53f2906dd29621215c73b59ce857 Mon Sep 17 00:00:00 2001 From: AdamRashid96 <71362382+AdamRashid96@users.noreply.github.com> Date: Thu, 18 Jan 2024 11:17:54 -0800 Subject: [PATCH] Updates to docs for gaussian splatting (#2782) * additions to docs --------- Co-authored-by: Justin Kerr --- docs/nerfology/methods/splat.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/nerfology/methods/splat.md b/docs/nerfology/methods/splat.md index 745eccb1de..8e613e251b 100644 --- a/docs/nerfology/methods/splat.md +++ b/docs/nerfology/methods/splat.md @@ -1,7 +1,23 @@ # Gaussian Splatting +

Real-Time Radiance Field Rendering

+ + +```{button-link} https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ +:color: primary +:outline: +Paper Website +``` + [3D Gaussian Splatting](https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/) was proposed in SIGGRAPH 2023 from INRIA, and is a completely different method of representing radiance fields by explicitly storing a collection of 3D volumetric gaussians. These can be "splatted", or projected, onto a 2D image provided a camera pose, and rasterized to obtain per-pixel colors. Because rasterization is very fast on GPUs, this method can render much faster than neural representations of radiance fields. ### Installation + +```{button-link} https://docs.gsplat.studio/ +:color: primary +:outline: +GSplat +``` + Nerfstudio uses [gsplat](https://github.com/nerfstudio-project/gsplat) as its gaussian rasterization backend, an in-house re-implementation which is designed to be more developer friendly. This can be installed with `pip install gsplat`. The associated CUDA code will be compiled the first time gaussian splatting is executed. Some users with PyTorch 2.0 have experienced issues with this, which can be resolved by either installing gsplat from source, or upgrading torch to 2.1. ### Data @@ -13,6 +29,12 @@ Because gaussian splatting trains on *full images* instead of bundles of rays, t ### Running the Method To run gaussian splatting, run `ns-train gaussian-splatting --data `. Just like NeRF methods, the splat can be interactively viewed in the web-viewer, loaded from a checkpoint, rendered, and exported. +#### Quality and Regularization +The default settings provided maintain a balance between speed, quality, and splat file size, but if you care more about quality than training speed or size, you can decrease the alpha cull threshold +(threshold to delete translucent gaussians) and disable culling after 15k steps like so: `ns-train gaussian-splatting --pipeline.model.cull_scale_thresh=0.005 --pipeline.model.continue_cull_post_densification=False --data ` + +A common artifact in splatting is long, spikey gaussians. [PhysGaussian](https://xpandora.github.io/PhysGaussian/) proposes an example of a scale-regularizer that encourages gaussians to be more evenly shaped. To enable this, set the `use_scale_regularization` flag to `True`. + ### Details For more details on the method, see the [original paper](https://arxiv.org/abs/2308.04079). Additionally, for a detailed derivation of the gradients used in the gsplat library, see [here](https://arxiv.org/abs/2312.02121).