Skip to content

0.2.0

Latest
Latest
Compare
Choose a tag to compare
@github-actions github-actions released this 30 Jan 00:05
0.2.0
82e7c68
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

image

Brush 0.2 goes from a proof of concept to a tool ready for real world data! It still only implements the “basics” of Gaussian Splatting, but trains as fast as gsplat to a (slightly) higher quality than gsplat. It also comes with nicer workflows, a CLI, dynamic gaussian rendering, and lots of other new features.

The next release will focus on going beyond the basics of Gaussian Splatting, and implementing extensions that help to make Brush more robust, faster, and higher quality than other splatting alternatives. This might mean that the outputs are no longer 100% compatible with other splat viewers, so more work will also be done to make the Brush web viewer a great experience.

Features

  • Brush now measures higher PSNR/SSIM than gsplat on the mipnerf360 scenes. Of course, gsplat with some more tuned settings might reach these numbers as well, but this shows Brush is grown up now!

  • Faster training overall by optimizing the kernels, fixing various slowdowns, and reducing memory use.

  • Brush now has a CLI!

    • Simply run brush –help to get an overview. The basic usage is brush PATH –args.
    • Any command works with --with-viewer which opens the UI for easy debugging.

  • Add flythrough controls supporting both orbiting, FPS controls, flythrough controls, and panning.

    • See the ‘controls’ popout in the scene view for a full overview.

  • Load data from a URL. If possible the data will be streamed in, and the splat will update in real-time.
    -For a web version, just pass in ?url=

  • On the web, pass in ?zen=true to enable ‘zen’ mode which makes the viewer fullscreen.

  • Add support for viewing dynamic splats

    • Either loaded as a sequence of PLY files (in a folder or zip)
    • Or as a custom data format “ply with delta frames”
    • This was used for Cat4D and for Cap4D
    • Felix kindly shared their script to export this data for reference.

  • Open directories directly, instead of only zip files.

    • ZIP files are still supported for all operations - as this is important for the web version.

  • Support transparent images.

    • Images with alpha channels will force the output splat to match this transparency.
    • Alternatively, you can include a folder of ‘masks’. This will ignore those parts of the image while training.

  • More flexible COLMAP & nerfstudio dataset format

    • Support more of the various options, and differing file structures.

    • If your dataset has a single ply file, it will be used for the initial point cloud.

    • While training, the up-axis is rotated such that the ground is flat (thanks to @fhahlbohm)

      • Note: The exported ply will however still match your input data. I’m investigating how to best handle this in the future - either as an option to rotate the splat, or by writing metadata into the exported splat.

Enhancements

  • Add alpha_loss_weight arg to control how heavy to weigh the alpha loss
    • Nb: not applicable to masks mode
  • Log memory usage to rerun while training
  • Fix SH clamping values to 0 (#76 thanks to @fhahlbohm)
  • Better logic to pick ‘nearest’ dataset view
  • Better splat pruning logic
  • Remove ESC to close
  • The web version has SSIM enabled again
  • Display more detailed error traces in the UI and CLI when something goes wrong
  • Different method of emitting tile intersections (#63 for details)
    • Fixes some potential corruptions depending on your driver/shader compiler.
  • Read up-axis from PLY file if it’s included
  • Eval PSNR/SSIM now simulate a 8 bit roundtrip for fair comparison
  • Add an option --export-every to export a ply file every so many steps
    • See --export-path and --export-name for the location of the ply
  • Add an option --eval-save-to-disk to save eval images to disk
    • See –export-path for
  • Add notes in CLI & UI about running in debug mode (advising to compile with --release).
  • Relax camera constraints, allow further zoom in/out
  • Relax constraints on fields in the UI - now can enter values outside of slider range.
  • Improvements to the UI, less unnecessary padding.

Highlighted Fixes

  • Dataset and scene view now match exactly 1:1
  • Fix UI sometimes not updating when starting a new training run.
  • Sort eval images to be consistent with the MipNeRF eval images
  • Fix a crash from the KNN initialization

Demo (Chrome only currently)

Reference Garden scene (650MB)

Mushroom I captured on a walk (capture was only 50 images or so, background is a bit blurry!)

Thanks

Thanks to everybody in the Brush discord, in particular @fasteinke for reporting many breakages along the way, @fhahlbohm for contributions and helping me fix my results table, @Simon.Bethke and @gradeeterna for test data, @felixtaubner for the 4D splat export script.

Download brush-app 0.2.0

File Platform Checksum
brush-app-aarch64-apple-darwin.tar.xz Apple Silicon macOS checksum
brush-app-x86_64-pc-windows-msvc.zip x64 Windows checksum
brush-app-x86_64-unknown-linux-gnu.tar.xz x64 Linux checksum

Contributors

fasteinke, fhahlbohm, and 2 other contributors
Assets 12
Loading