Improved 3D Gaussian Splatting

Dictionary of God that is ideas (at the moment lame):

• ** Strategy-1 : ** Perform NAFNet deblurring, and then SPSR superresolution of images before COLMAP routine + GS routine
• ** Strategy-2 : ** Peform Real-ESRGAN deblurring and then SPSR superresolution of images before COLMAP routine + GS routine
• ** Strategy-3 : ** Poisson blending of Strategy-1 and Strategy-2 outputs (before COLMAP routine + GS routine)

Installation

Platform: Ubuntu 22+

Setting up Real-ESRGAN

cd Real-ESRGAN
pip install basicsr
pip install -r requirements.txt
python setup.py develop

wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-x4v3.pth

Basic Inference

For inference, the basic commands are:

python inference_realesrgan.py -n realesr-general-x4v3 -i <folder of images>
rm -rf esrgan_output
mv results esrgan_output

Setting up NAFNet

cd NAFNet
gdown https://drive.google.com/file/d/14D4V4raNYIOhETfcuuLI3bGLB-OYIv6X/view?usp=sharing --fuzzy
mv NAFNet-REDS-width64.pth experiments/pretrained_models
pip install -r requirements.txt
python setup.py develop --no_cuda_ext

Basic Inference

python deblur.py --dir <folder of images>

Running Commands

chmod +x run_strategy_1.sh
./run_strategy_1 <path to images folder>

chmod +x run_strategy_2.sh
./run_strategy_2 <path to images folder>

chmod +x run_strategy_3.sh
./run_strategy_3 <path to images folder>

Rendering and Metrics

I use Nerfstudio (which uses Splatfacto) and Hloc for rendering the gaussian splats, and use original GS repo code for evaluating metrics. For rendering and evaluation, run:

cd rendering_and_metrics
python gs_schedule.py --dir <folder containing images to render (which can be blurr)> --gt_dir <ground truths for those images, especially if they are blurr>

The results are stored in "args.dir"+"_gs" folder as renderings and metrics (.txt) file.

Some Results

Comparison with Blurry and Ground Truth

Ground Truth	Blurry	NAFNet->SPSR (Strategy-1)
SSIM : 0.8418334 PSNR : 34.5210227 LPIPS: 0.0444530	SSIM : 0.7365065 PSNR : 31.6519640 LPIPS: 0.2315612	SSIM : 0.7202776 PSNR : 31.8796042 LPIPS: 0.1643362

Action of Debluring Models on Rendering

Ground Truth Gaussian Splatting
SSIM: 0.8537942
PSNR: 34.8942894
LPIPS: 0.0486813

	Motion-Blurred	Lens-Blurred	Gaussian-Blurred
NAFNet	SSIM: 0.7144054 PSNR: 31.8041596 LPIPS: 0.2725016	SSIM: 0.8256776 PSNR: 33.8201837 LPIPS: 0.0865366	SSIM: 0.7774865 PSNR: 32.7108423 LPIPS: 0.1944816
Real-ESRGAN	SSIM: 0.7185786 PSNR: 31.4306795 LPIPS: 0.3563879	SSIM: 0.7592866 PSNR: 32.5180515 LPIPS: 0.1568963	SSIM: 0.6342124 PSNR: 28.8450565 LPIPS: 0.5116010
Bad-Gaussians	SSIM: 0.7338864 PSNR: 31.7400618 LPIPS: 0.3800035	SSIM: 0.7877066 PSNR: 33.4288919 LPIPS: 0.1447114	SSIM: 0.7671796 PSNR: 32.7544921 LPIPS: 0.2529144

Comparison with Other Deblurring Rendering Techniques

Original Gaussian Splatting	Preprocessed-Gaussian Splatting (Strategy-1)	Preprocessed-Gaussian Splatting (Strategy-3)

BAD-Gaussians	Deblur-NeRF