add backprop grad of opacity compensation #140
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Maybe we should upgrade/bump version |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why
In the previous PR #117 I found some side efforts when training with a lot tiny gaussians: the gaussian splatting training is less likely to densify tiny gaussians and as a result a lot fine details cannot be constructed when training in antialiased mode.
See an example below: the first image is the classic mode, where even though the color of bag at zoom in is not correct (due to aliasing), the geometric textures are reconstructed. The second image is the anti-aliased mode, while the color at zoom in is correct, but a lot geometric textures are lost.
After some deep dives, I realized that the root cause is we didn't back-propagate gradients through the opacity compensation in previous PR and as a result the gradient calculated for cov2d of tiny splats are no longer close to the true gradient when the compensation factor is away from 1. Adding back the backprop grad of opacity compensation seems to fix the issue in my experiments. Below is a video demo of this PR.
cov2d_true_grad.mp4
Sanity Checks
I further did some sanity checks and found that have the true gradient is beneficial in marginally improving PSNR metrics as well. So I consider this to be a necessary step toward enabling antialiased mode in splatfacto by default.
PSNR metrics of bicycle dataset (downsampled by factor of 4) in mipnerf360
Besides the GS metrics, I also modify the current unit test of project_gaussians to ensure the implementation of gradient is numerical correct.