Skip to content

Commit

Permalink
initial publishing
Browse files Browse the repository at this point in the history
  • Loading branch information
innimu committed Sep 3, 2024
1 parent 03dcaea commit e0f82f4
Show file tree
Hide file tree
Showing 5 changed files with 63 additions and 74 deletions.
18 changes: 9 additions & 9 deletions _sources/docs/review/DreamBooth3D.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,20 +260,20 @@ DreamBooth3D Overview

---

### **3.3.1 Stage-1️⃣: 3D with Partial DreamBooth**
### **3.3.1 Stage 1️⃣: 3D with Partial DreamBooth**

:::{figure-md}
<img src="../../pics/DreamBooth3D/Untitled_9.png" alt="Untitled_9" class="bg-primary mb-1">
<img src="../../pics/DreamBooth3D/Untitled_9.png" alt="Untitled_9" class="bg-primary mb-1" style="width:400px; display:block; margin: 0 auto;" />

Stage-1 : 3D with Partial DreamBooth
:::

- 입력된 Subject 이미지를 사용하여 DreamBooth 모델 $\hat{\mathcal{D}}_\theta$ 를 훈련

---
🌟 **DreamBoothT2I 모델의 초기 체크포인트가 (=부분적으로 파인튜닝한 결과) 주어진 subject view에 과적합되지 않음을 확인**
🌟 **DreamBoothT2I 모델의 초기 체크포인트가 (=부분적으로 파인튜닝한 결과) 주어진 subject view에 과적합되지 않음을 확인**

**⇒ partial DreamBooth (부분적으로 파인튜닝한 Dreambooth)**
⇒ partial DreamBooth (부분적으로 파인튜닝한 Dreambooth)
---

- **partial DreamBooth** 모델 하에 DreamFusion은 더 일관된 3D NeRF를 생성가능
Expand All @@ -290,10 +290,10 @@ Stage-1 : 3D with Partial DreamBooth

---

### **3.3.2 Stage-2️⃣: Multi-view Data Generation**
### **3.3.2 Stage 2️⃣: Multi-view Data Generation**

:::{figure-md}
<img src="../../pics/DreamBooth3D/Untitled_10.png" alt="Untitled_10" class="bg-primary mb-1">
<img src="../../pics/DreamBooth3D/Untitled_10.png" alt="Untitled_10" class="bg-primary mb-1 "style="width:400px; display:block; margin: 0 auto;" />

Stage-2: Multi-view Data Generation
:::
Expand Down Expand Up @@ -331,18 +331,18 @@ Stage-2: Multi-view Data Generation
- 위 그림을 통해 체크할 부분

:::{figure-md}
<img src="../../pics/DreamBooth3D/Untitled_11.png" alt="Untitled_11" class="bg-primary mb-1">
<img src="../../pics/DreamBooth3D/Untitled_11.png" alt="Untitled_11" class="bg-primary mb-1" style="width:400px; display:block; margin: 0 auto;" />

:::

- **fully-trained DreamBooth** 를 사용한 Img2Img 변환의 샘플 출력
- 입력 NeRF 렌더링의 시점을 유지하면서도 subject 이미지와 더 유사한 모습
- 기존 연구들과 달리 Img2Img 변환을 DreamBooth, NeRF 3D assets 과 결합하여 사용 (기존 연구의 경우 Img2Img 변환을 이미지 editing 응용으로만 사용)

### **3.3.3 Stage-3️⃣: Final NeRF with Multi-view DreamBooth**
### **3.3.3 Stage3️⃣: Final NeRF with Multi-view DreamBooth**

:::{figure-md}
<img src="../../pics/DreamBooth3D/Untitled_12.png" alt="Untitled_12" class="bg-primary mb-1">
<img src="../../pics/DreamBooth3D/Untitled_12.png" alt="Untitled_12" class="bg-primary mb-1" style="width:400px; display:block; margin: 0 auto;" />

Stage-3 : Final NeRF with Multi-view DreamBooth
SDS와 multi-view reconstruction 손실을 사용한 최종 NeRF 최적화
Expand Down
38 changes: 19 additions & 19 deletions docs/experiments/js_exp.html
Original file line number Diff line number Diff line change
Expand Up @@ -529,7 +529,7 @@ <h2>1. 개요<a class="headerlink" href="#id1" title="Permalink to this heading"
<figure class="align-default" id="id3">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/4classes.png"><img alt="4classes" class="bg-primary mb-1" src="../../_images/4classes.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 653 </span><span class="caption-text">4 classes of leaves</span><a class="headerlink" href="#id3" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 652 </span><span class="caption-text">4 classes of leaves</span><a class="headerlink" href="#id3" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<ul class="simple">
Expand All @@ -550,7 +550,7 @@ <h2>2. Baseline 구축<a class="headerlink" href="#baseline" title="Permalink to
<figure class="align-default" id="id4">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/multiple_ex.png"><img alt="multiple_ex" class="bg-primary mb-1" src="../../_images/multiple_ex.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 654 </span><span class="caption-text">4 classes of leaves</span><a class="headerlink" href="#id4" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 653 </span><span class="caption-text">4 classes of leaves</span><a class="headerlink" href="#id4" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
</section>
Expand All @@ -562,7 +562,7 @@ <h2>3. Stable diffusion fine tuning<a class="headerlink" href="#stable-diffusion
<figure class="align-default" id="id5">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/multiple_sd.png"><img alt="multiple_sd" class="bg-primary mb-1" src="../../_images/multiple_sd.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 655 </span><span class="caption-text">prompt: “a photo of leaves with multiple diseases</span><a class="headerlink" href="#id5" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 654 </span><span class="caption-text">prompt: “a photo of leaves with multiple diseases</span><a class="headerlink" href="#id5" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<ul class="simple">
Expand All @@ -573,7 +573,7 @@ <h2>3. Stable diffusion fine tuning<a class="headerlink" href="#stable-diffusion
<figure class="align-default" id="id6">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/multiple_db.png"><img alt="multiple_db" class="bg-primary mb-1" src="../../_images/multiple_db.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 656 </span><span class="caption-text">prompt: “a photo of a &lt;diseaes-leaf&gt; leaf”</span><a class="headerlink" href="#id6" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 655 </span><span class="caption-text">prompt: “a photo of a &lt;diseaes-leaf&gt; leaf”</span><a class="headerlink" href="#id6" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<ul class="simple">
Expand All @@ -584,13 +584,13 @@ <h2>3. Stable diffusion fine tuning<a class="headerlink" href="#stable-diffusion
<figure class="align-default" id="id7">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/leaf_sd.png"><img alt="leaf_sd" class="bg-primary mb-1" src="../../_images/leaf_sd.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 657 </span><span class="caption-text">fine tuning 전</span><a class="headerlink" href="#id7" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 656 </span><span class="caption-text">fine tuning 전</span><a class="headerlink" href="#id7" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<figure class="align-default" id="id8">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/leaf_db.png"><img alt="leaf_db" class="bg-primary mb-1" src="../../_images/leaf_db.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 658 </span><span class="caption-text">fine tuning 후</span><a class="headerlink" href="#id8" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 657 </span><span class="caption-text">fine tuning 후</span><a class="headerlink" href="#id8" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<ul class="simple">
Expand All @@ -600,13 +600,13 @@ <h2>3. Stable diffusion fine tuning<a class="headerlink" href="#stable-diffusion
<figure class="align-default" id="id9">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/photo_sd.png"><img alt="photo_sd" class="bg-primary mb-1" src="../../_images/photo_sd.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 659 </span><span class="caption-text">fine tuning 전</span><a class="headerlink" href="#id9" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 658 </span><span class="caption-text">fine tuning 전</span><a class="headerlink" href="#id9" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<figure class="align-default" id="id10">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/photo_db.png"><img alt="photo_db" class="bg-primary mb-1" src="../../_images/photo_db.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 660 </span><span class="caption-text">fine tuning 후</span><a class="headerlink" href="#id10" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 659 </span><span class="caption-text">fine tuning 후</span><a class="headerlink" href="#id10" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<ul class="simple">
Expand All @@ -616,13 +616,13 @@ <h2>3. Stable diffusion fine tuning<a class="headerlink" href="#stable-diffusion
<figure class="align-default" id="id11">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/multiple_pp.png"><img alt="multiple_pp" class="bg-primary mb-1" src="../../_images/multiple_pp.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 661 </span><span class="caption-text">multiple diseases class 이미지 생성 결과, prompt: “&lt;diseaes-leaf&gt; leaf”</span><a class="headerlink" href="#id11" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 660 </span><span class="caption-text">multiple diseases class 이미지 생성 결과, prompt: “&lt;diseaes-leaf&gt; leaf”</span><a class="headerlink" href="#id11" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<figure class="align-default" id="id12">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/leaf_pp.png"><img alt="leaf_pp" class="bg-primary mb-1" src="../../_images/leaf_pp.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 662 </span><span class="caption-text">leaf 생성 결과, prompt: “leaf”</span><a class="headerlink" href="#id12" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 661 </span><span class="caption-text">leaf 생성 결과, prompt: “leaf”</span><a class="headerlink" href="#id12" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<ul class="simple">
Expand All @@ -631,7 +631,7 @@ <h2>3. Stable diffusion fine tuning<a class="headerlink" href="#stable-diffusion
<figure class="align-default" id="id13">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/photo_pp.png"><img alt="photo_pp" class="bg-primary mb-1" src="../../_images/photo_pp.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 663 </span><span class="caption-text">photo 생성 결과, prompt: “photo”</span><a class="headerlink" href="#id13" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 662 </span><span class="caption-text">photo 생성 결과, prompt: “photo”</span><a class="headerlink" href="#id13" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<ul class="simple">
Expand All @@ -652,7 +652,7 @@ <h2>4. 성능 비교<a class="headerlink" href="#id2" title="Permalink to this h
<figure class="align-default" id="id14">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/result_base.png"><img alt="result_base" class="bg-primary mb-1" src="../../_images/result_base.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 664 </span><span class="caption-text">result_base</span><a class="headerlink" href="#id14" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 663 </span><span class="caption-text">result_base</span><a class="headerlink" href="#id14" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<p>생성한 이미지를 추가 데이터로 활용한 경우</p>
Expand All @@ -662,7 +662,7 @@ <h2>4. 성능 비교<a class="headerlink" href="#id2" title="Permalink to this h
<figure class="align-default" id="id15">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/result_new.png"><img alt="result_new" class="bg-primary mb-1" src="../../_images/result_new.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 665 </span><span class="caption-text">result_now</span><a class="headerlink" href="#id15" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 664 </span><span class="caption-text">result_now</span><a class="headerlink" href="#id15" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<ul class="simple">
Expand All @@ -688,37 +688,37 @@ <h2>6. Appendix<a class="headerlink" href="#appendix" title="Permalink to this h
<figure class="align-default" id="id16">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/cat.png"><img alt="cat" class="bg-primary mb-1" src="../../_images/cat.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 666 </span><span class="caption-text">cat 생성 결과, prompt: “cat”</span><a class="headerlink" href="#id16" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 665 </span><span class="caption-text">cat 생성 결과, prompt: “cat”</span><a class="headerlink" href="#id16" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<figure class="align-default" id="id17">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/sea.png"><img alt="sea" class="bg-primary mb-1" src="../../_images/sea.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 667 </span><span class="caption-text">sea 생성 결과, prompt: “sea”</span><a class="headerlink" href="#id17" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 666 </span><span class="caption-text">sea 생성 결과, prompt: “sea”</span><a class="headerlink" href="#id17" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<figure class="align-default" id="id18">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/pirate.png"><img alt="pirate" class="bg-primary mb-1" src="../../_images/pirate.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 668 </span><span class="caption-text">pirate 생성 결과, prompt: “pirate”</span><a class="headerlink" href="#id18" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 667 </span><span class="caption-text">pirate 생성 결과, prompt: “pirate”</span><a class="headerlink" href="#id18" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<figure class="align-default" id="id19">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/illustration.png"><img alt="illustration" class="bg-primary mb-1" src="../../_images/illustration.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 669 </span><span class="caption-text">illustration 생성 결과, prompt: “illustration”</span><a class="headerlink" href="#id19" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 668 </span><span class="caption-text">illustration 생성 결과, prompt: “illustration”</span><a class="headerlink" href="#id19" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<figure class="align-default" id="id20">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/animation.png"><img alt="animation" class="bg-primary mb-1" src="../../_images/animation.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 670 </span><span class="caption-text">animation 생성 결과, prompt: “animation”</span><a class="headerlink" href="#id20" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 669 </span><span class="caption-text">animation 생성 결과, prompt: “animation”</span><a class="headerlink" href="#id20" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
<figure class="align-default" id="id21">
<a class="bg-primary mb-1 reference internal image-reference" href="../../_images/wallpaper.png"><img alt="wallpaper" class="bg-primary mb-1" src="../../_images/wallpaper.png" style="width: 700px;" /></a>
<figcaption>
<p><span class="caption-number">Fig. 671 </span><span class="caption-text">wallpaper 생성 결과, prompt: “wallpaper”</span><a class="headerlink" href="#id21" title="Permalink to this image">#</a></p>
<p><span class="caption-number">Fig. 670 </span><span class="caption-text">wallpaper 생성 결과, prompt: “wallpaper”</span><a class="headerlink" href="#id21" title="Permalink to this image">#</a></p>
</figcaption>
</figure>
</section>
Expand Down
Loading

0 comments on commit e0f82f4

Please sign in to comment.