You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<h4style="text-align:center"><em>Multi-Concept Prompt Learning (MCPL) pioneers the novel task of mask-free text-guided learning for multiple prompts from one scene. Our approach not only enhances current methodologies but also paves the way for novel applications, such as facilitating knowledge discovery through natural language-driven interactions between humans and machines. </em></h4>
<!-- <h4 style="text-align:center"><em>Multi-Concept Prompt Learning (MCPL) pioneers the novel task of mask-free text-guided learning for multiple prompts from one scene. Our approach not only enhances current methodologies but also paves the way for novel applications, such as facilitating knowledge discovery through natural language-driven interactions between humans and machines. </em></h4> -->
35
+
<h4style="text-align:center"><em>TL;DR: We propose a framework that allows us to discover and manipulate multiple concepts in a given image with partial text instructions. </em></h4>
<h2>Learning Multiple Concepts from Single Image and Editing</h2>
48
-
<h3>teddybear and skateboard example</h3>
49
+
<h3>teddy bear and skateboard example</h3>
49
50
<divclass="cat-hat-main">
50
-
<divclass="intro-text"> Our method learn multiple new concepts and assuring disentangled and precise prompt-concept correlation (verified with per-prompt attention map). <br></div>
51
-
<divclass="intro-text"> We can then modifying each local concept by replacing the prompts/words to generate novel images (click words below to try editing).<br></div>
51
+
<divclass="intro-text"> Our method learns multiple new concepts and assures disentangled and precise prompt-concept correlation (click to view per-prompt attention maps). <br></div>
52
+
<divclass="intro-text"> We can then modify each concept by replacing the prompts/words to generate novel images (click words below to try editing).<br></div>
52
53
</div>
53
54
<br>
54
55
@@ -83,7 +84,8 @@ <h3>banana and basket example</h3>
83
84
84
85
<divclass="content">
85
86
<h2>Discovering OOD Concepts from Medical Image and Disentangling</h2>
86
-
</p>Our method opens an avenue for discovering/introducing new concepts the model have not seen before, from abundantly available natural language annotations such as paired textbook figures and captions. </p>
87
+
<divclass="body-text"> Our method opens an avenue for discovering/introducing new concepts the model have not seen before, from abundantly available natural language annotations such as paired textbook figures and captions. <br></div>
88
+
<br><br>
87
89
<h3>cardiac MRI example</h3>
88
90
<divclass="cat-hat-main">
89
91
<divclass="intro-text"> We learn out-of-distribution concepts using biomedical figures and their simplified captions. <br></div>
<i>MCPL</i> takes a sentence (top-left) and a sample image (top-right) as input, feeding them into a pre-trained text-guided diffusion model comprising a text encoder \(c_\phi\) and a denoising network \(\epsilon_\theta\). The string's multiple prompts are encoded into a sequence of embeddings which guide the network to generate images \(\tilde{X}_0\) close to the target one \(X_0\). MCPL focuses on learning multiple learnable prompts (coloured texts), updating only the embeddings \(\{v^*\}\) and \(\{v^\&\}\) of the learnable prompts while keeping \(c_\phi\) and \(\epsilon_\theta\) frozen. We introduce <i>Prompts Contrastive Loss (PromptCL)</i> to help separate multiple concepts within learnable embeddings. We also apply <i>Attention Masking (AttnMask)</i>, using masks based on the average cross-attention of prompts, to refine prompt learning on images. Optionally we associate each learnable prompt with an adjective (e.g., "brown" and "rolling") to improve control over each learned concept, referred to as <i>Bind adj.</i>
178
180
</p>
179
181
</div>
180
182
181
183
<divclass="content">
182
-
<h2>Introducing MCPL-one and MCPL-diverse training strategies</h2>
184
+
<h2>Introducing MCPL-one and MCPL-diverse Training Strategies</h2>
Learning and Composing “ball” and “box”. We learned the concepts of “ball” and “box” using different methods (top row) and composed them into unified scenes (bottom row). We compare three learning methods: Textural Inversion (Gal et al., 2022), which learns each concept separately from isolated images (left); MCPL-one, which jointly learns both concepts from un- cropped examples using a single prompt string (middle); and MCPL-diverse, which advances this by learning both concepts with per-image specific relationships (right).
Visual comparison of MCPL-diverse versus MCPL-one in learning per-image different concept tasks (cat with different hat example). As MCPL-diverse are specially designed for such tasks, it outperforms MCPL-one, which fails to capture per image different hat styles.
195
197
</p>
196
198
197
-
<h3>learning more than two concepts from a single image</h3>
199
+
<h3>Learning more than two concepts from a single image</h3>
A qualitative comparison between our method (MCPL-diverse) and mask-based approaches. Our MCPL-diverse, which neither uses mask inputs nor updates model parameters, showed decent results, outperforming most mask-based approaches and was closer to SoTA Break-A-Scene. Images modified from Break-A-Scene (Avrahami et al., 2023).
0 commit comments