Objectives:
Part 2 will focus on all the other common NLP tasks: token classification, language modeling (causal and masked), translation, summarization and question answering. It will also take a deeper dive in the whole Hugging Face ecosystem, in particular 🤗 Datasets and 🤗 Tokenizers.
Course link: https://huggingface.co/course/chapter5?fw=pt
Goal: Create a text-to-image search engine that allows users to search for images based on natural language queries using techniques like Multilingual Knowledge Distillation 2 to extend the embeddings to new languages.
Below are some CLIP model examples.
- https://github.com/orpatashnik/StyleCLIP
- https://huggingface.co/openai/clip-vit-base-patch32
- https://www.sbert.net/examples/applications/image-search/README.html#examples
CLIP model overview:
The CLIP model was proposed in Learning Transferable Visual Models From Natural Language Supervision by Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever. CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. Source: HuggingFace
Short Description: I made use of the CLIP Model implemented in the HuggingFace transformers library. The openai/clip-vit-base-patch32 was used as a pertained model using the unsplash-25k-photos dataset.
Please check the demo on the link below. You can use the examples in the Botton left corner for ideas on how it works. The demo was build using Hugging Face Spaces and gradio.
Link: https://huggingface.co/spaces/marcelcastrobr/CLIP-image-search
