This repository guides users through creating a semantic search using Amazon SageMaker and Amazon OpenSearch services
This code repository is for Semantic and Vector Search with Amazon OpenSearch Service Workshop. For more information about semantic search, please refer the workshop content.
- The workshop can only be deployed in us-east-1 region
- Use the Cloudformation template
cfn/semantic-search.yamlto create CF stack - Cloudformation stack name must be
semantic-searchas we use this stack name in our lab - You can click the following link to deploy CloudFormation Stack
| Region | Launch Template |
|---|---|
| US East (N. Virginia) |  |
There are 8 modules in this workshop:
-
Module 1 - Search basics: You will learn fundamentals of text search and semantic search. This section also introduces differences between a best matching algorithm, popularly known as BM25 similarity and semantic similarity.
-
Module 2 -Text search: You will learn text search with Amazon OpenSearch Service. In information retrieval this type of searching is traditionally called 'Keyword' search.
-
Module 3 - Semantic search: You will learn semantic search with Amazon OpenSearch Service and Amazon SageMaker. You will use a machine learning technique called Bidirectional Encoder Representations from transformers, popularly known as BERT. BERT uses a pre-trained natural language processing (NLP) model that represents text in the form numbers or in other words, vectors. You will learn to use vectors with kNN feature in Amazon OpenSearch Service.
-
Module 4 - Fullstack semantic search: You will bring together all the concepts learnt earlier with an user interface that shows the advantages of using semantic search with text search. You will be using Amazon OpenSearch Service, Amazon SageMaker, AWS Lambda, Amazon API Gateway and Amazon S3 for this purpose.
-
Module 5 - Fine tuning semantic search: Large language models like BERT show better results when they are trained in-domain, which means fine tuning the general model to fit ones particular business requirements in the domain of its application. You will learn how to fine tune the model for semantic search with the chosen data set.
-
Module 6 - Neural Search: Implement semantic search with OpenSearch Neural Search Plugin.
-
Module 7 - Retrieval Augmented Generation: Use semantic search result as context, combine the user input and context as prompt for large language models to generate factual content for knowledge intensive applications.
-
Module 8 - Conversational Search: Search with history context while leveraging RAG.
Please refer Semantic Search Workshop for lab instruction.
In this workshop, we use OpenSearch internal database to store username and password to simplify the lab. However in production env, you should design your security solution per your requirements. For more information , please refer Fine-grained access control and Identity and Access Management.
If you have any questions or feedback, please reach us by sending email to [email protected].
This library is licensed under the MIT-0 License. See the LICENSE file.