Semantic search in the context of generative AI, or any AI system, refers to the capability of the system to understand and process user queries based on the intent and contextual meaning rather than just relying on keywords.
You cannot be left behind in today’s generative AI world. Large Language Models (LLMs) and things like Retrieval Augmented Generation (RAG), LangChain and LlamaIndex are revolutionizing the world with their unique capabilities.
While Natural Language Processing (NPL) has made computers understand human language, there is one more advancement when it comes to searching and retrieving the data needed. That is where semantic search comes into play.
Sometimes models do not understand user queries with keyword search, misinterpreting and creating confusion while retrieving the required output. With the application of semantic search functionality, applications will have stronger search capabilities.
Today, we’ll dive deeper into what semantic search is, and how it works in the world of generative AI.
What is semantic search?
In the context of generative AI (or any AI system), semantic search refers to the system’s ability to understand and process user queries based on the intent and contextual meaning rather than just relying on keywords. Semantic search focuses on understanding the meaning and intent behind user queries, leveraging natural language processing (NLP) and machine learning to improve the accuracy and relevance of search results. A semantic search engine enhances search accuracy by interpreting the intent and contextual meaning behind user queries, relating terms, prioritizing results based on user context, and differentiating between similar phrases.
Semantic search plays a vital role in generative AI, since it's not just about retrieving information but also about generating content that aligns with the user's intent and context. For example, if a user is looking to generate a story based on a specific theme, the AI would need to comprehend that theme semantically to produce a relevant and coherent story.
How does semantic search work?
Semantic search stands at the forefront of a paradigm shift in the way we interact with information, embodying a transition from keyword-based retrieval to a more nuanced, intent-driven dialogue with data. The development of generative AI models like OpenAI’s GPT series has made significant strides in semantic understanding, allowing for more natural and contextually relevant interactions between users and AI. Semantic search work leverages methodologies and technologies such as natural language processing and machine learning to understand user intent and context, ultimately improving search relevance and user experience.
Here's a step-by-step explanation of the semantic search process as shown in the diagram:
User submits query. The user initiates the process by entering a search query into the system.
- Analyze intent and context. The LLM analyzes the query to understand the user's intent and context of the query.
- Extract intent and relationships. Semantic search processes the query to determine the relationships between the terms and the overall semantic meaning.
- Return intent and relationships. The extracted intent and relationships are sent back to the LLM.
- Retrieve relevant data. The LLM uses the understood intent to retrieve data that is relevant to the query.
- Rank data based on relevance. The ranking algorithm evaluates the retrieved data from a vector database, ranking it according to its relevance to the query.
- Return ranked results. The ranked results are then sent back to the LLM.
- Present generated content/output. Finally, the LLM presents the generated content or search results to the user, completing the semantic search process.
Understanding Natural Language and Context
Semantic search relies heavily on natural language processing (NLP) to understand and process human language. NLP enables search engines to interpret the meaning of words and phrases, taking into account the context in which they are used. This allows semantic search engines to deliver more accurate and relevant results, even when the search query is ambiguous or contains synonyms. By leveraging NLP, semantic search engines can understand the nuances of human language, ensuring that users receive relevant results that match their search intent.
Importance of semantic search and understanding search intent
It is very important for any system to understand the user queries and present them in a more accurate — if not contextual — format. Imagine you’re browsing your favorite eCommerce website; you enter a query in the search bar for the product you are looking for, and find the search results are broken — all you see is a set of clothes presented on your screen. This of course has negatively impacted your entire user experience.
Now, that is where semantic search functionality plays a vital role.
The essence of semantic search lies in its ability to understand the intent and contextual nuances behind user queries, transforming the search experience from a simplistic keyword match to a sophisticated, intent-driven interaction. This leap is critical as it ensures users find genuinely relevant content, not just pages with keyword matches.
Semantic search's pivotal role in improving data retrieval accuracy across industries — from eCommerce to healthcare — streamlines operations, empowers informed decision making and enriches the overall user experience. By capturing the subtleties of human language, semantic search is reshaping our access to and interaction with the vast expanse of digital information.
Amazon has integrated semantic search with their eCommerce websites around the globe. Some other companies that use semantic search include Google, Microsoft (Bing, IBM's watsonx, OpenAI, Anthropic, etc. Even Elon Musk is interested in adding semantic search functionality to X (formerly Twitter).
Comparison to keyword search
Unlike traditional keyword search engines that operate by matching words and utilizing techniques such as query expansion and natural language processing, semantic search focuses on the meanings behind queries. This distinction is crucial as it highlights the limitations of keyword searches in understanding user intent compared to semantic approaches. For example, Google search has evolved significantly, employing a crawling process to index web content and retrieve information through various input types. While it is effective in broad searches, it still faces limitations in understanding deeper user intent. This is where semantic search offers an advantage over more broad keyword searches.
Benefits and applications of semantic search
Semantic search offers several benefits and applications, including:
- Improved user satisfaction: By delivering more relevant results, semantic search engines can improve user satisfaction and reduce the time spent searching for information.
- Increased conversions: By understanding the intent behind a search query, semantic search engines can deliver more relevant results, increasing the chances of conversion and revenue.
- Enhanced user experience: Semantic search enables users to input vague search queries and get specific results, making the search experience feel more like human interaction.
- Better for business: Understanding user intent can boost sales and customer satisfaction, improving the customer’s relationship with the brand.
Of course, the benefits of this technology range far beyond these few points, but you get the idea!
Enough of the theory, let’s understand how semantic search works through a simple tutorial.
Semantic search tutorial
We understand that semantic search is about understanding the query's context and intent to return the most relevant results, not just matching keywords. To demonstrate this, we can use the sentence-transformers library to create embeddings for a set of documents and a query, and then perform a similarity search to find the most relevant document.
SingleStore Notebooks extends the capabilities of Jupyter Notebook to enable data and AI professionals to easily work and play around.
What is SingleStoreDB?
SingleStoreDB empowers the world’s leading organizations to build and scale modern applications using the only database that allows you to transact, analyze and contextualize data in real time. It offers streaming data ingestion, support for both transactions and analytics, horizontal scalability and hybrid vector search capabilities.
Here is a step-by-step tutorial you can follow in a SingleStore Notebook.
But first, we need to sign up to the free Singlestore Helios account to use the Notebook feature. When you sign up, you will receive $600 in free computing resources.
Once you sign in to your Singlestore Helios account, you’ll see the following dashboard — and where you need to click ‘Notebooks’ as shown.
Then, create a blank Notebook and name it as you wish. I am naming mine ‘semantic-search-demo’.
Once you create your Notebook, you will be presented with a dashboard where you can add code snippets and start working.
Follow along the tutorial and make sure you add the code shown in the next steps into the Notebooks and run it every time. Let’s get started!
Step 1. Install the necessary libraries
First, you need to install the sentence-transformers library. Run this in a Jupyter Notebook cell:
1
!pip install sentence-transformers
For the first time, let me show you how to add the preceding command into your Notebook and run it:
Now, you should understand how to add the code into the Notebook and run it every time. You’ll do the same for the following commands and code snippets.
Step 2. Import the libraries
1
from sentence_transformers import SentenceTransformer, util2
import numpy as np
Step 3. Load the pre-trained model
We will use a pre-trained model from the sentence-transformers library. This model is trained to generate embeddings that are useful for semantic similarity tasks.
1
model = SentenceTransformer('all-MiniLM-L6-v2')
Step 4. Define your documents and query
Define some documents and a query. The documents can be sentences, paragraphs or longer blocks of texts.
1
# Example documents2
documents = [3
"The quick brown fox jumps over the lazy dog.",4
"I had a great time at the park with my friends.",5
"The economy is showing signs of recovery after the pandemic.",6
"The surface of Mars is red due to iron oxide.",7
"Machine learning models have become very sophisticated."8
]9
10
# Example query11
query = "Natural language processing models"
Step 5. Encode the documents and the query
We will create embeddings for both our documents and the query.
1
# Encode the documents2
document_embeddings = model.encode(documents)3
4
# Encode the query5
query_embedding = model.encode(query)
Step 6. Perform semantic search
Now, we will use cosine similarity to find the most semantically similar document to the query.
1
# Compute similarity scores of the query against all document embeddings2
similarity_scores = util.pytorch_cos_sim(3
query_embedding,4
document_embeddings5
)6
7
# Find the index of the highest score8
highest_score_index = np.argmax(similarity_scores)9
10
print("The most semantically similar document to the query:")11
print(documents[highest_score_index])
The output will be the document from your predefined list that has the highest cosine similarity score with your query "natural language processing models". This score is a numerical representation of how similar the document is to the query in the context of the embedding space created by the sentence-transformers model.
Here's what the output might look like:
In this example, the model has determined that the document discussing machine learning models is most semantically similar to your query about natural language processing models. This is because both sentences are related to the field of AI and the underlying concepts of models and learning, even though the exact words from the query may not be present in the document.
You can enhance the output to provide more information or format it differently according to your needs. Here are some suggestions:
Create a DataFrame display. If you prefer a table format, you can use pandas to create a DataFrame that shows documents and their similarity scores.
1
import pandas as pd2
3
# Create a DataFrame for better visualization4
scores = similarity_scores[0].tolist()[0]5
df = pd.DataFrame({'Document': documents, 'Similarity Score': scores})6
7
# Sort the DataFrame based on similarity scores8
df = df.sort_values(by='Similarity Score', ascending=False)9
10
print(df)
You can see the output here:
Visualize similarity scores. You can create a bar chart to visualize the similarity scores for each document.
1
import matplotlib.pyplot as plt2
3
# Plot the similarity scores4
plt.bar(range(len(documents)), similarity_scores[0].tolist()[0])5
plt.xticks(range(len(documents)), range(1, len(documents)+1))6
plt.xlabel('Document Number')7
plt.ylabel('Similarity Score')8
plt.title('Semantic Similarity Scores')9
plt.show()
See the output here:
There is much more you can do with these Notebooks to understand the concepts clearly. Check out all available tutorials in SingleStore Spaces.
SingleStoreDB as a semantic search engine
SingleStoreDB aids in semantic search by enabling the storage and querying of high-dimensional vector data within its distributed SQL database system. With its patented Universal Storage, SingleStoreDB is optimized for both OLTP and OLAP workloads — crucial for modern semantic search platforms that require fast transactional and analytical processing.
Developers can store vector embeddings directly in the database using binary or blob columns and employ built-in functions for efficient vector operations, including similarity matching through dot_product, to perform semantic queries.
SingleStoreDB's architecture is designed to handle large-scale vector similarity workloads with ease, utilizing its distributed nature, parallelization and Intel SIMD-based vector processing for rapid retrieval and real-time analytics. This means SingleStoreDB powers applications to perform semantic search that understands the context and nuance of user queries, delivering precise and relevant results at speed.
Conclusion: Achieving relevant search results
Semantic search represents a monumental leap in how we interact with the digital world. By prioritizing intent and meaning over mere keywords, it redefines the boundaries of user engagement and information retrieval. As this technology continues to evolve, it will not only refine the accuracy of search results but also revolutionize the way we navigate and utilize the burgeoning universe of online content.
The future of search is undeniably semantic — promising a more intuitive, efficient and contextually aware landscape for users to explore the depths of human knowledge with unprecedented ease.
![[r]evolution Summer 2022: Distributed SQL “Workspaces” Power Modern Applications](https://images.contentstack.io/v3/assets/bltac01ee6daa3a1e14/blte4a5e6fc2c34627b/6433d905fbe24d10f96bc1a7/featured_distributed-sql-workspaces-power-modern-applications.png?height=224&disable=upscale&auto=webp)