New

A Deep Dive Into Vector Databases

Notebook

SingleStore Notebooks

A Deep Dive Into Vector Databases

Required Installations

In [1]:

1
!pip install openai numpy pandas singlestoredb langchain==0.1.8 langchain-community==0.0.21 langchain-core==0.1.25 langchain-openai==0.0.6

Vector Embedding Example

In this example, we demonstrate a rule based system that generates vector embeddings based on a word. The embedding that we generate contains 5 main features:

Length of word
Number of vowels in the word (normalized to the length of the word)
Whether the word starts with a vowel (1) or not (0)
Whether the word ends with a vowel (1) or not (0)
Percentage of consonants in the word

This is a simple implementation of a rule based system to demonstrate the essence of what vector embedding models do. However, they utlize neural networks that are trained on vast datasets to learn key features and self-corrects using gradient descent.

In [2]:

1
def word_to_vector(word):
2
    # Define some basic rules for our vector components
3
    vector = [0] * 5  # Initialize a vector of 5 dimensions
4

5
    # Rule 1: Length of the word (normalized to a max of 10 characters for simplicity)
6
    vector[0] = len(word) / 10
7

8
    # Rule 2: Number of vowels in the word (normalized to the length of the word)
9
    vowels = 'aeiou'
10
    vector[1] = sum(1 for char in word if char in vowels) / len(word)
11

12
    # Rule 3: Whether the word starts with a vowel (1) or not (0)
13
    vector[2] = 1 if word[0] in vowels else 0
14

15
    # Rule 4: Whether the word ends with a vowel (1) or not (0)
16
    vector[3] = 1 if word[-1] in vowels else 0
17

18
    # Rule 5: Percentage of consonants in the word
19
    vector[4] = sum(1 for char in word if char not in vowels and char.isalpha()) / len(word)
20

21
    return vector
22

23
# Example usage
24
word = "example"
25
vector = word_to_vector(word)
26
print(f"Word: {word}\nVector: {vector}")

Vector Similarity Example

In this example, we demonstrate a way to determine the similarity between two vectors. There are many techniques to find the similiarity between two vectors but one of the most popular ways is using cosine similarity. Consine similarity is the the dot product between the two vectors divided by the product of the vector's normals (magnitudes).

This is just an example to show how vector databases search for similar vectors. The fundamental problem with a system like this is our rule-based embedding because it does not give us a semantic understanding of the word/sentences/paragraphs. Instead, it gives us a classification of a single word's structure.

In [3]:

1
import numpy as np
2

3
def cosine_similarity(vector_a, vector_b):
4
    # Calculate the dot product of vectors
5
    dot_product = np.dot(vector_a, vector_b)
6
    # Calculate the norm (magnitude) of each vector
7
    norm_a = np.linalg.norm(vector_a)
8
    norm_b = np.linalg.norm(vector_b)
9
    # Calculate cosine similarity
10
    similarity = dot_product / (norm_a * norm_b)
11
    return similarity
12

13
# Example usage
14
word1 = "example"
15
word2 = "sample"
16
vector1 = word_to_vector(word1)
17
vector2 = word_to_vector(word2)
18

19
# Calculate and print cosine similarity
20
similarity_score = cosine_similarity(vector1, vector2)
21
print(f"Cosine similarity between '{word1}' and '{word2}': {similarity_score}")

Embedding Models

In order to generate semantic understanding of language within vectors, embedding models are required. Embedding models are trained on vast corpus of language data. Training embedding models starts by initializing word embeddings with random vectors. Each word in the vocabulary is assigned a vector of real numbers. They use neural networks trained on large datasets to predict a word from its context (Continuous Bag of Words model) or to predict the context given a word (Skip-Gram model). During training, the model adjusts the word vectors to minimize some loss function, often related to the likelihood of observing a word given its context (or vice versa) through gradient descent.

Examples of embedding models include Word2Vec, GloVe, BERT, OpenAI text-embedding.

In [4]:

1
OPENAI_KEY = "INSERT OPENAI KEY"
2

3
from openai import OpenAI
4
client = OpenAI(api_key=OPENAI_KEY)
5

6
def openAIEmbeddings(input):
7
  response = client.embeddings.create(
8
      input="input",
9
      model="text-embedding-3-small"
10
  )
11
  return response.data[0].embedding
12

13
print(openAIEmbeddings("Golden Retreiver"))

As you can see, this is a huge vector! Over 1000 dimensions just in this one vector. This is why it is important for us to have good dimensionality reduction techniques during the similarity searches.

Creating a vector database with SingleStoreDB

In the following code we create a vector datbase with SingleStoreDB. We utilize Langchain to chunk and split the raw text into documents and use the OpenAI embeddings model to generate the vector embeddings. We then take the raw documents and embeddings and create a table with the columns "docs" and "embeddings".

To test this out, we perform a similarity search based on a query and it returns the most similar document in the vector database.

In [5]:

1
import openai
2
from langchain.text_splitter import CharacterTextSplitter
3
from langchain_community.document_loaders import TextLoader
4
from langchain_community.embeddings import OpenAIEmbeddings
5
from langchain_community.vectorstores.singlestoredb import SingleStoreDB
6
from openai import OpenAI
7
import os
8
import pandas as pd
9

10

11
# Load and process documents
12
loader = TextLoader("michael_jackson.txt") # use your own document
13

14
documents = loader.load()
15
text_splitter = CharacterTextSplitter(chunk_size=2000, chunk_overlap=0)
16
docs = text_splitter.split_documents(documents)
17

18
# Generate embeddings and create a document search database
19
embeddings = OpenAIEmbeddings(api_key=OPENAI_KEY)
20

21
# Create Vector Database
22
vector_database = SingleStoreDB.from_documents(docs, embeddings, table_name="mjackson") # create your own table
23

24
query = "How old was Michael Jackson when he died?"
25
docs = vector_database.similarity_search(query)
26
print(docs[0].page_content)

Retrieval Augmented Generation System

RAG combines large language models with a retrieval mechanism to search a database for relevant information before generating responses. It utilizes real-world data from retrieved documents to ground responses, enhancing factual accuracy and reducing hallucinations. Documents are vectorized using embeddings and stored in a vector database for efficient retrieval. SingleStoreDB serves as a great vector database. The user query is converted into a vector, and a vector search is performed in the database to find documents relevant to that specific query. The system returns the documents with the highest relevance scores, which are then fed to the chatbot for generating informed responses.

In [6]:

1
import os
2
import openai
3
from langchain.text_splitter import CharacterTextSplitter
4
from langchain_community.document_loaders import TextLoader
5
from langchain_community.embeddings import OpenAIEmbeddings
6
from langchain_community.vectorstores.singlestoredb import SingleStoreDB
7
from openai import OpenAI
8

9
# Set up API keys and database URL
10
client = OpenAI(api_key=OPENAI_KEY)
11

12
# Load and process documents
13
loader = TextLoader("michael_jackson.txt")
14
documents = loader.load()
15
text_splitter = CharacterTextSplitter(chunk_size=2000, chunk_overlap=0)
16
docs = text_splitter.split_documents(documents)
17

18
# Generate embeddings and create a document search database
19
embeddings = OpenAIEmbeddings(OPENAI_KEY)
20
docsearch = SingleStoreDB.from_documents(docs, embeddings, table_name="mjackson")
21

22
# Chat loop
23
while True:
24
    # Get user input
25
    user_query = input("\nYou: ")
26

27
    # Check for exit command
28
    if user_query.lower() in ['quit', 'exit']:
29
        print("Exiting chatbot.")
30
        break
31

32
    # Perform similarity search
33
    docs = docsearch.similarity_search(user_query)
34
    if docs:
35
        context = docs[0].page_content
36

37
        # Generate response using OpenAI GPT-4
38
        response = client.chat.completions.create(
39
            model="gpt-4",
40
            messages=[
41
                {"role": "system", "content": "Context: " + context},
42
                {"role": "user", "content": user_query}
43
            ],
44
            stream=True,
45
            max_tokens=500,
46
        )
47

48
        # Output the response
49
        print("AI: ", end="")
50
        for chunk in response:
51
            if chunk.choices[0].delta.content is not None:
52
                print(chunk.choices[0].delta.content, end="")
53

54
    else:
55
        print("AI: Sorry, I couldn't find relevant information.")

Details

About this Template

Using SingleStoreDB as a vector database and vector database use cases.

This Notebook can be run in Standard and Enterprise deployments.

License

This Notebook has been released under the Apache 2.0 open source license.

See Notebook in action

Launch this notebook in SingleStore and start executing queries instantly.

A Deep Dive Into Vector Databases

Notebook

A Deep Dive Into Vector Databases

Vector Embedding Example

Vector Similarity Example

Embedding Models

Creating a vector database with SingleStoreDB

Retrieval Augmented Generation System

Details

About this Template

Tags

License

See Notebook in action