Build AI Chatbots with Open-Source LLMs, LangChain, and Streamlit/

...

/

Retrieval Strategies: Retrievers

Retrieval Strategies: Retrievers

Learn how to retrieve the relevant data chunks from the vector stores.

We'll cover the following...

Let’s start by integrating the LangChain retrievers. For all the exercises below, we will define the user query as “query = Who is Alan Turing?”

Vector store-backed retriever

A vector store retriever employs vector storage to fetch documents through similarity searches, wrapping the vector store’s capabilities into a retriever interface for seamless query processing.

Press + to interact
# Define a retriever and retrieve the relevant documents
retriever = db.as_retriever(search_kwargs={"k": 5})
docs = retriever.invoke(query)
# Print the output
print('Relevant documents:')
print('-'*80)
for idx, doc in enumerate(docs):
    print(f'Relevant text {idx+1}: {doc.page_content}')
    print('-'*80)

In this code, we perform the following steps:

  • Lines 1–2: We initialize a retriever object from the existing vector store (db) with specific search parameters where k=5 which means that the top 5 most similar chunks will be retrieved.

  • Line 3: We execute the retrieval for the user query and get the relevant documents.

  • Lines 5–10: We print the retrieved documents and the relevant content from each.

The output will be as follows:

Relevant documents:
--------------------------------------------------------------------------------
Relevant text 1: Turing was a British mathematician, logician, and computer scie
--------------------------------------------------------------------------------
Relevant text 2: advancement.

Alan Turing's contributions to science and technolo
--------------------------------------------------------------------------------
Relevant text 3: work in breaking the Enigma code during World War II. Turing's
--------------------------------------------------------------------------------
Relevant text 4: father of modern computer science and is best known for his work
--------------------------------------------------------------------------------
Relevant text 5: intelligence.

The foundational work of Alan Turing, who delved in
--------------------------------------------------------------------------------

The retriever retrieves the sentences that are the most relevant to the query.

Multi-Query Retriever

MultiQueryRetriever deals with the nuances of query wording and semantic intricacies by generating multiple query perspectives with an LLM. It automates the process of prompt tuning by using an LLM to generate multiple queries from different perspectives for a given user input query. For each query, it retrieves a set of relevant chunks and takes the unique union across all queries to get a larger set of ...