Ollama guide: Building local RAG chatbots with LangChain

In the rapidly evolving world of artificial intelligence (AI) and natural language processing (NLP), Ollama has emerged as a game-changer for developers and enthusiasts looking to run large language models (LLMs) locally. By enabling the deployment of LLMs on personal computers, Ollama offers significant advantages such as enhanced privacy, cost-efficiency, and reduced latency. This powerful, open-source tool simplifies the process of downloading, running, and managing LLMs, making advanced AI capabilities more accessible than ever before.

This blog explores Ollama’s features, functionalities, and potential impact. It explains what Ollama offers and how to use it to build a Retrieval-Augmented Generation (RAG) chatbot using Streamlit.

What is Ollama?#

Ollama is an open-source project allowing users to run LLMs locally on their machines. It provides a simple command-line interface for downloading, running, and managing various LLMs, including popular models like Llama 3, Mistral, Gemma 2, and LLaVA.

Key features of Ollama#

The following are some key features of Ollama:

• Easy installation: Ollama can be installed with a single command on macOS and Linux systems.

• Wide model support: It supports a variety of models, from smaller, faster options to larger, more capable ones. Here, you will find the complete list of models supported by Ollama.

• Custom model creation: Users can create and share their own custom models using Modelfiles.

• API access: Ollama provides a RESTful API, allowing integration with other applications and services.

• Efficient resource management: It optimizes resource usage, making it possible to run models on consumer-grade hardware.

• Cross-platform compatibility: While primarily designed for macOS and Linux, there’s growing support for Windows users as well.

Getting started with Ollama#

Ollama is a powerful tool designed for efficiently running LLMs on your local machine. Whether you’re a developer looking to integrate AI capabilities into your application or someone interested in experimenting with language models, Ollama provides a user-friendly experience.

We’ll walk you through the installation process, running models, managing them, and even creating custom models tailored to your needs. Let’s dive in!

Installation#

Ollama is now available for Windows, macOS, and Linux! You can download it from the Ollama website.

Running a model#

Once installed, you can run a model with a simple command:

This downloads and runs the Llama 3.1 model. You can replace llama3.1 with any other supported model name.

Model management#

Ollama provides commands for listing, removing, and updating models:

• List models: ollama list

• Remove a model: ollama rm modelname

• Update models: ollama pull modelname

Creating custom models#

One of Ollama’s most powerful features is the ability to create custom models using Modelfiles. These are similar to Dockerfiles and allow you to define a model’s base, add extra data, and set various parameters.

Here’s a simple example of a Modelfile:

Let’s break down this Modelfile:

• FROM llama3.1: This specifies that we’re using the Llama 3.1 model as our base. Llama 3.1 is a powerful language model that can handle complex queries and provide detailed responses.

• PARAMETER temperature 0.7: This sets the temperature parameter to 0.7, which balances coherence and creativity. It allows the AI to provide varied responses while maintaining accuracy, which is crucial for technical assistance.

• SYSTEM "...": This block defines the AI assistant’s role and behavior. It instructs the AI to:

  • Specialize in Ollama-related information

  • Assist with various aspects of Ollama (installation, usage, model management, troubleshooting)

  • Provide accurate and concise information

  • Acknowledge when unsure and guide users to other resources

  • Prioritize up-to-date information about Ollama

To use this Modelfile with Ollama:

1. Save the Modelfile in a text file named Modelfile (without any file extension).

2. Open a terminal and navigate to the directory containing the Modelfile.

3. Run the following command to create the custom Ollama assistant model: ollama create ollama-assistant -f Modelfile

4. Once created, you can run the Ollama AI assistant using: ollama run ollama-assistant

This will start an interactive session where you can ask questions and get assistance related to Ollama. The AI will respond with helpful information about Ollama, its features, usage, and any other relevant topics.

Ollama API#

Ollama provides a powerful RESTful API that allows developers to directly integrate LLM capabilities into their applications. This API opens up a world of possibilities for creating AI-powered features without the need for complex setups or cloud services.

Understanding RESTful APIs#

Before diving into the Ollama API, let’s briefly explain what a RESTful API is:

• REST stands for Representational State Transfer.

• It’s an architectural style for designing networked applications.

• RESTful APIs use HTTP requests to perform CRUD (create, read, update, delete) operations on resources.

• They typically use JSON for data formatting.

Ollama API overview#

The Ollama API provides several endpoints for different functionalities:

• Generating text

• Managing models (listing, creating, deleting)

• Embedding generation

• Model information retrieval

For this example, we’ll focus on the text generation endpoint.

• Lines 1–2: We import the requests library, a popular Python package for making HTTP requests, and the json library, which provides methods for parsing and handling JSON data.

• Lines 4–7: We send a POST request to the specified URL:

  • URL: 'http://localhost:11434/api/generate'

    • localhost: This indicates that the API is running on our local machine.

    • 11434: This is the default port for the Ollama API.

    • /api/generate: This is endpoint for text generation.

  • json={}: This send data in JSON format in the request body.

    • 'model': 'llama3.1': This specifies the model to use for generation.

    • 'prompt': 'Why is the sky blue?': This is the input text for the model to respond to.

• Lines 10–11: We output the raw text of the response for debugging purposes.

• Lines 14–15: We retrieve and print the content type from the response headers.

• Line 18: We break the response text into individual lines for processing.

• Line 21: We extract the "response" key from each line containing the generated text.

• Line 24: We join the extracted responses into a single string.

• Line 26: We output the complete generated text to the console.

Running the example#

To run this example:

• Ensure Ollama is installed and running on your machine.

• Make sure you have the requests library installed (pip install requests).

• Save the code in a Python file (e.g., ollama_api_example.py).

• Run the script using python3 ollama_api_example.py.

Building a local RAG-based chatbot with Streamlit and Ollama#

Let’s create an advanced Retrieval-Augmented Generation (RAG) based chatbot using Streamlit, Ollama, and other powerful libraries. For instance, a customer service team can deploy this chatbot to handle frequently asked questions by accessing and referencing internal documents such as FAQs, product manuals, and support guides. By doing so, the chatbot ensures that customers receive accurate and consistent responses quickly, reducing the workload on human agents and improving overall response times.

Let’s break down the process step by step.

Step 1: Setting up the environment#

Before we begin coding, we must ensure that our development environment has all the necessary libraries. These libraries will enable us to build our chatbot with advanced natural language processing capabilities.

First, let’s install the required packages:

This command installs Streamlit for our web interface, PyPDF2 for PDF processing, LangChain for our language model interactions, Pillow for image processing, and PyMuPDF for PDF rendering.

Make sure you pull the Llama 3.1 and NOMICnomic-embed-text is a powerful model that converts text into numerical representations (embeddings) for tasks like search, clustering, and classification. It excels at handling long text passages and outperforms similar models. models before moving to the next step using the following commands:

Step 2: Importing the required libraries#

Now that our environment is set up, let’s start by importing all the necessary libraries. These imports will give us access to the tools we need for building our chatbot.

• Line 1: We import the streamlit library for creating web applications aliased as st.

• Line 2: We import PyPDF2, a library for reading and manipulating PDF files.

• Line 3: We import OllamaEmbeddings from LangChain’s community embeddings module.

• Line 4: We immport RecursiveCharacterTextSplitter from LangChain for splitting text into smaller chunks.

• Line 5: We immport Chroma, a vector store from LangChain’s community module, to efficiently manage and query vector embeddings.

• Line 6: We import ConversationalRetrievalChain from LangChain for creating a conversational AI chain.

• Line 7: We import ChatOllama, a chat model from LangChain’s community module for generating conversational responses.

• Line 8: We import the HumanMessage and AIMessage classes from LangChain’s schema to structure and handle chat interactions between users and AI.

• Line 9: We import ChatMessageHistory from LangChain’s community module for storing chat history.

• Line 10: We import ConversationBufferMemory from LangChain for maintaining conversation context.

• Line 11: We import the Image module from PIL (Python Imaging Library) for image processing.

• Line 12: We import fitz from PyMuPDF, a library for working with PDF documents.

Step 3: Configuring the Streamlit page#

Let’s set up our Streamlit page with a custom configuration. This will give our chatbot a professional look and feel.

The above code configures the Streamlit page with a custom title "Ollama RAG Chatbot", sets a robot emoji as the page icon, and uses a wide layout for better space utilization.

Step 4: Creating the sidebar for PDF upload and preview#

Next, we’ll create a sidebar for PDF upload and preview functionality. This allows users to easily upload documents and view their content.

• Lines 1–3: We create a sidebar with a title "PDF Upload" and add a file uploader specifically for PDF files.

• Lines 5–6: We display a success message when a PDF is uploaded.

• Lines 9–12: We add a "PDF Preview" section in the sidebar. We open the uploaded PDF using PyMuPDF (fitz) and create a number input for page selection.

• Lines 13–15: We load the selected page, render it as a pixmap with 2x scaling, and convert it to a PIL Image object.

• Line 17: We display the rendered page image in the sidebar with a caption showing the page number.

Step 5: Setting up the main content area#

Now, let’s set up the main content area of our chatbot interface. This is where we’ll display the chat history and input field.

• Line 1: We display a large, bold title at the top of the main Streamlit app area, introducing the chatbot.

• Line 3: We check if a 'chain' key exists in Streamlit’s session state. This is likely used to store the conversation chain.

• Line 4: If 'chain' doesn’t exist in the session state, it’s initialized to None.

• Line 6: We check if a 'chat_history' key exists in the session state. This is used to maintain conversation history across reruns.

• Line 7: If 'chat_history' doesn’t exist, it’s initialized as an empty list to store conversation messages.

Step 6: Implementing PDF processing#

To work with the uploaded PDF, we need a function to extract its text content. Here’s how we can do that:

• Lines 1–6: This function processes a PDF file, extracting text from all pages and combining it into a single string. It uses PyPDF2 to read the PDF, iterate through each page, extract the text, and concatenate it. The function returns the entire PDF text content as a single string.

Step 7: Setting up the RAG pipeline#

The heart of our chatbot is the RAG pipeline. This system processes the PDF content, creates embeddings, and sets up the conversational chain.

• Lines 1–2: We check if a PDF file is uploaded and if the conversation chain hasn’t been initialized yet.

• Lines 3–4: We display a spinner while processing the PDF, then extract text from the uploaded PDF using the process_pdf function.

• Lines 6–7: We split the extracted text into smaller chunks using RecursiveCharacterTextSplitter from LangChain.

• Line 9: We create metadata for each text chunk, associating it with a source identifier.

• Lines 11–12: We initialize OllamaEmbeddings with the "nomic-embed-text" model and create a Chroma vector store from the text chunks.

• Lines 14–20: We set up the conversation history and memory components using LangChain’s ChatMessageHistory and ConversationBufferMemory.

• Lines 22–28: We create a ConversationalRetrievalChain using the ChatOllama model, the Chroma vector store as a retriever, and the previously set-up memory. This chain is stored in the Streamlit session state.

• Line 30: We display a success message indicating that the PDF has been processed successfully.

Step 8: Implementing the chat interface#

With our RAG pipeline set up, we can now create a chat interface where users can interact with the AI.

• Lines 1–2: We display a subheader for the chat section and create a text input for user questions.

• Lines 4–6: We check if the user has entered a question, and if the conversation chain hasn’t been initialized (i.e., no PDF uploaded), then we display a warning.

• Lines 7–11: If a chain exists, we use a spinner to indicate processing, then invoke the chain with the user’s question and extract the answer and source documents.

• Lines 13–11: We append the user’s question and the AI’s answer to the chat history in the session state.

Step 9: Displaying chat history#

Finally, display the chat history and source documents for each AI response.

• Lines 1–2: We create a container for displaying the chat history.

• Lines 3–7: We iterate through the chat history in reverse order, displaying human messages with a person emoji and AI messages with a robot emoji using Streamlit’s markdown function.

• Lines 9–12: For AI messages, we create an expandable section to show source documents. Each source is displayed with a snippet of its content (the first 150 characters).

Running the chatbot#

To run your newly created chatbot, save all the code in a file (e.g., rag_chatbot. py) and execute it using the following command:

This will launch a web interface where users can upload a PDF, preview it, and engage in a conversation with the AI about its contents.

Use cases for Ollama#

Here are several use cases for Ollama:

• Local development: Test and prototype AI applications without relying on cloud services.

• Privacy-focused applications: Run AI models locally to ensure data privacy.

• Educational tools: Learn about and experiment with LLMs in a controlled environment.

• Offline AI capabilities: Develop applications that can function without internet connectivity.

• Custom assistants: Create specialized AI assistants for specific domains or tasks.

Limitations and considerations#

While Ollama is powerful, it’s important to note some limitations:

• Hardware requirements: Running large models locally, such as Llama 3.1 405B, requires significant computational resources.

• Model availability: Not all state-of-the-art models are available or optimized for Ollama, like Google’s PaLM 2 model, since it’s not in Ollama’s library.

• Continuous updates: Keeping open-source models up-to-date requires consistent effort. To ensure optimal performance, new versions and improvements must be integrated into the Ollama environment.

The future of Ollama#

As the field of AI continues to advance, tools like Ollama are likely to play an increasingly important role in democratizing access to powerful language models. Future developments may include support for more models, improved performance optimizations, and enhanced integration capabilities.

Conclusion#

Ollama represents a significant step forward in making LLMs accessible to developers and enthusiasts. By simplifying the process of running these models locally, it opens up new possibilities for AI application development, research, and education. As we’ve seen with our RAG-based chatbot example, Ollama can be easily integrated into practical applications, allowing for the creation of powerful, privacy-preserving AI tools. As the field continues to evolve, tools like Ollama will undoubtedly play a crucial role in shaping the future of AI development and deployment.

Next steps#

To further enhance your understanding of RAG and its applications, consider exploring the following resources and projects:

Explore these hands-on projects to enrich your understanding of RAG:

• Build an LLM-Powered Wikipedia Chat Assistant with RAG

• Build an Interactive PDF Reader Using LangChain and Streamlit

• Building a Retrieval-Augmented Generation System Using FastAPI