🔗 Building a RAG-Enabled Assistant with OpenWebUI: Integrating Local and Cloud LLMs via API

Date Created: 2025-05-05
By: 16BitMiker
[ BACK.. ]

OpenWebUI offers a powerful and intuitive interface for interacting with large language models (LLMs), whether you're running them locally via Ollama or accessing cloud-based services like OpenAI. But what really sets it apart is its extensibility: you’re not limited to the browser UI. OpenWebUI provides a robust API for customizing assistants, integrating tools, and enabling Retrieval-Augmented Generation (RAG).

In this post, we’ll walk through how to build a RAG-enabled assistant in OpenWebUI using both local and cloud models. Then, we’ll show how to access it programmatically using the OpenAI-compatible API.

Let’s dive in. 🏊

🧠 What Is a Custom Assistant?

In OpenWebUI, a “custom assistant” is a user-defined configuration that wraps around a base model. Think of it as an intelligent persona with memory, tools, and specific knowledge sources.

Each assistant can include:

• Base Model: Local (e.g., LLaMA 3 via Ollama) or remote (e.g., OpenAI’s GPT-4).

• System Prompt: Defines tone, behavior, and role.

• Knowledge Bases: Enable RAG by adding relevant documents.

• Tool Integrations: Access web search, code execution, or custom plugins.

• Model Parameters: Control creativity, response length, and repetition.

• Prompt Suggestions: Offer pre-defined queries to streamline UX.

Once defined, the assistant can be used through the UI or via API.

📋 Step 1: Choose Your Model Backend

OpenWebUI is backend-agnostic — you can connect it to local or cloud-hosted models.

🖥️ Local with Ollama

Ollama makes it easy to run models like LLaMA 3, Mistral, or Qwen entirely on your machine.

✅ To get started:

# Install Ollama from https://ollama.com/download
ollama run llama3

OpenWebUI will auto-detect your local Ollama instance (typically available at http://localhost:11434) and make it available as a model backend.

🏔️ Ideal for: Air-gapped environments, privacy-conscious deployments, and tinkering.

☁️ Cloud with OpenAI, Azure, or Custom APIs

If you want access to high-performance models like GPT-4 or Claude, you can connect OpenWebUI to any OpenAI-compatible API.

✅ Example: Set up OpenAI

1. Go to Workspace > Models

2. Click + Create Model

3. Choose “OpenAI-Compatible”

4. Enter your endpoint (e.g., https://api.openai.com/v1) and your API key

OpenWebUI will treat this like any other backend, standardizing interactions.

📦 This same approach works with self-hosted APIs like vLLM, LM Studio, or FastChat.

📚 Step 2: Upload Documents for RAG

RAG enhances your assistant by grounding its answers in your documents. OpenWebUI embeds these documents and indexes them for semantic retrieval.

✅ To add a knowledge base:

1. Go to Workspace > Knowledge

2. Click + Create Knowledge Base

3. Give it a name and description

4. Upload PDFs, Markdown, or plain text files

5. OpenWebUI will process them into embeddings using its vector store backend

🔍 These documents will be automatically referenced by the assistant during conversations.

🧰 Step 3: Create and Configure a Custom Assistant (Model)

Now that your model backend and knowledge base are ready, let’s create a custom assistant.

1. Navigate to Workspace > Models

2. Click + Create a model

🧩 Configuration Options:

• Model Name: Descriptive name for the assistant (e.g., HR-Bot, Legal Counsel)

• Base Model: Choose the model backend (e.g., llama3, gpt-4)

• System Prompt: Define role and behavior (e.g., “You are a helpful HR assistant...”)

• Avatar Image: Optional image for UI

• Prompt Suggestions: Preload useful queries like “What is our remote work policy?”

🔗 Enable RAG:

• Under the “Knowledge” tab, link the knowledge base you created earlier.

🔌 Enable Tools (Optional):

• Add tools like:

  • Web search

  • Code execution

  • Custom API calls

🎛️ Advanced Settings:

• Temperature: Controls creativity (0.0 = deterministic, 1.0 = creative)

• Max Tokens: Limit response length

• Frequency Penalty: Discourage repetition

✅ Save your assistant. It’s now available in OpenWebUI and via API.

🔑 Step 4: Generate an API Key

To call your assistant from code, you’ll need an access token.

1. Go to Settings > Account > API Keys

2. Click ➕ to generate a key

3. Copy and store it securely

🔐 This API key will be used as a Bearer token in your HTTP requests.

▶️ Step 5: Call the Assistant via API

OpenWebUI exposes an OpenAI-style endpoint:

xxxxxxxxxx
curl -X POST http://localhost:3000/api/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "your-custom-model-id",
        "messages": [
          {
            "role": "user",
            "content": "What is our VPN access policy?"
          }
        ]
      }'

📌 Notes:

• Replace "your-custom-model-id" with the assistant’s ID

• This endpoint is compatible with OpenAI clients and SDKs

• Assistant settings (prompt, tools, knowledge) are applied automatically

🔄 Optional: Streaming Responses

For real-time interaction (e.g., live chat):

xxxxxxxxxx
{
  "model": "your-custom-model-id",
  "messages": [...],
  "stream": true
}

Use a client that supports Server-Sent Events (SSE) to process streamed tokens.

📈 Great for dashboards, bots, and typing animations.

🛠️ Example: Use OpenWebUI in Python

Here’s a simple way to query your assistant from Python:

​x
import requests

# Your OpenWebUI API key and assistant model ID
API_KEY = "your-api-key"
MODEL_ID = "your-custom-model-id"

# HTTP headers including authorization
headers = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
}

# The payload to send to the assistant
payload = {
    "model": MODEL_ID,
    "messages": [
        {"role": "user", "content": "Explain our password policy."}
    ]
}

# Make the POST request to OpenWebUI's chat completions endpoint
response = requests.post("http://localhost:3000/api/chat/completions", headers=headers, json=payload)

# Output the assistant's response
print(response.json())

📍 This works whether your assistant is backed by a local model (via Ollama) or a cloud model (like OpenAI's GPT).

🗂️ Use Cases

OpenWebUI + API opens up a wide range of possibilities:

👥 Internal Tools:

• HR assistant with policy documents

• Legal bot with contract templates

• IT helpdesk with workflow guides

🔌 System Integrations:

• Slack bots

• Web dashboards

• CLI tools

🏔️ Offline-First:

• Fully air-gapped deployments using Ollama

• No cloud dependency

🔚 Conclusion

OpenWebUI transforms how you interact with LLMs—whether through its sleek UI or programmable API. By combining local and cloud models, integrating RAG, and enabling tool use, it becomes a powerful development environment for intelligent assistants.

With just a few steps, you can create domain-specific assistants that are fast, private, and deeply integrated into your workflows.

✅ Whether you're building a chatbot, automating support, or querying internal documents, OpenWebUI provides the flexibility and control you need.

📚 Read More

• OpenWebUI Docs – RAG

• RAG Tutorial

• OpenWebUI Assistant Examples

• Reddit: Use Tools via API

Happy building! 🧠💻