RAG Configuration Guide

This document explains how to configure and customize your RAG pipeline. You will:

Initialize a vector store
Download and point to a local embedding model
Configure an inference provider (LLM)
Choose a RAG strategy (Inline RAG or Tool RAG)

Introduction
Prerequisites
- Set Up the Vector Database
- Download an Embedding Model
Configure BYOK Knowledge Sources
Configure Dynamic Vector Store Providers
Add an Inference Model (LLM)
Complete Configuration Reference
System Prompt Guidance for RAG (as a tool)
RAG annotations
References

Introduction

Lightspeed Core Stack (LCS) supports two complementary RAG strategies:

Inline RAG: context is fetched from BYOK vector stores and/or OKP and injected before the LLM request. No tool calls are required.
Tool RAG: the LLM can call the file_search tool during generation to retrieve context on demand from BYOK vector stores and/or OKP.

Both strategies can be enabled independently via the rag section of lightspeed-stack.yaml. See BYOK Feature Documentation for configuration details.

For runtime-created vector stores (POST /v1/vector-stores), configure vector_store instead of rag.byok.stores. BYOK registers static corpora with a fixed vector_db_id; dynamic providers only declare capacity (provider id, storage, default embeddings).

Inline RAG chunk flow

flowchart TD
    subgraph Sources["Source Fetching"]
        B1["BYOK Store 1"] --> BPool
        B2["BYOK Store 2"] --> BPool
        BN["BYOK Store N"] --> BPool
        BPool["BYOK Pool\ncapped at rag.byok.max_chunks"]
        OKP["OKP (Solr)\ncapped at rag.okp.max_chunks"]
    end

    BPool --> Pool["Merged Pool\n(all chunks, sorted by score)"]
    OKP --> Pool

    Pool --> Decision{Reranker\nenabled?}

    Decision -->|Yes| Rerank["Cross-Encoder Rerank\n+ BYOK score boost"]
    Decision -->|No| Cut

    Rerank --> Cut["Top K cut\nrag.retrieval.inline.max_chunks"]

    Cut --> Context["Final Inline RAG Context"]

Each BYOK store is queried in parallel, and the merged BYOK results are capped at rag.byok.max_chunks total. OKP fetches up to rag.okp.max_chunks. Together these form the reranking pool. If the reranker is enabled, the full pool is reranked with a cross-encoder and BYOK score boosts are applied. The result is capped at rag.retrieval.inline.max_chunks.

The Embedding Model is used to convert queries and documents into vector representations for similarity matching.

[!NOTE] The same Embedding Model should be used to both create the vector store and to query it.

Prerequisites

Set Up the Vector Database

Use the rag-content repository to build a compatible vector database.

[!IMPORTANT] The resulting DB must be in a supported format (e.g., FAISS with SQLite metadata). This can be configured when using the tool to generate the index.

Download an Embedding Model

Download a local embedding model such as sentence-transformers/all-mpnet-base-v2 by using the script in rag-content or manually download and place in your desired path.

[!NOTE] The embedding model can also be downloaded automatically at first start-up (which will be slower). In the rag.byok.stores section of lightspeed-stack.yaml, specify a supported model name as embedding_model instead of a local path. The model will be downloaded to the ~/.cache/huggingface/hub folder.

Configure BYOK Knowledge Sources

BYOK knowledge sources are configured in the rag.byok.stores section of lightspeed-stack.yaml. The required configuration is automatically generated at startup when using make run, make run-stack, docker-compose, or library mode — no manual enrichment is needed.

FAISS example

rag:
  byok:
    stores:
      - rag_id: custom-index
        backend: faiss
        embedding_model: sentence-transformers/all-mpnet-base-v2  # or path to local model
        embedding_dimension: 768
        vector_db_id: vs_8c94967b-81cc-4028-a294-9cfac6fd9ae2                                    # Generated by rag-content during index creation
        db_path: <path-to-vector-index>                            # e.g. /home/USER/vector_db/faiss_store.db

Where:

embedding_model is the embedding model identifier or path to the local model folder
db_path is the path to the vector index (.db file in this case)
vector_db_id is the ID generated by rag-content during index creation (e.g. vs_8c94967b-81cc-4028-a294-9cfac6fd9ae2)

See the full working config example for more details.

pgvector example

This example shows how to configure a remote PostgreSQL database with the pgvector extension for storing embeddings.

You will need to install PostgreSQL with a matching version to pgvector, then log in with psql and enable the extension with:
CREATE EXTENSION IF NOT EXISTS vector;

Each pgvector-backed table follows this schema:

id (text): UUID identifier of the chunk
document (jsonb): json containing content and metadata associated with the embedding
embedding (vector(n)): the embedding vector, where n is the embedding dimension and will match the model’s output size (e.g. 768 for all-mpnet-base-v2)

[!NOTE] The vector_store_id (e.g. rhdocs) is used to point to the table named vector_store_rhdocs in the specified database, which stores the vector embeddings.

rag:
  byok:
    stores:
      - rag_id: pgvector-example
        backend: pgvector
        embedding_model: sentence-transformers/all-mpnet-base-v2
        embedding_dimension: 768
        vector_db_id: rhdocs  # becomes PostgreSQL table 'vector_store_rhdocs'
        host: ${env.POSTGRES_HOST}
        port: ${env.POSTGRES_PORT}
        db: ${env.POSTGRES_DATABASE}
        user: ${env.POSTGRES_USER}
        password: ${env.POSTGRES_PASSWORD}

[!NOTE] Connection fields (host, port, db, user, password) default to ${env.POSTGRES_*} environment variable references when omitted. Use environment variables for credentials.

Configure Dynamic Vector Store Providers

Use vector_store when clients create vector stores at runtime (for example POST /v1/vector-stores flows). This is not BYOK: do not put a static corpus here, and do not use a byok_ provider id prefix.

Requirements:

Supported types: faiss, pgvector
embedding_model and embedding_dimension are required on every provider entry
When providers is non-empty, default_provider is required and must match one of providers[].id
When providers is empty, default_provider must be omitted
Provider id must match [a-z0-9_-]+ and must not start with byok_
Applied in unified Llama Stack synthesis only (llama_stack.use_as_library_client: true with a synthesis input such as llama_stack.config, inference.providers, or vector_store.providers)

default_provider becomes vector_stores.default_provider_id and that provider’s embedding model becomes default_embedding_model in the synthesized Llama Stack config. FAISS entries also get a dedicated storage backend named vsprov_<id>_storage.

FAISS example

vector_store:
  default_provider: example
  providers:
    - id: example
      type: faiss
      embedding_model: /example/embeddings_model
      embedding_dimension: 768
      config:
        path: /example/abc/faiss_store.db

pgvector example

vector_store:
  default_provider: example-pg
  providers:
    - id: example-pg
      type: pgvector
      embedding_model: sentence-transformers/all-mpnet-base-v2
      embedding_dimension: 768
      config:
        # host/port/db/user/password optional; default to ${env.POSTGRES_*}
        host: ${env.POSTGRES_HOST}
        port: ${env.POSTGRES_PORT}
        db: ${env.POSTGRES_DATABASE}
        user: ${env.POSTGRES_USER}
        password: ${env.POSTGRES_PASSWORD}

Field reference is in the configuration schema (VectorStoreConfiguration, FaissVectorStoreProvider, PgvectorVectorStoreProvider).

Add an Inference Model (LLM)

vLLM on RHEL AI (Llama 3.1) example

[!NOTE] The following example assumes that podman’s CDI has been properly configured to enable GPU support.

The vllm-openai Docker image is used to serve the Llama-3.1-8B-Instruct model.
The following example shows how to run it on RHEL AI with podman:

podman run \
  --device "${CONTAINER_DEVICE}" \
  --gpus ${GPUS} \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}" \
  -p ${EXPORTED_PORT}:8000 \
  --ipc=host \
  docker.io/vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json --chat-template examples/tool_chat_template_llama3.1_json.jinja

The example command above enables tool calling for Llama 3.1 models. For other supported models and configuration options, see the vLLM documentation: vLLM: Tool Calling

After starting the container, configure the vLLM provider in your run.yaml, matching model_id with the model provided in the podman run command.

[...]
models:
[...]
- model_id: meta-llama/Llama-3.1-8B-Instruct # Same as the model name in the 'podman run' command
  provider_id: vllm
  model_type: llm
  provider_model_id: null

providers:
  [...]
  inference:
  - provider_id: vllm
    provider_type: remote::vllm
    config:
      url: http://localhost:${env.EXPORTED_PORT:=8000}/v1/ # Replace localhost with the url of the vLLM instance
      api_token: <your-key-here> # if any

OpenAI example

Add a provider for your language model in your run.yaml (e.g., OpenAI):

models:
[...]
- model_id: my-model 
  provider_id: openai
  model_type: llm
  provider_model_id: <model-name> # e.g. gpt-4o-mini

providers:
[...]
  inference:
  - provider_id: openai
    provider_type: remote::openai
    config:
      api_key: ${env.OPENAI_API_KEY}

Make sure to export your API key:

export OPENAI_API_KEY=<your-key-here>

[!NOTE] When experimenting with different models, providers and vector_dbs, you might need to manually unregister the old ones via the CLI.

Azure OpenAI

Not yet supported.

Ollama

The remote::ollama provider does not support tool calling, so RAG as a tool is not available. However, inline RAG is supported.

vLLM Mistral

The RAG tool calls where not working properly when experimenting with mistralai/Mistral-7B-Instruct-v0.3 on vLLM.

OKP/Solr Vector IO

The OKP (Offline Knowledge Portal) Solr Vector IO is a read-only vector search provider that integrates with Apache Solr for enhanced vector search capabilities. It enables retrieving contextual information from Solr-indexed Red Hat documents to enhance query responses with support for hybrid search and chunk window expansion.

How to Enable OKP/Solr Vector IO

1. Configure Lightspeed Stack (lightspeed-stack.yaml):

rag:
  retrieval:
    inline:
      sources:
        - okp               # inject OKP context before the LLM request
    tool:
      sources:
        - okp               # expose OKP as the file_search tool
  okp:
    rhokp_url: ${env.RH_SERVER_OKP}   # OKP base URL (env var or literal URL)
    offline: true         # true = use parent_id for source URLs (offline mode)
                          # false = use reference_url (online mode)

Set rhokp_url to the base URL of your OKP server under rag.okp. Use ${env.RH_SERVER_OKP} to read the URL from the environment; when omitted or empty, a default from the application constants is used.

[!NOTE] When okp is listed in rag.retrieval.inline.sources or rag.retrieval.tool.sources, Lightspeed Stack automatically enriches the underlying configuration at startup with the required vector_io provider and registered_resources entries for the OKP vector store. No manual registration is needed.

Query Request Example:

curl -sX POST http://localhost:8080/v1/query \
    -H "Content-Type: application/json" \
    -d '{"query" : "how do I secure a nodejs application with keycloak?"}' | jq .

Query Processing:

When OKP is enabled, queries use the portal-rag vector store
Vector search is performed with configurable parameters:
- k: Number of results (default: 5)
- score_threshold: Minimum similarity score (default: 0.0)
- mode: Search mode (default: “hybrid”). Per-request configurable.
Results include document metadata and source URLs
Document URLs are built based on the offline setting:
- Offline mode: Uses parent_id with Mimir base URL
- Online mode: Uses reference_url from document metadata

Query Filtering:

To further filter the OKP context, set the chunk_filter_query field in the rag.okp section of lightspeed-stack.yaml. Filters follow the OKP key:value format and are applied as a static fq parameter on every OKP search request.

rag:
  okp:
    rhokp_url: ${env.RH_SERVER_OKP}
    chunk_filter_query: "product:*openshift*"

Per-request filtering is also available on all inference endpoints via request field solr: mode (semantic, hybrid, or lexical) and filters (key:value format). Legacy payloads that omit mode/filters and send filter key:value pairs at the top level still work with mode set to hybrid.

Example:

{
  "query": "How do I configure routes?",
  "solr": {
    "mode": "hybrid",
    "filters": { "fq": ["product:*openshift*"] }
  }
}

Prerequisites:

The OKP server must be running and accessible at the URL given in rag.okp.rhokp_url (or ${env.RH_SERVER_OKP}). For instructions on how to pull and run the OKP image, visit: https://github.com/lightspeed-core/lightspeed-providers/lightspeed_stack_providers/providers/remote/solr_vector_io/solr_vector_io/README.md

Chunk volume:

OKP and BYOK scores are not directly comparable (different scoring systems), so score_multiplier (a BYOK-only concept) does not apply to OKP results. However, when the reranker is enabled, it normalizes scores across sources using a cross-encoder model. To control the number of retrieved chunks, configure max_chunks in lightspeed-stack.yaml:

Config path	Default	Description
`rag.retrieval.inline.max_chunks`	10	Hard upper bound on the final merged inline RAG chunks (BYOK + OKP) delivered to the LLM
`rag.okp.max_chunks`	5	Fetch limit for OKP (Inline RAG); controls how many chunks enter the reranking pool
`rag.byok.max_chunks`	10	Fetch limit for BYOK stores (Inline RAG); controls how many chunks enter the reranking pool
`rag.retrieval.tool.max_chunks`	10	Max chunks retrieved via Tool RAG (`file_search`); independent from inline max_chunks

Limitations:

This is a read-only provider - no insert/delete operations

Complete Configuration Reference

To enable RAG functionality, configure the rag section (including rag.byok.stores and rag.retrieval) in your lightspeed-stack.yaml. Add vector_store when you also need runtime POST /v1/vector-stores capacity.

Below is an example of a working lightspeed-stack.yaml configuration with:

A local all-mpnet-base-v2 embedding model
A FAISS-based static BYOK corpus
Optional FAISS capacity for runtime-created stores
Inline and Tool RAG enabled

[!TIP] We recommend starting with a minimal working configuration and extending it as needed.

name: Lightspeed Core Service (LCS)
service:
  host: localhost
  port: 8080
  auth_enabled: false

# Optional: capacity for runtime POST /v1/vector-stores (not a static corpus)
vector_store:
  default_provider: example
  providers:
    - id: example
      type: faiss
      embedding_model: sentence-transformers/all-mpnet-base-v2
      embedding_dimension: 768
      config:
        path: /home/USER/lightspeed-stack/vector_dbs/example/faiss_store.db

rag:
  byok:
    stores:
      - rag_id: ocp-docs
        backend: faiss
        embedding_model: sentence-transformers/all-mpnet-base-v2
        embedding_dimension: 768
        vector_db_id: vs_3a7f9b2e-45dc-4e1a-b8f2-1c9d0e3f5a6b
        db_path: /home/USER/lightspeed-stack/vector_dbs/ocp_docs/faiss_store.db
  retrieval:
    inline:
      sources:
        - ocp-docs
    tool:
      sources:
        - ocp-docs

BYOK providers and registered resources are generated at startup from rag.byok.stores. Dynamic providers and create defaults are generated from vector_store during unified synthesis. Embedding models for those providers are registered automatically when needed. Inference models and providers must still be configured separately (for example in your baseline / profile run.yaml).

System Prompt Guidance for RAG (as a tool)

When using RAG, the knowledge_search tool must be explicitly referenced in your system prompt. Without clear instructions, models may inconsistently use the tool.

Tool-Aware sample instruction:

You are a helpful assistant with access to a 'knowledge_search' tool. When users ask questions, ALWAYS use the knowledge_search tool first to find accurate information from the documentation before answering.

RAG annotations

The top-level vector_stores block in run.yaml may include annotation_prompt_params to control whether extra RAG annotation instructions are injected into the model prompt (for example, citation-style markers). The default configuration sets enable_annotations: false under that block to avoid unwanted annotations.

When vector_store is configured, default_provider overwrites vector_stores.default_provider_id and default_embedding_model during unified synthesis. Annotation settings are not managed by that enricher — keep them in the Llama Stack baseline/profile or native_override.

RAG Configuration Guide

Table of Contents

Introduction

Inline RAG chunk flow

Prerequisites

Set Up the Vector Database

Download an Embedding Model

Configure BYOK Knowledge Sources

FAISS example

pgvector example

Configure Dynamic Vector Store Providers

FAISS example

pgvector example

Add an Inference Model (LLM)

vLLM on RHEL AI (Llama 3.1) example

OpenAI example

Azure OpenAI

Ollama

vLLM Mistral

OKP/Solr Vector IO

How to Enable OKP/Solr Vector IO

Complete Configuration Reference

System Prompt Guidance for RAG (as a tool)

RAG annotations