Building a RAG pipeline with Spring AI and pgvector

Retrieval-augmented generation, or RAG, answers a question by first finding relevant passages in your own documents and then asking a language model to answer from them. That keeps answers tied to real sources instead of the model's memory.

This article walks through how the Spring RAG AI project puts that idea together with Spring Boot 4, Spring AI and the pgvector extension for PostgreSQL, from ingestion to answer evaluation. It covers the parts that decide quality: chunking, query rewriting, routing and evaluation.

The building blocks

The stack is Java 25 with Spring Boot 4.0.3 and Spring AI 2.0.0-M2. The chat model is OpenAI (GPT-4o-mini by default), and embeddings come from text-embedding-ada-002, which produces 1536-dimension vectors. Vectors are stored in PostgreSQL 17 with the pgvector extension, so document chunks, metadata and embeddings live in one database that most teams already know how to run.

Spring AI 2.0.0-M2 is a milestone release, so APIs can still change before general availability. Pin the version and read the release notes when you upgrade.

Step 1: a vector store in Docker

Docker Compose starts PostgreSQL with pgvector, and its health check lets you confirm the container is healthy before the application starts. Spring AI's pgvector store is then configured with an HNSW index and cosine distance:

application.yml

spring:
  ai:
    vectorstore:
      pgvector:
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        dimensions: 1536
        initialize-schema: true

Why those settings

HNSW gives fast approximate nearest-neighbour search, cosine distance is a common fit for text embeddings, and the dimensions value must match the embedding model. Setting initialize-schema lets Spring AI create the vector table on first run.

Step 2: ingest documents as an ETL pipeline

Ingestion follows extract, transform, load. Format-specific readers extract the text: Apache PDFBox for PDF, and Apache Tika for Word, PowerPoint, Excel and HTML, with plain text and Markdown also supported.

Chunking is where most RAG quality is won or lost, because fixed-size splitting cuts sentences and lists in half. The project uses a structure-aware splitter instead:

It splits on paragraph boundaries and treats each bullet or numbered item as its own chunk.
It keeps section headings as metadata, so a chunk knows which part of the document it came from.
It merges tiny blocks and subdivides oversized ones with a token splitter.
Chunks are capped at 200 tokens, with an 80-character minimum to avoid embedding fragments.

Metadata is what makes the store useful

Every chunk stores its source file, a document ID, its position, the chunk type, the section heading and an ingestion timestamp. That makes metadata filtering possible, supports source attribution in answers, and means you can delete every chunk of one document later with a single call.

Large uploads are processed in the background on a small thread pool while a WebSocket reports each stage: queued, extracting, transforming, loading, then completed or failed.

Step 3: improve the question before searching

Users rarely write good search queries, and follow-up questions depend on earlier turns. Two techniques run before retrieval.

Conversation-aware rewritingUses the last six messages to resolve references. A question about combining annual and sick leave, asked after an answer about a 7-day consecutive limit, becomes a standalone question about exceeding that limit.
Multi-query expansionGenerates three alternative phrasings and searches with all of them, which raises recall when the documents use different words than the user. Parsing is tolerant, because model output is not always perfectly formatted.

Retrieval

Retrieval runs a cosine-distance search over the HNSW index, with optional metadata filters and a configurable top-K. Results are ranked and assembled into a context window, and the final answer keeps source attribution so a reader can check where a claim came from.

Step 4: route and evaluate with two small agents

Two patterns from Spring AI's Effective Agents guidance wrap the pipeline. The router classifies each message as a knowledge query, general chat, greeting or follow-up. A greeting does not need retrieval, so skipping it saves a model call and keeps irrelevant context out of the prompt. Very short queries skip routing altogether, and low-confidence results fall back automatically.

Routing configuration

spring.ai.agents.routing:
  enabled: true
  confidence-threshold: 0.7
  min-query-length: 10

Using the model to judge the model

The evaluator acts as an LLM judge. It scores relevance (0 to 3), grounding (0 to 3), completeness (0 to 2) and accuracy (0 to 2), and when the result falls below the quality threshold it retries, up to a set number of iterations.

Evaluation configuration

spring.ai.agents.evaluation:
  enabled: true
  quality-threshold: 0.7
  max-iterations: 2
  min-answer-length: 50

Why the retry loop is bounded

Every retry is another model call, so a hard cap keeps cost and latency predictable. Skipping evaluation for very short answers avoids paying for a judgement that adds nothing.

Step 5: serve it

The API is small. A chat request carries the message, a conversation ID that scopes memory, and the provider and model to use:

POST /api/chat

{
  "message": "What is the leave policy?",
  "conversationId": "conv-123",
  "provider": "openai",
  "model": "gpt-4o-mini"
}

POST /api/chat/streamThe same request, streamed token by token over Server-Sent Events.
POST /api/documents/file/asyncIngests in the background and returns a job ID and the WebSocket URL for progress.
DELETE /api/documents?source=Removes every chunk that came from one document.

Provider abstraction and threading

Model access sits behind an AiChatClient interface and a registry, so adding another provider means implementing one interface instead of changing the pipeline. Streaming runs on its own thread pool (10 to 50 threads), separate from the ingestion pool (2 to 5 threads), so long-lived streams cannot starve ingestion.

Tracing across async boundaries

Micrometer Tracing with Brave propagates trace and span IDs through asynchronous work, and reactive context propagation is switched on with Hooks.enableAutomaticContextPropagation(). Log lines carry the trace and span IDs, so a single question can be followed through routing, retrieval and evaluation.

Limitations

OpenAI is the only implemented provider. The interface is ready for others, but they are not written yet.
Conversation memory is held in memory, so it is lost on restart and is not shared between instances.
Spring AI 2.0.0-M2 is a milestone release, so expect API changes when the stable version arrives.