Spring RAG AI

How a question is answered

Every message passes through the same sequence, and each step is configurable in application.yml.

• 1. RouteA routing agent classifies the message as a knowledge query, general chat, greeting or follow-up. Routing uses a confidence threshold of 0.7, greetings take a fast path, and low-confidence results fall back automatically.
• 2. RewriteA conversation-aware transformer resolves pronouns such as "it" or "both" using the last six messages, so the query stands on its own. Multi-query expansion then generates three alternative phrasings to raise recall.
• 3. RetrieveThe rewritten queries search pgvector using an HNSW index and cosine distance, with metadata filtering and a configurable top-K.
• 4. GenerateThe model answers from the retrieved context, and the response keeps source attribution so a reader can see which document a claim came from.
• 5. EvaluateAn evaluation agent acts as an LLM judge. It scores relevance, grounding, completeness and accuracy, and if the score falls below a 0.7 quality threshold the answer is regenerated, up to two iterations.

Follow-up questions that still make sense

Retrieval works poorly on a question like "Can I combine annual and sick leave?" when the important detail sits in the previous turn. The transformer rewrites it using the conversation so far. In the README example, a user asks how many consecutive leave days are allowed, hears "maximum 7 days at a time", and then asks about combining annual and sick leave. The system rewrites that into a standalone question about combining them to exceed the 7-day consecutive limit, and retrieval succeeds.

Document ingestion

Ingestion follows an extract, transform, load flow. Format-specific readers extract text: Apache PDFBox for PDF, and Apache Tika for Word, PowerPoint, Excel and HTML, with plain text and Markdown also supported.

A custom structure-aware splitter does the transform step. It splits on paragraph boundaries, keeps each bullet as its own chunk, preserves headings as metadata, merges tiny blocks and subdivides oversized ones, with chunks capped at 200 tokens. Every chunk stores its source file, document ID, position, chunk type and section heading, and is loaded into the vector store with embeddings.

Large files can be ingested asynchronously on a small thread pool. Progress is pushed over a WebSocket through the stages queued, extracting, transforming, loading and completed or failed.

API

• POST /api/chatSynchronous chat that returns the full answer.
• POST /api/chat/streamStreaming chat over Server-Sent Events, token by token.
• GET /api/chat/models/{provider}Lists the models available for a provider.
• POST /api/documents/textIngests plain text with optional metadata.
• POST /api/documents/fileIngests an uploaded document synchronously.
• POST /api/documents/file/asyncIngests in the background and returns a job ID and WebSocket URL for progress.
• DELETE /api/documents?source=Removes every chunk that came from one document.

Design patterns used

• StrategyAn AiChatClient interface with a registry resolves the provider at runtime. OpenAI is implemented, and the structure leaves room for others such as Ollama, Anthropic or Azure OpenAI.
• FactoryA document reader factory chooses the right reader from the file extension.
• Router-DispatcherThe query router classifies intent and sends the request to the right handler.
• Evaluator-OptimizerThe answer evaluator judges output and drives the bounded retry loop.
• ETL pipelineExtract, transform and load stages keep ingestion steps separate and testable.
• ObserverWebSocket handlers publish ingestion progress as events.

Observability and operations

The application uses Micrometer Tracing with Brave, and trace and span IDs are propagated across asynchronous boundaries, including reactive context. Ingestion and Server-Sent Events streaming run on separate thread pools, shutdown is graceful, and Spring Boot Actuator health checks make it ready for Kubernetes readiness and liveness probes.