Building a Lightweight RAG Chat Application

A retrieval‑augmented generation (RAG) application combines a language model with a searchable collection of documents. Rather than relying solely on information encoded during training, it retrieves relevant passages at query time and feeds them to the model as context. This approach allows the system to answer questions about your own documents, such as documentation, runbooks, policies, and meeting notes, without the need for fine‑tuning a model.

This article uses LangChain and ChromaDB as concrete examples. Both are real, actively maintained libraries, and the same concepts apply to LlamaIndex, Haystack, or any other RAG framework.

The Core Flow

Every RAG system follows a consistent five‑step process:

1. Split documents into useful chunks.
2. Convert these chunks into embeddings, which are dense vector representations.
3. Store these embeddings in a vector index.
4. At query time, embed the question and retrieve the most similar chunks.
5. Pass the retrieved chunks to a language model and ask it to answer the question using that context.

While retrieval can improve freshness and traceability, it cannot correct incomplete or contradictory documents. If the source material omits a detail or presents conflicting statements, the model has no reliable basis to resolve the gap. The quality of the answers is limited by the quality of the source material. Thus, well‑curated, up‑to‑date documents are essential for trustworthy output.

A Minimal Working Example

from langchain_community.document_loaders import DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA

# 1. Load documents from a local directory
loader = DirectoryLoader("./knowledge-base", glob="**/*.md")
docs = loader.load()

# 2. Split into chunks (overlap preserves context across boundaries)
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

# 3. Embed and store in a local vector database
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./chroma_db")

# 4 & 5. Retrieve and generate
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)

answer = qa.invoke("How do I reset my password?")
print(answer["result"])

A production implementation also requires document updates, access control, observability, rate limits, prompt‑injection defenses, and evaluation datasets. Document updates ensure that newly added or revised files are re‑embedded promptly; access control filters results according to user permissions; observability lets us monitor latency, errors, and usage patterns; rate limits protect downstream APIs from overload; prompt‑injection defenses guard against malicious inputs that could steer the model toward unsafe behavior; and evaluation datasets give us a benchmark to measure recall and precision over time.

Real-World Use Cases

Customer Support

A documentation assistant can answer questions about an API and maintain conversational context. It can keep track of prior turns so that follow‑up queries refer back to earlier clarifications. It should cite or link to source material whenever possible and escalate uncertain cases to a human. Citations let users verify the answer, and escalation prevents the system from silently guessing. Without source citations, users cannot verify answers, and the model will occasionally be incorrect.

Internal Knowledge Bases

Teams can search policies, runbooks, and meeting records using natural language. This frees them from combing through folders of PDFs. Access control must be applied during retrieval: a model should not reveal documents merely because it can retrieve them. We therefore filter the candidate chunks by the requester’s permissions before they ever reach the language model. Filter by user permissions before passing chunks to the model.

Code Review Assistance

A system can summarize a change, retrieve relevant conventions, and suggest questions for reviewers. It can point the reviewer to the style guide or previous implementations that illustrate the intended pattern. However, it should not replace tests, security scanning, or the owner's approval.

How the Components Work

Embedding Layer

An embedding model maps text to vectors that capture semantic relationships. Similar meaning produces similar vectors, so a question about "password reset" will retrieve chunks about "account recovery" even if the exact phrase doesn't match. The vector space thus acts as a fuzzy matcher that bridges synonyms and paraphrases. The choice of model should align with the language, domain, and privacy requirements, OpenAI's text-embedding-3-small is a practical default; nomic-embed-text is a strong local alternative via Ollama.

Retrieval Layer

A vector index returns candidate passages based on similarity. Approximate nearest‑neighbor indexes like HNSW (used by ChromaDB and pgvector) make large searches fast. Retrieval quality depends on chunking strategy, metadata filters, and evaluation, not just the index algorithm. Effective chunking preserves context while keeping vectors manageable; metadata filters enforce scope such as date or author; and systematic evaluation reveals gaps.

Generation Layer

The selected passages are placed in the model's context window. The application should instruct the model to distinguish evidence from inference, cite sources, and abstain when the context is insufficient:

from langchain.prompts import PromptTemplate

PROMPT = PromptTemplate(
    template="""Use only the context below to answer the question.
If the answer is not in the context, say "I don't have that information."
Cite the source document for each claim.

Context:
{context}

Question: {question}
Answer:""",
    input_variables=["context", "question"],
)

Why Not a Traditional Chatbot?

Decision trees and keyword systems are predictable for narrow flows. RAG systems handle paraphrases and multi‑turn questions more flexibly, but they are probabilistic and can produce convincing errors. Because the model generates text rather than selecting a predefined answer, it can hallucinate plausible‑sounding but unfounded statements. The choice depends on the risk and variability of the task. For a narrow, well‑defined flow (e.g., "book a meeting"), a deterministic system is often more reliable. For open‑ended questions over a large document corpus, RAG is usually the better fit.

Trade‑offs

Answer quality depends on source quality and retrieval quality. Stale documents, poor chunking, ambiguous passages, and missing permissions can all lead to bad answers. Each of these issues reduces the grounding that keeps the model honest.

Latency and Cost: Each turn involves embedding, search, and generation. Caching and precomputed embeddings help, but they must be invalidated when content or permissions change; otherwise stale vectors would retrieve irrelevant or unauthorized material.

Hallucination: RAG reduces some forms of hallucination by grounding the model in retrieved text. However, it does not eliminate them. A model can misread a passage, combine unrelated passages, or answer beyond the available evidence. Evaluation datasets with known correct answers are essential for measuring this.

The Bottom Line

A lightweight RAG application can be a practical foundation for documentation search, internal knowledge access, and domain‑specific assistance. It gives non‑technical users a conversational interface to the information they already own. Its success depends less on the framework than on disciplined data preparation, retrieval evaluation, access control, and human escalation for uncertain cases.