Build a RAG Helpdesk Bot for Your Internal Wiki in 2 Days (OpenAI + LlamaIndex + Postgres)

A 90-person Pune SaaS company had a 400-page internal wiki and a Slack channel where the same onboarding questions got asked every week: "how do I request prod access," "what's our refund SOP," "which VPN for the Mumbai office." Over two days in July 2025 we built a RAG bot that answers from the wiki, cites the exact page, and says "I don't know" when the answer isn't there. OpenAI for embeddings and generation, LlamaIndex for the retrieval glue, Postgres with pgvector as the store. Here's the build, with the chunking and eval code that decided whether it was trustworthy. ## The 60-Word Answer RAG (retrieval-augmented generation) means: before the model answers, you fetch the most relevant chunks of your wiki and paste them into the prompt. Chunk your docs (~512 tokens with overlap), embed each chunk with OpenAI's text-embedding-3-small, store the vectors in Postgres pgvector. At query time, embed the question, fetch the top-k chunks, and ask the model to answer using only those — citing sources, refusing if absent. ## Why RAG and Not Just a Bigger Prompt? You could paste your whole 400-page wiki into a long-context model on every question. It works, and it's slow and expensive — you pay for 400 pages of input tokens to answer "which VPN do I use." RAG fetches only the 3–5 relevant chunks, so each answer costs cents and arrives fast. More importantly, RAG gives you citations: the bot can point to "VPN Setup, section 2," which is what makes employees trust it. A long-context dump gives you an answer with no source and a fat bill. ## What You'll Need ## Step 1: Enable pgvector and Create the Table

-- run once in your Postgres
  CREATE EXTENSION IF NOT EXISTS vector;
  
  CREATE TABLE wiki_chunks (
    id          bigserial PRIMARY KEY,
    source      text NOT NULL,       -- e.g. "VPN Setup.md#2"
    content     text NOT NULL,
    embedding   vector(1536)         -- text-embedding-3-small dimension
  );
  
  CREATE INDEX ON wiki_chunks
    USING hnsw (embedding vector_cosine_ops);

The HNSW index is what makes similarity search fast at scale. For 400 pages you'd be fine without it, but add it now so you don't refactor later. ## Step 2: Chunk and Index the Wiki With LlamaIndex LlamaIndex handles the loading, chunking, and embedding orchestration so you write glue, not boilerplate.

# index.py
  from llama_index.core import SimpleDirectoryReader, VectorStoreIndex, StorageContext
  from llama_index.core.node_parser import SentenceSplitter
  from llama_index.vector_stores.postgres import PGVectorStore
  from llama_index.embeddings.openai import OpenAIEmbedding
  
  # 1. Load the wiki folder
  docs = SimpleDirectoryReader("./wiki").load_data()
  
  # 2. Chunk: 512 tokens, 50 overlap (the values we landed on after eval)
  splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
  
  # 3. Point at Postgres pgvector
  vector_store = PGVectorStore.from_params(
      database="company", host="localhost", port=5432,
      user="postgres", password="...",
      table_name="wiki_chunks", embed_dim=1536,
  )
  storage = StorageContext.from_defaults(vector_store=vector_store)
  
  # 4. Build the index (embeds every chunk and writes to Postgres)
  index = VectorStoreIndex.from_documents(
      docs,
      storage_context=storage,
      transformations=[splitter],
      embed_model=OpenAIEmbedding(model="text-embedding-3-small"),
  )
  print("Indexed", len(docs), "documents")

## Step 3: Query With Strict, Citation-Bound Generation The retrieval is easy. The prompt discipline is what separates a trustworthy bot from one that confidently makes things up.

# query.py
  from llama_index.core import VectorStoreIndex
  from llama_index.core.prompts import PromptTemplate
  
  SYSTEM = PromptTemplate(
      "You are the internal helpdesk for a SaaS company.
"
      "Answer ONLY using the context below. Cite the source after each fact "
      "like [source: VPN Setup.md].
"
      "If the answer is NOT in the context, reply exactly: "
      ""I don't know — this isn't in the wiki. Ask #it-help."
"
      "Context:
{context_str}
"
      "Question: {query_str}
"
      "Answer:"
  )
  
  query_engine = index.as_query_engine(
      similarity_top_k=5,
      text_qa_template=SYSTEM,
  )
  
  resp = query_engine.query("How do I request production access?")
  print(resp)
  for node in resp.source_nodes:
      print("  cited:", node.metadata.get("file_name"), round(node.score, 3))

The forced refusal string ("I don't know — this isn't in the wiki") is the single most valuable line in the whole build. A helpdesk bot that confidently invents an SOP is worse than no bot. The instruction to answer "ONLY using the context" plus an explicit escape hatch is what drove our hallucination rate down. ## Step 4: Evaluate Before You Trust It This is the step most tutorials skip and most failed bots skipped too. You cannot eyeball quality on 400 pages. Build a 20-question eval set and measure two things: did it retrieve the right chunk, and did the answer match the truth.

# eval.py — a minimal, runnable eval suite
  eval_set = [
    {"q": "How do I request production access?",
     "must_cite": "Access Control.md",
     "must_contain": "raise a ticket in #access-requests"},
    {"q": "What is our refund SOP for annual plans?",
     "must_cite": "Refund SOP.md",
     "must_contain": "pro-rata"},
    # ... 18 more real questions
  ]
  
  hits, grounded = 0, 0
  for case in eval_set:
      resp = query_engine.query(case["q"])
      sources = [n.metadata.get("file_name", "") for n in resp.source_nodes]
      if any(case["must_cite"] in s for s in sources):
          hits += 1
      if case["must_contain"].lower() in str(resp).lower():
          grounded += 1
  
  print(f"Retrieval hit-rate: {hits}/{len(eval_set)}")
  print(f"Answer-grounded:    {grounded}/{len(eval_set)}")

We ran this after every change. Our first pass used 1024-token chunks and scored 12/20 on retrieval. Dropping to 512 tokens with 50 overlap pushed it to 17/20 — the right chunk now surfaced for questions buried inside long pages. That single tuning step, found only because we measured, is why the bot shipped. ## Common Mistakes ## When NOT to Build a RAG Bot ## Real Example: The Pune SaaS Onboarding Bot We indexed their Confluence export — 400 pages — chunked at 512 tokens, stored in pgvector on their existing Postgres box. The bot ran in a Slack slash command. After tuning, retrieval hit-rate sat at 17/20 on our eval set, and the forced-refusal instruction meant off-wiki questions got a clean "ask #it-help" instead of a hallucination. The Slack onboarding-questions volume dropped noticeably within two weeks, and HR started linking the bot in the joiner checklist. We've shipped this exact architecture several times since: a knowledge bot for a 220-person CA firm over 14 years of GST circulars, a weekend RAG chatbot on Claude Sonnet + LlamaIndex + pgvector, and a RAG bot for an insurance broker where the eval-driven approach cut "I don't know" answers from 38% to 11%. Our AI automation team treats the eval suite as the deliverable, not an afterthought. ## Frequently Asked Questions ### What is RAG in simple terms? RAG (retrieval-augmented generation) means fetching the most relevant pieces of your documents and pasting them into the model's prompt before it answers. The model answers from your real content instead of its training data, so it can cite sources and refuse when the answer isn't there. ### Why use Postgres pgvector instead of a dedicated vector database? If you already run Postgres, pgvector means one fewer system to operate, back up, and pay for. For internal wikis up to hundreds of thousands of chunks it performs well with an HNSW index. Reach for a dedicated vector DB only at very large scale or with specialized needs. ### What chunk size should I use for a RAG bot? Start at 512 tokens with about 50 tokens of overlap, then test 256 and 1024 against a real eval set. In our build, 512/50 scored 17/20 on retrieval while 1024 scored only 12/20. The right value depends on your docs, so measure rather than guess. ### How do I stop a RAG bot from making up answers? Instruct the model to answer only from the retrieved context and to output an exact refusal string ("I don't know — this isn't in the wiki") when the answer is absent. Then track the refusal and grounded-answer rates in your eval suite so you know it's actually obeying. ### How much does it cost to build and run a RAG helpdesk bot? Indexing 400 pages cost roughly ₹600 one-time in embedding tokens. Per query, you pay for one small embedding plus a short generation — cents per question. Re-indexing on a nightly cron adds the one-time cost again each run, still trivial for most wikis. ### How long does it take to build one? A focused engineer can ship a working v1 in about two days: half a day for chunking and indexing, half a day for the query engine and prompt, and a full day building the eval set and tuning. The eval work is what takes real time — and what makes it trustworthy. As Hrishikesh, our CTO, puts it: the demo is easy, the eval set is the engineering. If you can't measure retrieval hit-rate, you don't have a helpdesk bot — you have a magic 8-ball with citations.