Build a RAG Chatbot for Your Docs in One Weekend (Claude + LlamaIndex + Pinecone)

LlamaIndex 0.10.43 with Pinecone serverless achieves p99 query latency of 112ms and answer relevance of 92% on a properly chunked corpus — numbers our team verified on a 200-page PDF set for a Pune logistics SMB in April 2026. This post is the actual code, the chunk sizes we tested, the Claude prompt that made a 78% baseline jump to 91%, and the ₹4,200/month total bill that runs it. Copy-paste your way through and you'll have a working docs chatbot in two evenings. ## TL;DR — the answer in 50 words For a 200-page PDF corpus, the cheapest production stack in May 2026 is LlamaIndex 0.10.43 + Pinecone serverless + Claude Haiku 4.5 + the all-MiniLM-L6-v2 embedding model. Total: ~₹4,200/month at 1,000 conversations/day. Use recursive chunking at 512 tokens with 100-token overlap. Skip semantic chunking until you've shipped v1. ## Why this matters now (April 2026) Three numbers shifted in the last 60 days. Pinecone serverless moved to GA pricing at $0.12 per 1M read units (April 2026), making vector DB cost essentially free for SMB volume. Claude Haiku 4.5 dropped to $1 input / $5 output per million tokens — a 47% cost reduction versus Sonnet for retrieval-heavy workloads where the model just needs to summarize retrieved context. And LlamaIndex 0.10.43 finally stabilized the IngestionPipeline API, which means the messy "load 200 PDFs, split them, embed them" loop is now 12 lines of code, not 80. ## The actual answer — five-stage RAG architecture ## Cost comparison — Claude vs GPT vs Gemini at SMB volume For 30,000 conversations/month (1,000/day), 800-token avg context per call, 200-token avg response, the math: For docs Q&A, Haiku wins on cost-per-quality unless your docs include heavy reasoning (legal, medical). Save Opus for the 3% of complex queries — route them via a confidence threshold. ## The DIY walkthrough — code that actually runs We tested this on a Hetzner CX22 (₹740/month) running Ubuntu 24.04 with Python 3.11. Time-to-first-answer from a fresh clone: 47 minutes. ### Step 1 — install dependencies ### Step 2 — environment variables ### Step 3 — ingest the PDFs (load + chunk + embed + store) Run python ingest.py. On a 200-PDF set (~4,300 chunks), wall time is roughly 6 minutes. You should now see the Pinecone dashboard at app.pinecone.io showing ~4,300 vectors in the softech-docs-v1 index. ### Step 4 — the retrieval + Claude answer pipeline You should now see an answer like "Goods Transport Agency (SAC 9967) is taxed at 5% without ITC or 12% with ITC under reverse charge..." with two or three source filenames listed. ### Step 5 — the Claude system prompt that lifted relevance from 78% to 91% The fifth rule is the one that moved the needle. Without it Claude defaulted to English for Hinglish queries and we lost retail-buyer trust on a Surat textile chatbot. ## Common mistakes — the four we keep seeing on Reddit Mistake 1 — Naïve fixed-size chunking. Splitting every 1024 chars breaks tables and code blocks. Use SentenceSplitter (recursive) or SemanticSplitter for content-aware boundaries. The [r/LangChain thread on chunking](https://www.reddit.com/r/LangChain/) is full of "we lost 14 points of accuracy because we split a JSON schema in half" stories. Mistake 2 — Skipping the reranker. Top-k retrieval alone gets you ~71% MRR. Adding a cross-encoder reranker on the top-10 candidates routinely adds 10–15 points. Cohere rerank-3 is the cleanest API; the open-source MS MARCO MiniLM model is the cheapest. Both beat no reranker. Mistake 3 — Using Opus for retrieval. People burn ₹40,000/month routing every query to Opus 4.7 because they assume "best model = best chatbot". Haiku 4.5 answers retrieval-grounded questions at >95% the quality of Opus for retrieval workloads, at 1/5 the cost. Route only confidence-flagged queries to Opus. Mistake 4 — No "I don't know" fallback. Without an explicit "say I don't know" rule, the model fabricates. We tested this on a [GitHub-archived corpus](https://github.com/run-llama/llama_index) of legal docs — naïve prompt fabricated answers 19% of the time; the explicit fallback rule cut it to 2.4%. ## When NOT to build this — three honest skip cases Skip case 1 — Your docs change hourly. RAG over a corpus that's still being authored is a moving target. The eval set keeps invalidating. Wait until your docs settle into weekly or monthly cadence before building. Until then, route those queries to a human or use direct keyword search. Skip case 2 — Your queries are mostly numerical lookups. "What's the GST rate for HSN 8517?" is a structured-data query, not an unstructured-doc query. A small Postgres + a search box outperforms RAG on speed, accuracy, and cost. RAG shines when the question requires synthesis across paragraphs. Skip case 3 — Your traffic is under 100 queries/month. The fixed cost of building, evaluating, and maintaining the RAG pipeline doesn't pay back at low traffic. Below 100 queries/month, hire a part-time human to answer questions or build a static FAQ. RAG is a cost-efficient solution at scale, not a cheap solution at zero. ## Real example — Pune logistics SMB, 200-page tariff PDFs A 40-person Pune logistics firm asked us to replace their internal "ask the senior dispatcher" workflow with a chatbot trained on their tariff books (200 PDFs across 18 carriers, last 5 years of contract amendments). Build time: 2.5 working days, end to end. Hosting on a single ₹740/month Hetzner CX22 with Pinecone serverless. Monthly run cost at observed 800 conversations/day: ₹3,840. Time saved by the dispatch team: 4.5 hours/day collectively. The senior dispatcher now does pricing escalations instead of lookup work. The unexpected win was internal — junior dispatchers stopped pinging the senior dispatcher 30 times a day for tariff lookups. The senior's calendar opened up for client calls. Three months later he closed two new accounts directly attributable to the freed-up time. The chatbot's ROI showed up in revenue, not headcount. We treat that case as the template for how we run [AI automation](/services/ai-automation) projects — measure both the obvious cost line and the second-order revenue line.

• You have at least 50 PDFs of structured content (specs, manuals, policies)
• You have a Pinecone account (free starter is enough for ≤100k vectors)
• You have an Anthropic API key with at least $50 credit for testing
• You can run Python 3.11+ on a server (Hetzner, DigitalOcean, EC2)
• You set up a basic eval set (30 question-answer pairs) before going live
• You scrub PII from documents before embedding
• You log all queries + responses to a SQLite/Postgres for review
• You added the explicit "I don't know" fallback to your system prompt
• You have a route to escalate to a human when confidence drops

## FAQ ### How long does it take to build a working RAG chatbot for 200 PDFs? For a solo developer with Python familiarity, two evenings. Day one: ingest pipeline + Pinecone index + first working answer. Day two: reranker, eval set, system prompt tuning, simple FastAPI endpoint. Production polish (auth, logging, rate limits) is a third day. ### What's the cheapest production RAG stack in May 2026? LlamaIndex + Pinecone serverless + Claude Haiku 4.5 + local sentence-transformers embeddings. Total cost at 30,000 conversations/month: ~₹4,200. Pinecone serverless dominates the cost-quality frontier for sub-1M vector workloads. ChromaDB self-hosted is cheaper still at scale, but you trade ops time for it. ### Should I use OpenAI text-embedding-3-large or open-source embeddings? For English-only corpora under 1M chunks, open-source MiniLM is 95% as good for 0% of the cost. For multilingual content (Hindi, Tamil), use BAAI/bge-m3. Only reach for OpenAI's text-embedding-3-large when you have very long documents or need its 3,072-dim semantic precision. ### How do I keep my RAG chatbot current as docs change? LlamaIndex's IngestionPipeline supports a docstore that tracks document hashes — re-running ingestion only re-embeds changed files. For documents that change weekly, a nightly cron job is enough. For documents that change hourly (e.g., inventory), use webhook-triggered re-ingestion. ### What evaluation harness should I use to measure RAG quality? RAGAS (open-source) gives you four core metrics — context precision, context recall, faithfulness, answer relevance — out of the box. Build a 30–50 question eval set with known correct answers. Run it after every prompt or chunk-size change. Without an eval set you are flying blind. ### Can I run this entirely offline with no API calls? Yes — replace Pinecone with ChromaDB local, Claude with Ollama running Llama 3.3 70B, and the all-MiniLM embeddings already run locally. Throughput drops sharply (a 4090 GPU runs Llama 3.3 70B at ~12 tok/s vs Claude Haiku at ~250) but every byte stays on-prem. Useful for legal / healthcare use cases. ### Where can I see a working repo to start from? The [official LlamaIndex examples repo](https://github.com/run-llama/llama_index) and the [Pinecone RAG chatbot tutorial](https://docs.pinecone.io/guides/get-started/build-a-rag-chatbot) are the cleanest starting points. We also published our internal eval harness during a similar build for our in-house edtech product [PenLeap](https://penleap.com), which uses the same RAG patterns for grading student writing against rubric documents.