What is RAG?... and how to get it right
Amit Kohli
Access Social Care - Head of Data
& freelance consultant
What you’ll learn today
- What is Retrieval Augmented Generation (RAG)
- The traps you’ll probably fall into
- Tangible things you can implement right away
What is RAG?
I WANT...
- MY STUFF
- DOCS I CAN TRUST
- BALANCE
and serious use-cases:
- Find information faster
- Bespoke chatbots
- Generate 'on-brand' content faster
OK, let's dive in!
Really, what's a RAG?
- RAGs "picks" the n chunks that are most like the question
Mathy stuff
Cosine Similarity
RAG in it's full glory
Problems you might run into
Hallucinations
Plagiarism risks
Corpus imbalance (your data sources are lopsided)
Your corpus is probably a mess
Source confusion
The paragraph needs to know where it came from
Recency vs. accuracy trade-offs
Chunks that don’t understand context
Chunk permissions?
Solutions
- Control the corpus!
- chaos / one source dominate
- You need metadata!
- especially date
- Evaluate outputs against clear criteria
- Human in the loop and internal until you're sure
- Don't trust simple solutions!
- "everything should be made as simple as possible, but no simpler"
What did I do?
- Data sources and metadata are stored in Monday.com
- Built an app: Ingest Docs / Ask Questions
Tech stack
Streamlit app, chromadb vector store, ChatGPT LLM, Langsmith eval
Key takeaways
- Start with your corpus: What docs do you trust?
- Metadata systems for document storage are key
- Similarity scores aren’t the whole story
Extra credit
- Chunk re-ranking
- Smarter chunking / multimodal digestion
- Knowledge graphs / graphRAG
What is RAG?... and how to get it right Amit Kohli Access Social Care - Head of Data & freelance consultant