Building an Agentic RAG locally with Ollama and Milvus

1 | © Copyright 8/16/23 Zilliz
Stephen Batifol | Zilliz
Unstructured Data Meetup, June 25th
Using LLM Agents with Llama
3, LangGraph and Milvus

Stephen Batifol
Developer Advocate, EMEA, Zilliz
stephen.batifol@zilliz.com
https://www.linkedin.com/in/stephen-batifol/
https://twitter.com/stephenbtl
Speaker

27K+
GitHub
Stars
25M+
Downloads
250+
Contributors
2,600
+
Forks
Milvus is an open-source vector database for GenAI projects. pip install on your
laptop, plug into popular AI dev tools, and push to production with a single line of
code.
Easy Setup
Pip-install to start
coding in a notebook
within seconds.
Reusable Code
Write once, and
deploy with one line
of code into the
production
environment
Integration
Plug into OpenAI,
Langchain,
LlmaIndex, and
many more
Feature-rich
Dense & sparse
embeddings,
filtering, reranking
and beyond

Seamless integration with all popular AI toolkits

| © Copyright 8/16/23 Zilliz
5
RAG
(Retrieval Augmented Generation)

Basic Idea
Use RAG to force the LLM to work with your data
by injecting it via a vector database like Milvus

Basic RAG Architecture

01 Tech Stack

• Framework for building LLM Applications
• Focus on retrieving data and integrating with
LLMs
• Integrations with most AI popular tools
🦜🔗 LangChain

🦜🕸 LangGraph by LangChain
• Build Stateful apps with LLMs and Multi-Agents workflow
• Cycles and Branching
• Human-in-the-Loop
• Persistence

Ollama
• Run LLMs anywhere
• Run Embedding Models

02 Agentic RAG

• Routing: Adaptive RAG
• Route Questions to different retrieval approaches
• Fallback: Corrective RAG
• Fallback to web search if docs are not relevant to query
• Self-Correction: Self-RAG
• Try to fix answers with hallucinations or don’t address question
General Ideas

General Ideas for Agents
• Reflection
• Self-Correction Mechanism
• Planning:
• The agent doesn’t just react to the query
• Lays out a step-by-step process to retrieve or generate the best answer
• Tool use
• Search for Knowledge Base in Milvus
• Search the web for more information

General Ideas

milvus.io
github.com/milvus-io/
@milvusio
@stephenbtl
/in/stephen-batifol
Questions?

02 Advanced RAG techniques

● Divide & Conquer
○ Query Enhancement: better express or process the query intent.
○ Indexing Enhancement: data cleanup, better parser and chunking
○ Retriever Enhancement: more retrievers and hybrid search strategy
○ Generator Enhancement: prompt engineering and more powerful LLM
Types of RAG Enhancement Techniques

Meta Storage
Root Query Data Index
Coordinator Service
Proxy
Proxy
etcd
Log Broker
SDK
Load Balancer
DDL/DCL
DML
NOTIFICATION
CONTROL SIGNAL
Object Storage
Minio / S3 / AzureBlob
Log Snapshot Delta File Index File
Worker Node QUERY DATA DATA
Message Storage
VECTOR
DATABASE
Access Layer
Query Node Data Node Index Node
Milvus Architecture

Building an Agentic RAG locally with Ollama and Milvus

More Related Content

What's hot

Similar to Building an Agentic RAG locally with Ollama and Milvus

More from Zilliz

Recently uploaded

Building an Agentic RAG locally with Ollama and Milvus