---
slug: "mcprag"
source_type: "readme"
source_url: "https://cdn.jsdelivr.net/gh/rajagopal17/mcpRAG@main/README.md"
repo: "https://github.com/rajagopal17/mcpRAG"
source_file: "README.md"
branch: "main"
---
# mcpRAG - Retrieval-Augmented Generation System

A production-ready RAG (Retrieval-Augmented Generation) implementation combining open-source embeddings, FAISS vector database, and Gemini LLM for accurate document-based question answering.

## Overview

This project implements a complete RAG pipeline for querying text documents using:

- **Embeddings**: Nomic embeddings via Ollama (local, open-source)
- **Vector Database**: FAISS (Facebook AI Similarity Search)
- **LLM**: Gemini 2.0 Flash (fast, cost-effective)
- **Document Processing**: Chunking with overlap for context preservation
- **Incremental Updates**: Add new documents without rebuilding entire index

## Architecture

```
Documents (TXT) → Chunking (60 words, 10-word overlap)
                ↓
         Nomic Embeddings (Ollama, 768 dimensions)
                ↓
         FAISS Vector Index (L2 distance similarity)
                ↓
User Query → Query Embedding → FAISS Search → Top-K Chunks
                                              ↓
                                     Context + Query → Gemini LLM
                                              ↓
                                           Answer
```

## Key Features

- **Local Embeddings**: Uses Ollama with Nomic model - no external API calls for embeddings
- **Efficient Search**: FAISS IndexFlatL2 for fast similarity search
- **Chunking Strategy**: 60-word chunks with 10-word overlap to preserve context
- **Incremental Updates**: `appendEmbeds.py` adds new documents without rebuilding
- **Metadata Tracking**: Stores document name, chunk ID, and chunk text for traceability
- **Persistent Storage**: Index and metadata saved locally for reuse

## Installation

### Prerequisites

- Python 3.11+
- Ollama installed and running locally
- Gemini API key
- Nomic embedding model downloaded in Ollama

### Setup

1. **Install Ollama** (if not already installed):

```bash
# macOS/Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Windows
# Download from https://ollama.ai/download
```

2. **Pull the Nomic embedding model**:

```bash
ollama pull nomic-embed-text
```

Verify Ollama is running:
```bash
curl http://localhost:11434/api/embeddings -d '{"model":"nomic-embed-text","prompt":"test"}'
```

3. **Clone and navigate to the project**:

```bash
cd mcpRAG
```

4. **Create virtual environment and install dependencies**:

```bash
python -m venv .venv

# Windows
.venv\Scripts\activate

# macOS/Linux
source .venv/bin/activate

pip install -r requirements.txt
```

5. **Configure API credentials**:

Create a `.env` file:

```env
GEMINI_API_KEY=your_gemini_api_key_here
```

Get your API key from [Google AI Studio](https://makersuite.google.com/app/apikey)

6. **Prepare your documents**:

Place your `.txt` files in the `docfiles/` or `sapfiles/` directory.

## Usage

### 1. Create Initial Embeddings and Index

Run `ollamaEmbed.py` to process your first document and create the FAISS index:

```python
# Edit ollamaEmbed.py to set your document path
file_path = Path("C:\\path\\to\\your\\document.txt")

# Then run
python ollamaEmbed.py
```

**What it does**:
- Reads the specified text file
- Chunks text into 60-word segments with 10-word overlap
- Generates embeddings using Ollama's Nomic model
- Creates FAISS index with L2 distance metric
- Saves index to `my_faiss_index.faiss`
- Saves metadata to `my_metadata.json`

**Output**:
```
Processing document.txt...
✅ Indexed 142 chunks
✅ Saved FAISS index to my_faiss_index.faiss
✅ Saved metadata to my_metadata.json
```

### 2. Query the RAG System

Run `ragModel.py` to search the index and generate answers:

```python
# Edit ragModel.py to set your query
query = "how does SAP integrate with other systems?"
top_k = 5  # Number of relevant chunks to retrieve

# Then run
python ragModel.py
```

**What it does**:
1. Loads the FAISS index and metadata
2. Converts your query to embeddings using Nomic
3. Searches FAISS for the top-K most similar chunks
4. Passes retrieved context + query to Gemini 2.0 Flash
5. Returns a contextual answer

**Example Output**:
```
Query: how does SAP integrate with other systems?

Result 1:
  Document: SAP_Joule.txt
  Chunk: SAP integrates with various external systems through APIs...
  Chunk ID: SAP_Joule_42
--------------------

Gemini Response: SAP integrates with other systems primarily through REST APIs
and OData services. The integration layer supports real-time data synchronization...
```

### 3. Add New Documents to Existing Index

Run `appendEmbeds.py` to incrementally add documents without rebuilding:

```python
# Edit appendEmbeds.py to set your new document path
new_file_path = "C:\\path\\to\\new_document.txt"

# Then run
python appendEmbeds.py
```

**What it does**:
- Loads existing FAISS index and metadata
- Processes the new document (chunking + embeddings)
- Appends new embeddings to existing FAISS index
- Updates metadata with new chunks
- Saves updated index and metadata

**Output**:
```
Processing SAP_GENAI.txt...
✅ Added 87 new chunks to the index
✅ Saved updated FAISS index to my_faiss_index.faiss
✅ Saved updated metadata to my_metadata.json
```

## Project Files

| File | Purpose |
|------|---------|
| `ollamaEmbed.py` | Initial document processing, embedding generation, FAISS index creation |
| `ragModel.py` | Query interface - search index and generate answers with Gemini |
| `appendEmbeds.py` | Incrementally add new documents to existing index |
| `requirements.txt` | Python dependencies |
| `.env` | API credentials (create this file) |
| `my_faiss_index.faiss` | Saved FAISS vector index (auto-generated) |
| `my_metadata.json` | Chunk metadata with document names and text (auto-generated) |
| `docfiles/` | Directory for source documents |
| `sapfiles/` | Directory for SAP-related documents |

## Configuration

### Chunking Parameters

Adjust in any of the Python files:

```python
CHUNK_SIZE = 60        # Number of words per chunk
CHUNK_OVERLAP = 10     # Overlapping words between chunks
```

**Recommendations**:
- **Smaller chunks (30-50 words)**: Better precision, more chunks, slower indexing
- **Larger chunks (80-100 words)**: More context, fewer chunks, faster indexing
- **Overlap (10-20 words)**: Prevents context loss at chunk boundaries

### Retrieval Settings

In `ragModel.py`, adjust `top_k`:

```python
top_k = 5  # Number of chunks to retrieve
```

**Recommendations**:
- **top_k=3**: Fast, focused answers
- **top_k=5**: Balanced (default)
- **top_k=10**: More comprehensive context, slower

### LLM Model

Change Gemini model in `ragModel.py`:

```python
model="gemini-2.0-flash"      # Fast, cost-effective (default)
model="gemini-1.5-pro"         # More capable, higher cost
model="gemini-1.5-flash"       # Balanced
```

### Embedding Model

Change Ollama embedding model (requires model pull):

```python
"model": "nomic-embed-text"     # 768 dimensions (default)
"model": "mxbai-embed-large"    # 1024 dimensions
```

If changing models, delete existing `.faiss` and `.json` files and rebuild.

## How It Works

### 1. Document Chunking

Text is split into overlapping chunks:

```
Original: "The quick brown fox jumps over the lazy dog..."

Chunk 1: "The quick brown fox jumps..." (words 0-60)
Chunk 2: "...fox jumps over the lazy..." (words 50-110, 10-word overlap)
Chunk 3: "...lazy dog runs through..." (words 100-160, 10-word overlap)
```

### 2. Embedding Generation

Each chunk is converted to a 768-dimensional vector via Ollama:

```python
response = requests.post(
    "http://localhost:11434/api/embeddings",
    json={"model": "nomic-embed-text", "prompt": chunk}
)
embedding = np.array(response.json()["embedding"])  # Shape: (768,)
```

### 3. FAISS Indexing

Embeddings are stored in a FAISS IndexFlatL2 (L2 distance):

```python
dimension = 768
index = faiss.IndexFlatL2(dimension)
index.add(np.stack(all_chunks))  # Add all embeddings
```

**FAISS IndexFlatL2** uses Euclidean distance:
- **Closer vectors** (smaller L2 distance) = more similar
- Exhaustive search guarantees best matches
- Fast for up to ~100K vectors

### 4. Semantic Search

Query is embedded and searched against index:

```python
query_embedding = get_embedding(query)  # Shape: (768,)
D, I = index.search(query_embedding.reshape(1, -1), top_k=5)
# D: distances, I: indices of most similar chunks
```

### 5. Answer Generation

Retrieved chunks are passed to Gemini as context:

```python
context = "\n".join([metadata[i]["chunk"] for i in I[0]])
prompt = f"Query: {query}\n\nContext: {context}\n\nAnswer:"
response = client.models.generate_content(model="gemini-2.0-flash", contents=prompt)
```

## Example Workflow

### Scenario: Building a SAP Documentation RAG System

1. **Initial setup** with SAP Joule documentation:

```bash
# Edit ollamaEmbed.py
file_path = Path("sapfiles/SAP_Joule.txt")

python ollamaEmbed.py
# Output: ✅ Indexed 142 chunks
```

2. **Query the system**:

```bash
# Edit ragModel.py
query = "What is SAP Joule?"

python ragModel.py
# Output: SAP Joule is an AI assistant that helps users...
```

3. **Add more documentation**:

```bash
# Edit appendEmbeds.py
new_file_path = "sapfiles/SAP_GENAI.txt"

python appendEmbeds.py
# Output: ✅ Added 87 new chunks to the index
```

4. **Query with expanded knowledge base**:

```bash
# Query now searches across both documents
python ragModel.py
```

## Advanced Usage

### Batch Processing Multiple Documents

Modify `ollamaEmbed.py` to process a directory:

```python
from pathlib import Path

all_chunks = []
metadata = []

for file_path in Path("docfiles").glob("*.txt"):
    chunks, meta = process_document(file_path)
    all_chunks.extend(chunks)
    metadata.extend(meta)

# Then create index as usual
```

### Custom Similarity Metrics

Replace `IndexFlatL2` with other FAISS indices:

```python
# Inner product (for normalized embeddings)
index = faiss.IndexFlatIP(dimension)

# Approximate search (faster for large datasets)
index = faiss.IndexIVFFlat(quantizer, dimension, nlist=100)
```

### Multi-Query RAG

Query multiple times with refined prompts:

```python
queries = [
    "What is SAP Joule?",
    "How does SAP Joule integrate with SAP systems?",
    "What are the key features of SAP Joule?"
]

for query in queries:
    results = search_index(index, metadata, query, top_k=5)
    # Process results...
```

## Performance Optimization

### For Large Document Collections (>10K chunks)

1. **Use IVFPQ index** (Inverted File with Product Quantization):

```python
nlist = 100  # Number of clusters
m = 8        # Number of subquantizers
quantizer = faiss.IndexFlatL2(dimension)
index = faiss.IndexIVFPQ(quantizer, dimension, nlist, m, 8)
index.train(embeddings)  # Training required
index.add(embeddings)
```

2. **Batch embedding generation**:

```python
# Process chunks in batches
batch_size = 50
for i in range(0, len(chunks), batch_size):
    batch = chunks[i:i+batch_size]
    # Generate embeddings for batch...
```

### For Faster Queries

1. **Reduce top_k**: Retrieve fewer chunks (3 instead of 5)
2. **Use Gemini Flash**: Faster than Pro models
3. **Cache embeddings**: Store query embeddings for repeated queries

## Troubleshooting

### Ollama Connection Error

**Error**: `requests.exceptions.ConnectionError: Failed to establish a new connection`

**Solution**:
```bash
# Check if Ollama is running
curl http://localhost:11434/api/tags

# If not, start Ollama
ollama serve
```

### Model Not Found

**Error**: `model 'nomic-embed-text' not found`

**Solution**:
```bash
ollama pull nomic-embed-text
ollama list  # Verify model is downloaded
```

### FAISS Dimension Mismatch

**Error**: `RuntimeError: Error in void faiss::IndexFlat::add_with_ids()`

**Solution**: Delete existing `.faiss` and `.json` files and rebuild:
```bash
rm my_faiss_index.faiss my_metadata.json
python ollamaEmbed.py
```

### Gemini API Error

**Error**: `google.api_core.exceptions.PermissionDenied: 403 API key not valid`

**Solution**: Verify your `.env` file:
```bash
cat .env  # Should show: GEMINI_API_KEY=AIza...
```

Get a new key from [Google AI Studio](https://makersuite.google.com/app/apikey)

### Memory Issues (Large Documents)

**Error**: `MemoryError: Unable to allocate array`

**Solution**: Process documents in smaller batches or use dimensionality reduction:
```python
# Use PCA to reduce dimensions
import faiss
pca_matrix = faiss.PCAMatrix(768, 256)  # Reduce to 256 dimensions
index = faiss.IndexFlatL2(256)
```

## Technical Details

### FAISS IndexFlatL2

- **Algorithm**: Exhaustive search (brute force)
- **Metric**: L2 (Euclidean) distance
- **Accuracy**: 100% (finds exact nearest neighbors)
- **Speed**: O(N*D) where N=number of vectors, D=dimension
- **Best for**: Up to 1M vectors with 768 dimensions

### Nomic Embeddings

- **Model**: `nomic-embed-text`
- **Dimensions**: 768
- **Context Length**: 8192 tokens
- **Trained on**: Billions of text pairs for semantic similarity
- **License**: Open-source (Apache 2.0)

### Chunking Strategy

**Why 60 words with 10-word overlap?**

- **60 words**: ~300-400 characters, fits typical semantic units (paragraphs)
- **10-word overlap**: Prevents context loss at boundaries
- **Result**: Each chunk has some context from neighbors

**Trade-offs**:
- More overlap → Better context, more storage
- Less overlap → Faster processing, potential context gaps

## Comparison with Alternatives

| Component | This Project | Alternative |
|-----------|--------------|-------------|
| **Embeddings** | Ollama (local, free) | OpenAI ($0.0001/1K tokens) |
| **Vector DB** | FAISS (local, fast) | Pinecone/Weaviate (cloud, $$) |
| **LLM** | Gemini Flash (cheap) | GPT-4 (expensive) |
| **Storage** | Local files | Cloud databases |
| **Privacy** | Embeddings local | Data sent to APIs |

**Advantages**:
- Lower cost (only LLM API calls)
- Privacy (embeddings stay local)
- Fast (local FAISS search)
- No vendor lock-in

**Disadvantages**:
- Requires Ollama setup
- Local storage needed
- No built-in scaling

## Extending the Project

### Add Support for PDF Documents

```bash
pip install PyPDF2
```

```python
import PyPDF2

def extract_text_from_pdf(pdf_path):
    with open(pdf_path, 'rb') as file:
        reader = PyPDF2.PdfReader(file)
        text = ""
        for page in reader.pages:
            text += page.extract_text()
    return text
```

### Add Conversational Memory

Store conversation history and include in context:

```python
conversation_history = []
conversation_history.append({"role": "user", "content": query})
conversation_history.append({"role": "assistant", "content": response.text})
```

### Add Web UI with Streamlit

```bash
pip install streamlit
```

```python
import streamlit as st

st.title("RAG Document Q&A")
query = st.text_input("Ask a question:")
if st.button("Search"):
    results = search_index(index, metadata, query, top_k=5)
    answer = generate_answer(results, query)
    st.write(answer)
```

### Deploy as MCP Server

Integrate with the MCP (Model Context Protocol) pattern:
- Expose RAG search as an MCP tool
- Add resources for document management
- See `../agenticMCP_rag/` for reference implementation

## References

- [FAISS Documentation](https://github.com/facebookresearch/faiss/wiki)
- [Ollama Documentation](https://ollama.ai/docs)
- [Nomic Embeddings](https://www.nomic.ai/blog/posts/nomic-embed-text-v1)
- [Gemini API](https://ai.google.dev/docs)
- [RAG Paper (Lewis et al., 2020)](https://arxiv.org/abs/2005.11401)

## License

Part of the experimental AI/ML monorepo. See root repository for license details.

## Related Projects

For more advanced RAG implementations, see:
- `../agenticMCP_rag/` - RAG with MCP integration and agent architecture
- `../R_RAG/` - Alternative RAG implementation
- `../perceptionAgent/` - Agent architecture with perception-decision-action pattern