---
slug: "rag-dataset-builder"
source_type: "readme"
source_url: "https://cdn.jsdelivr.net/gh/datapizza-labs/rag-dataset-builder@main/README.md"
repo: "https://github.com/datapizza-labs/rag-dataset-builder"
source_file: "README.md"
branch: "main"
---
# RAG Evaluation Dataset + Dataset Builder

This repository contains:
1. **A high-quality RAG evaluation dataset**: 56 question-answer pairs built from the D&D 5e System Reference Document
2. **A dataset builder pipeline**: Tools to create your own QA datasets from PDFs for evaluating Retrieval Augmented Generation (RAG) systems

---

## The Dataset

### Dataset Summary

A high-quality Question-Answering (QA) dataset built by the Datapizza AI Lab from the Dungeons & Dragons 5th Edition System Reference Document (SRD) version 5.2.1, designed to evaluate Retrieval Augmented Generation (RAG) systems.

**Dataset Statistics:**
- **Total Questions**: 56
  - **Easy Tier**: 25 questions
  - **Medium Tier**: 31 questions
- **Source Documents**: 20 markdown files from D&D 5e SRD
- **Domain**: Tabletop role-playing game rules and mechanics

### Dataset Tiers

#### Easy Tier

The Easy tier provides a baseline that is cheap and scalable. Questions and answers are generated automatically from the source documents and then human-checked for quality, making this tier ideal for establishing reproducible baselines in RAG evaluation.

**Characteristics:**
- Direct, single-source retrieval
- Straightforward factual queries
- Machine-generated with human verification
- Cost-effective to reproduce and scale

#### Medium Tier

> **⚠️ Cost Warning**: Generating the medium dataset using Claude Skills can be expensive. The 31 questions in our medium dataset cost approximately **$100** to generate (including reruns with hints) due to the extensive use of Claude's agentic capabilities and long context windows.

The Medium tier features questions (with optional hints) and answers that require more sophisticated reasoning. This tier reflects real evaluator intent and is more diagnostic of retrieval and reasoning gaps.

**Characteristics:**
- Multi-hop reasoning requirements
- Complex queries requiring synthesis across multiple sources
- Two question types:
  - **multi_hop**: Answered using Claude Agent Skills for multi-step reasoning
  - **wide**: Answered using LLM Retriever for wide-coverage questions

### Dataset Structure

Each entry in the dataset contains the following fields:

```
{
  "id": int,
  "question": string,
  "answer": string,
  "passages": [
    {
      "content": string,
      "document_path": string,
      "start_char": int,
      "end_char": int
    }
  ]
}
```

**Field Descriptions:**
- `id`: Unique identifier for the question-answer pair
- `question`: The question text
- `answer`: The answer text (may include citations and detailed explanations in Medium tier)
- `passages`: List of relevant passages used to answer the question
  - `content`: The text content of the passage
  - `document_path`: Path to the source document
  - `start_char`: Starting character position in the source document
  - `end_char`: Ending character position in the source document

### Using the Dataset

The dataset is available in the `dataset/qa_sets/` directory in both JSON and Parquet formats:
- `dataset/qa_sets/easy.json`
- `dataset/qa_sets/medium.json`

The source documents are available in `dataset/raw_data/`.

You can also find the dataset on [HuggingFace](https://huggingface.co/datasets/datapizza-ai-lab/DND_SRD_5_2_1).

---

## The Dataset Builder

Create your own high-quality QA datasets from PDFs to evaluate RAG systems. The pipeline parses PDFs to Markdown, generates questions, retrieves/supports evidence, and synthesizes answers with citations.

### Why Two Tiers?

Easy offers a reproducible baseline; Medium reflects real evaluator intent and is more diagnostic of retrieval/reasoning gaps.

### Dataset Builder Quick Links

- Pipeline overview: `src/dataset_builder/README.md`
- Preprocess PDFs: `src/dataset_builder/preprocess_pdf_parser/README.md`
- Easy Q&A: `src/dataset_builder/easy_answers_questions_generator/README.md`
- Medium (LLM Retriever): `src/dataset_builder/medium_answers_llm_retriever/README.md`
- Medium (Claude Skills): `src/dataset_builder/medium_answers_claude_skill/README.md`
- Postprocess Medium: Convert output format to match standardized QA
- Shared models/utilities: `src/dataset_builder/common/README.md`

### Quickstart

#### Install

```bash
git clone <repository-url>
cd rag-evaluation
uv sync    # or: pip install -e .
```

#### Set up environment variables

Export the required API keys in your shell:

```bash
export GOOGLE_API_KEY=your_google_api_key
export ANTHROPIC_API_KEY=your_anthropic_api_key
```

You only need to export the keys for the stages you plan to use:
- `GOOGLE_API_KEY` is required for: preprocess, easy_qa, llm_retriever
- `ANTHROPIC_API_KEY` is required for: claude_skill

#### Run the full pipeline

1) Configure the pipeline in `config/pipeline.yaml` (see the file for a complete example with all available options)

2) Run:

```bash
uv run python src/dataset_builder/main.py --config config/pipeline.yaml
```

Tips:
- Override any setting with `-s key=value` (see module READMEs for keys).
- Start small (few questions, small batch sizes). Costs scale with tokens.

### Dataset Format (Builder)

Models are defined in `src/dataset_builder/common/models.py` and documented in `src/dataset_builder/common/README.md`.

Minimal Easy example:

```json
{
  "document_path": "document.md",
  "question": "What is a saving throw?",
  "answer": "A saving throw is a d20 roll...",
  "passages": [
    { "start_sequence": "# Saving Throws", "end_sequence": "add your proficiency bonus." }
  ]
}
```

Medium examples come in two formats:

**Format 1: Retrieved format** (raw output from Claude Skills / LLM Retriever):

```json
{
  "id": 1,
  "question": "Which race best resists a Red Dragon's breath?",
  "answer": "Based on fire resistance...",
  "retrieved": [
    {
      "document_path": "DND5eSRD_313-332.md",
      "start_sequence": "# Red Dragons",
      "end_sequence": "Half damage.",
      "start_char": 22568,
      "end_char": 29038,
      "reconstructed_passage": "Full passage text..."
    }
  ]
}
```

**Format 2: Passages format** (standardized, after postprocessing):

```json
{
  "id": 1,
  "question": "Which race best resists a Red Dragon's breath?",
  "answer": "Based on fire resistance...",
  "passages": [
    {
      "content": "Full passage text...",
      "document_path": "DND5eSRD_313-332.md",
      "start_char": 22568,
      "end_char": 29038
    }
  ]
}
```

Use the `postprocess_medium` stage to convert from `retrieved` format to `passages` format. The passages format matches the Easy tier structure and is more suitable for downstream evaluation tasks.

Question types:
- **multi_hop** → answered by Claude Skills
- **wide** → answered by LLM Retriever

### Project Structure

```
rag-evaluation/
├── dataset/                    # The pre-built D&D 5e SRD dataset
│   ├── qa_sets/               # Question-answer pairs (JSON & Parquet)
│   └── raw_data/              # Source markdown documents
├── src/dataset_builder/       # Tools to build your own datasets
│   ├── main.py
│   ├── preprocess_pdf_parser/
│   ├── easy_answers_questions_generator/
│   ├── medium_answers_claude_skill/
│   ├── medium_answers_llm_retriever/
│   └── common/
├── config/                    # Pipeline configuration files
└── README.md
```

### Costs and Limits

Dataset generation can be expensive (long contexts, tool-use). 

> **⚠️ Important**: The medium tier using Claude Skills is particularly expensive. For reference, our 31-question medium dataset cost approximately **$100** to generate (including reruns with hints) due to Claude's agentic capabilities and long context requirements.

Control spend with:
- Smaller `ids` subsets, lower `batch_size`, conservative `max_tokens`
- Concurrency caps (retriever: `doc_concurrency`)
- Incremental runs (`only_missing: true`)

### Troubleshooting

- API rate limits → lower `batch_size`/`doc_concurrency`
- PDF parsing → verify `--breaks`, inspect extraction quality
- Missing passages → adjust start/end sequences or chunking strategy

## License

### Code (MIT License)
The source code and dataset builder tools in this repository are licensed under the **MIT License**. See the [LICENSE](https://github.com/datapizza-labs/rag-dataset-builder/tree/HEAD/LICENSE) file for details.

### Dataset (CC BY 4.0)
The dataset in the `dataset/` directory is licensed under the **Creative Commons Attribution 4.0 International License (CC BY 4.0)**.

The dataset is derived from the Dungeons & Dragons 5th Edition System Reference Document (SRD) version 5.2.1, which is available under CC BY 4.0 from Wizards of the Coast.

**When using the dataset, please provide appropriate attribution:**
```
D&D 5e SRD RAG Evaluation Dataset by Datapizza AI Lab
Based on the D&D 5e System Reference Document by Wizards of the Coast
Licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
```

## Citation

If you use this repository or dataset in your research, please cite:

```bibtex
@misc{datapizza_qa_dataset_builder_rag_evaluation,
  author = {Singh, Raul and Chen, Ling Xuan Emma and Foresi, Francesco},
  title = {D\&D 5e SRD QA RAG Evaluation Dataset + Dataset Builder},
  year = {2025},
  url = {https://github.com/datapizza-labs/rag-dataset-builder},
  note = {Dataset also available at \url{https://huggingface.co/datasets/datapizza-ai-lab/dnd5e-srd-qa}}
}
```

For dataset-only citations, you can also use:

```bibtex
@misc{datapizza_dnd_5_2_1_qa_dataset,
  author = {Singh, Raul and Chen, Ling Xuan Emma and Foresi, Francesco},
  title = {D\&D 5e SRD QA RAG Evaluation Dataset},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/datapizza-ai-lab/dnd5e-srd-qa}},
  note = {A high-quality Question-Answering dataset built from the D\&D 5th Edition System Reference Document for evaluating RAG systems}
}
```

## Acknowledgments

- Built with [`datapizza-ai`](https://github.com/datapizza-labs/datapizza-ai)
- Powered by Google Gemini and Anthropic Claude
- This work includes material from the System Reference Document 5.2.1 ("SRD 5.2.1") by Wizards of the Coast LLC, available at https://www.dndbeyond.com/srd. The SRD 5.2.1 is licensed under the Creative Commons Attribution 4.0 International License, available at https://creativecommons.org/licenses/by/4.0/legalcode.