---
slug: "rlm-replication-study"
source_type: "readme"
source_url: "https://cdn.jsdelivr.net/gh/Cornjebus/rlm-replication-study@main/README.md"
repo: "https://github.com/Cornjebus/rlm-replication-study"
source_file: "README.md"
branch: "main"
---
# Stateful Knowledge Graph Traversal vs. Recursive Language Models

A replication study demonstrating that explicit state management and termination conditions outperform the RLM approach on long-context reasoning tasks.

## Results on Real OOLONG Dataset

**We evaluated on the trec_coarse + spam subsets of the OOLONG validation split** (558 of 1,300 examples). We restrict to these subsets because our parser targets the `Date || User || Instance || Label` format.

### Results Comparison

| Metric | Skill-Based (Ours) | RLM (Zhang et al., 2024) |
|--------|-------------------|--------------------------|
| **Accuracy/F1*** | **87.8%** exact-match (490/558) | 23-58% F1 |
| Indexing cost | ~1 sec (regex parse) | None |
| Query-time LLM calls | **0** | 10-1000+ |
| Query latency | **0.26ms** avg | seconds-minutes |
| Termination | **Deterministic** | Brittle |
| Enforced state tracking | **Yes** | No |

*\*We report exact-match accuracy (categorical/integer answers); RLM reports F1 (set-like answers). Comparison is directional.*

### Breakdown by Dataset

| Dataset | Our Accuracy | Notes |
|---------|-------------|-------|
| trec_coarse | **99.2%** (248/250) | Question classification |
| spam | **78.6%** (242/308) | SMS spam detection |

### Breakdown by Task Type

| Task Type | Accuracy |
|-----------|----------|
| MOST_FREQ | 91.6% (98/107) |
| LEAST_FREQ | 90.3% (56/62) |
| RELATIVE_FREQ | 91.2% (208/228) |
| NUMERIC_ONE_CLASS | 78.3% (108/138) |
| SECOND_MOST_FREQ | 90.9% (10/11) |
| REPRESENTED_N_TIMES | 83.3% (10/12) |

### Why This Matters

RLM reports F1 scores of 23-58% on OOLONG variants, requiring:
- Hundreds to thousands of LLM calls per query
- Seconds to minutes of processing time
- No enforced termination (model must invent stopping conditions)

Our approach achieves **87.8% exact-match accuracy** on these subsets with:
- **Zero LLM calls** at query time (parsing is regex-based)
- **0.26ms** average query time (6 orders of magnitude faster)
- **Deterministic termination** (impossible to loop infinitely)

### Error Analysis (68 failures)

When we fail, it's due to parsing, not reasoning:
- Timeline/date queries: 42 (not yet implemented)
- User-label cross queries: 15 (complex join patterns)
- Format variance: 8
- Tie-breaking: 3

## Synthetic Benchmark Results

We also tested on a synthetic OOLONG-style benchmark (1,000 docs, 50 queries):

| Metric | Result |
|--------|--------|
| Accuracy | **100%** (50/50) |
| Query time | **0.16ms** avg |
| LLM calls | **0** |

See [Running the Benchmarks](#quick-start) for reproduction instructions.

## Paper

See [REPLICATION_STUDY.md](https://github.com/Cornjebus/rlm-replication-study/blob/HEAD/REPLICATION_STUDY.md) for the full academic writeup including:
- Methodology
- Experimental setup
- Detailed results
- Analysis of RLM failure modes
- Reproducibility instructions

## Quick Start

### Run against real OOLONG dataset (recommended)

```bash
# 1. Install dependencies
pip3 install datasets pandas pyarrow

# 2. Download OOLONG data (downloads ~1MB validation split)
python3 -c "
import requests, os
os.makedirs('benchmark_data/oolong', exist_ok=True)
for i in range(3):
    url = f'https://huggingface.co/datasets/oolongbench/oolong-synth/resolve/main/data/validation-0000{i}-of-00007.parquet'
    r = requests.get(url, verify=False)
    open(f'benchmark_data/oolong/validation-{i}.parquet', 'wb').write(r.content)
print('Downloaded OOLONG validation split')
"

# 3. Run OOLONG benchmark
python3 oolong_benchmark.py
```

### Run against synthetic benchmark

```bash
# 1. Generate benchmark data (1000 documents, 50 queries)
python3 generate_benchmark.py --records 1000 --queries 50

# 2. Build knowledge graph
python3 build_graph.py

# 3. Run benchmark
python3 run_full_benchmark.py

# 4. Verify results independently
python3 verify_results.py
```

## Repository Structure

```
rlm_replication/
├── README.md                    # This file
├── REPLICATION_STUDY.md         # Academic paper
├── oolong_benchmark.py          # Real OOLONG dataset benchmark
├── generate_benchmark.py        # Creates OOLONG-style test data
├── build_graph.py              # Builds knowledge graph from corpus
├── run_full_benchmark.py       # Runs benchmark with detailed output
├── verify_results.py           # Independent verification
├── run_benchmark.py            # Original benchmark runner
├── recursive-knowledge/        # The skill implementation
│   ├── SKILL.md               # Skill definition
│   ├── scripts/
│   │   ├── graph_ops.py       # Knowledge graph data structures
│   │   ├── query.py           # Stateful query engine
│   │   └── index_corpus.py    # Document indexer
│   └── references/
│       ├── graph-schema.md
│       ├── state-management.md
│       └── traversal-patterns.md
└── benchmark_data/             # Generated after running scripts
    ├── oolong/                # Real OOLONG dataset files
    │   └── *.parquet          # Downloaded parquet files
    ├── oolong_results.json    # OOLONG benchmark results
    ├── corpus/                # Synthetic documents
    ├── corpus.json            # Structured metadata
    ├── graph_structured.json  # Knowledge graph
    └── queries.json           # Test queries with ground truth
```

## Key Insight

The RLM paper ([arXiv:2512.24601v1](https://arxiv.org/abs/2512.24601)) documents these limitations:

> "RLM(Qwen3-Coder) made hundreds to thousands of recursive sub-calls for a single simple task" (Section 3.1)

> "The model tries to reproduce its correct answer more than five times before choosing the incorrect answer in the end" (Example B.3)

> "Distinguishing between a final answer and a thought is brittle" (Section 5)

These failures stem from **lack of enforced state management**. While RLM's REPL *can* store variables, any state structure must be invented by the model—and failures arise when it is not.

Our approach adds **guaranteed** state tracking:

```python
class TraversalState:
    visited_nodes: set[str]    # Prevents re-exploration
    visited_edges: set[str]    # Prevents redundant traversal
    confidence: float          # Knows when to stop

    def should_stop(self):
        if self.confidence >= 0.85: return True
        if self.current_depth >= max_depth: return True
        return False
```

This makes infinite loops and re-verification errors **impossible by construction**.

## Running on Real Datasets

To run a true apples-to-apples comparison on the exact datasets from the RLM paper:

### OOLONG Dataset

```bash
# Install dependencies
pip3 install datasets

# Download OOLONG
python3 -c "
from datasets import load_dataset
ds = load_dataset('oolongbench/oolong-synth', split='trec_coarse')
ds.to_json('benchmark_data/oolong.json')
"

# Index with LLM extraction (requires ANTHROPIC_API_KEY)
export USE_LLM=1
python3 recursive-knowledge/scripts/index_corpus.py \
    --input benchmark_data/oolong.json \
    --output benchmark_data/oolong_graph.json \
    --verbose

# Query
python3 recursive-knowledge/scripts/query.py \
    --graph benchmark_data/oolong_graph.json \
    --query "your question here" \
    --verbose
```

### BrowseComp-Plus Dataset

```bash
# Download BrowseComp-Plus (100K documents, ~15GB)
python3 -c "
from datasets import load_dataset
ds = load_dataset('Tevatron/browsecomp-plus-corpus')
ds['corpus'].to_json('benchmark_data/browsecomp.json')
"

# Note: This dataset is large. Consider sampling:
python3 -c "
import json
with open('benchmark_data/browsecomp.json') as f:
    docs = [json.loads(line) for line in f][:1000]  # First 1000
with open('benchmark_data/browsecomp_sample.json', 'w') as f:
    for doc in docs:
        f.write(json.dumps(doc) + '\n')
"
```

### Requirements for Real Dataset Testing

1. **Disk Space**: BrowseComp-Plus requires ~15GB
2. **API Key**: LLM-based entity extraction requires `ANTHROPIC_API_KEY`
3. **Time**: Indexing 100K documents takes hours with LLM extraction

### Expected Behavior Differences

On real datasets, you may observe:
- Lower accuracy if entity extraction misses domain-specific terms
- Need to tune confidence thresholds for different query complexity
- Multi-hop queries may require higher `max_depth` settings

The architectural advantage (stateful traversal vs. stateless iteration) remains constant regardless of dataset.

## Citation

If you use this work, please cite:

```bibtex
@misc{rlm_replication_2024,
  title={Stateful Knowledge Graph Traversal Outperforms Recursive Language Models},
  author={[Your Name]},
  year={2024},
  note={Replication study of arXiv:2512.24601v1}
}
```

## References

Zhang, A., et al. (2024). Recursive Language Models. arXiv:2512.24601v1.

## License

MIT