--- slug: "llm-reproducibility" source_type: "readme" source_url: "https://cdn.jsdelivr.net/gh/saross/llm-reproducibility@main/README.md" repo: "https://github.com/saross/llm-reproducibility" source_file: "README.md" branch: "main" --- # LLM-Based Research Extraction and Assessment [![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) [![License: CC BY 4.0](https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/) Automated extraction of claims, evidence, and methodology from research papers using Large Language Models, enabling systematic assessment of research transparency, reproducibility, and credibility. **Version:** 3.0.1 | **Schema:** v2.6 | **Workflow:** v5.0.0 (8-pass, session-per-pass) **Status:** Extraction system complete, assessment framework pilot-tested, reproduction system v1.1 **Target Domains:** Fieldwork-based research (archaeology, ecology, ethnography, field geology, etc.) **Manifest:** See [manifest.yaml](https://github.com/saross/llm-reproducibility/blob/HEAD/manifest.yaml) for component versions --- ## Overview This repository contains tools and workflows for: 1. **Extraction** (Complete): Systematically extract claims, evidence, implicit arguments, research designs, methods, and protocols from research papers 2. **Assessment** (Pilot-tested): Evaluate research transparency, replicability, and credibility via research-approach classification, quality gating, and the repliCATS Seven Signals adapted for Humanities, Arts, and Social Sciences (HASS) 3. **Reproduction** (v1.1): Docker-based reproduction of published computational analyses, with quantitative verification and adversarial review The pipeline uses multi-pass workflows with Claude (Anthropic) to create structured JSON representations of research papers, enabling downstream analysis, credibility assessment, and reproduction attempts. All three systems were validated in a completed pilot study of five Journal of Archaeological Science papers (see `studies/open-science-compliance/`). --- ## Quick Start ### 1. Install the Extraction Skill ```bash # Install the research-assessor skill in Claude Code # The skill is located at: .claude/skills/research-assessor/ ``` See [docs/research-assessor-guide/installation.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/installation.md) for detailed installation instructions. ### 2. Prepare Your Paper ```bash # Extract text from PDF python extraction-system/scripts/extract_pdf_text.py "your-paper.pdf" # Output will be in: sources/processed-md/your-paper.md ``` See [docs/user-guide/pdf-extraction.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/pdf-extraction.md) for more details. ### 3. Run Extraction Use the eight-pass workflow (Pass 0-7): 0. **Metadata Pass**: Extract publication metadata and paper structure 1. **Claims/Evidence Pass 1**: Liberal extraction of evidence, claims, implicit arguments 2. **Claims/Evidence Pass 2**: Rationalisation and consolidation 3. **RDMAP Pass 1a**: Liberal extraction of research designs, methods, protocols 4. **RDMAP Pass 1b**: Implicit RDMAP extraction 5. **RDMAP Pass 2**: Rationalisation and tier verification 6. **Infrastructure Pass**: Extract PIDs, FAIR assessment, funding, permits 7. **Validation Pass**: Structural integrity checks See [docs/user-guide/extraction-workflow.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/extraction-workflow.md) for complete workflow guide. --- ## Repository Structure ```text llm-reproducibility/ ├── input/ # Workflow entry points (queue, launch prompt, workflow) ├── extraction-system/ # All extraction tools (skill, prompts, schema, scripts) ├── assessment-system/ # Assessment tools (pilot-tested) ├── reproduction-system/ # Docker-based reproduction workflow (prompts, templates) ├── docs/ # User-facing documentation (guides, skill docs, background) ├── wiki/ # Research process record (continuity, lab notebook, planning) ├── studies/ # Research studies using the pipeline (open-science-compliance) ├── examples/ # Curated extraction examples ├── outputs/ # Extraction outputs organised by paper ├── reports/ # Key testing and QA reports ├── archive/ # Development history organised by version ├── manifest.yaml # Component version manifest (source of truth) ├── LICENSE-CODE # Apache 2.0 (code) ├── LICENSE-DOCS # CC-BY-4.0 International (documentation) └── CITATION.cff # Machine-readable citation ``` ### Key Directories - **[input/](https://github.com/saross/llm-reproducibility/tree/HEAD/input/)** - Workflow entry points - `workflow.md` - Authoritative 8-pass extraction workflow - `extraction-launch.md` - Quick-start primer for new extractions - `queue.yaml` - Paper processing queue with status tracking - **[extraction-system/](https://github.com/saross/llm-reproducibility/tree/HEAD/extraction-system/)** - Complete extraction toolkit - `prompts/` - Eight extraction prompts (Pass 0-7) - `schema/` - JSON schema for extraction output (v2.6) - `scripts/` - PDF text extraction and validation utilities - `extraction-plan-unified-model.md` - Flexible planning guidance - **[docs/](https://github.com/saross/llm-reproducibility/tree/HEAD/docs/)** - User-facing documentation: how to install and use the systems - `user-guide/` - Getting started, workflow, schema reference - `research-assessor-guide/` - Comprehensive skill documentation - `schema/` - Schema versioning and crosswalks - `background-research/` - Deep research reports - **[wiki/](https://github.com/saross/llm-reproducibility/tree/HEAD/wiki/)** - Versioned research process record: how the research is being done - `continuity.md` - Cross-session state and session log - `working-notes.md` - Empirical lab notebook - `reflections/` - Meta-research observations - `planning/` - Implementation plans and roadmaps - Kept in-repo (not a GitHub Wiki) so the process record is versioned and archived with the code — see `wiki/index.md` > **docs/ vs wiki/:** `docs/` documents the *product* (for users of the > extraction/assessment/reproduction systems); `wiki/` documents the *process* > (for provenance and project continuity). If you want to use the tools, start > in `docs/`; if you want to see how this project works day to day, start in > `wiki/`. - **[examples/](https://github.com/saross/llm-reproducibility/tree/HEAD/examples/)** - Example extractions - Complete extraction samples with annotations - Blank templates for new extractions - Quick reference guides - **[archive/](https://github.com/saross/llm-reproducibility/tree/HEAD/archive/)** - Development history - Organized by version (v2.0-v2.1 → v2.5) - Planning, documentation, reports, test outputs --- ## Features ### Extraction System (v2.6) **Six Object Types:** - Evidence - Observations, measurements, data points - Claims - Interpretations, assertions, generalizations - Implicit Arguments - Unstated assumptions, logical implications - Research Designs - Strategic framing and rationale (WHY) - Methods - Tactical approaches (WHAT was done) - Protocols - Operational procedures (HOW specifically) **Multi-Pass Workflow (8 passes, 0-7):** - Pass 0: Metadata extraction - Passes 1-2: Claims/Evidence (liberal extraction → rationalisation) - Passes 3-5: RDMAP (liberal → implicit → rationalisation) - Pass 6: Infrastructure (PIDs, FAIR, funding, permits) - Pass 7: Validation ensuring structural integrity **Provenance Tracking:** - Every item linked to source text via verbatim quotes - Location tracking (section, page, paragraph) - Cross-references between all object types - Complete consolidation metadata for traceability **Fieldwork-Specific:** - Respects abductive/inductive reasoning - Tracks opportunistic decisions and adaptations - Distinguishes planned vs. actual procedures - Documents emergent discoveries ### Assessment Framework (Pilot-tested, v1.0) Three Pillars framework (Transparency → Credibility → Reproducibility) with the repliCATS Seven Signals adapted for HASS disciplines: - **Pass 8 — Classification:** research approach (inductive/deductive/abductive) with HARKing detection - **Track A — Quality gating:** extraction quality checks before assessment - **Pass 9 — Seven Signals in three clusters:** Foundational Clarity (Comprehensibility, Transparency); Evidential Strength (Plausibility, Validity, Robustness, Generalisability); Reproducibility Reliability evidence from a 25-run variability test (5 papers × 5 runs): 100% classification stability, 96% verdict stability, aggregate-score coefficients of variation 1.9–3.4% (see [outputs/variability-test/](https://github.com/saross/llm-reproducibility/tree/HEAD/outputs/variability-test/)). See [wiki/planning/credibility-implementation-plan-v2.0.md](https://github.com/saross/llm-reproducibility/blob/HEAD/wiki/planning/credibility-implementation-plan-v2.0.md) for the framework design. ### Reproduction System (v1.1) Docker-based reproduction of published computational analyses (R-based papers; Python/Julia extension points documented): - **Four-session workflow:** planning (R-Plan), preparation (R-A), execution and verification (R-B), adversarial review (R-C, run in a fresh context) - **Paper-type classification** (batch-ready/Dockerfile through interactive/proprietary) driving effort estimation - **Quantitative verification** against every table, figure, and inline statistic, with a six-category discrepancy taxonomy including PAPER_ERROR (reproduction can surface errors in the published paper itself) - **Verdicts:** SUCCESSFUL / PARTIAL / FAILED / BLOCKED Pilot results (5 JAS papers): 4 SUCCESSFUL, 1 PARTIAL (sole cause: data inaccessibility), including exact reproduction of deterministic analyses and discovery of two calculation errors in a published table. See [reproduction-system/](https://github.com/saross/llm-reproducibility/tree/HEAD/reproduction-system/) and the pilot findings report in [studies/open-science-compliance/reports/](https://github.com/saross/llm-reproducibility/tree/HEAD/studies/open-science-compliance/reports/). --- ## Documentation ### User Guides - [Getting Started](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/getting-started.md) - Installation and first extraction - [Extraction Workflow](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/extraction-workflow.md) - Complete 8-pass workflow (Pass 0-7) - [PDF Extraction](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/pdf-extraction.md) - Preparing papers for analysis - [Schema Reference](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/schema-reference.md) - Understanding the extraction schema ### Research Assessor Skill Guide - [Overview](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/README.md) - What the skill does and when to use it - [Installation](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/installation-guide.md) - Installing and verifying the skill - [Usage Guide](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/usage-guide.md) - Complete usage reference - [Architecture](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/architecture.md) - How the skill works - [Quick Reference](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/quick-reference.md) - Cheat sheet for common tasks ### Reproduction System - [Overview](https://github.com/saross/llm-reproducibility/blob/HEAD/reproduction-system/README.md) - Docker-based reproduction workflow - [Planning Guide](https://github.com/saross/llm-reproducibility/blob/HEAD/reproduction-system/reproduction-plan-guide.md) - Paper-specific adaptation - [Launch Primer](https://github.com/saross/llm-reproducibility/blob/HEAD/reproduction-system/reproduction-launch.md) - Quick-start for reproductions ### Schema and Development - [Schema Reference](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/schema-reference.md) - Current schema documentation - [Version History](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/version.md) - Skill version information --- ## Validation The pipeline has been validated at three levels: - **Extraction corpus:** 11 papers fully extracted (archaeology, ancient history, environmental science; see [input/queue.yaml](https://github.com/saross/llm-reproducibility/blob/HEAD/input/queue.yaml)). A run comparison demonstrated the session-per-pass workflow captures 75% more claims and 100% more research designs than single-session extraction. - **Reliability:** 25-run variability test (5 papers × 5 context-cleared runs) — extraction counts vary between runs, but assessments are stable: 100% classification stability, 96% verdict stability, aggregate-score CV 1.9–3.4%. Conclusion: a single run suffices for credibility assessment. See [outputs/variability-test/](https://github.com/saross/llm-reproducibility/tree/HEAD/outputs/variability-test/). - **End-to-end pilot study:** 5 Journal of Archaeological Science papers through the complete pipeline (extraction → assessment → FAIR scoring → Docker reproduction), 4 SUCCESSFUL / 1 PARTIAL reproductions. See [studies/open-science-compliance/](https://github.com/saross/llm-reproducibility/tree/HEAD/studies/open-science-compliance/). Earlier v2.5-era extraction-quality reports remain in [reports/extraction-testing/](https://github.com/saross/llm-reproducibility/tree/HEAD/reports/extraction-testing/). --- ## Dependencies **Extraction and Assessment:** - Claude with Skills support (via Claude Code) - Python 3.8+ for PDF extraction scripts **Reproduction System:** - Docker (all reproductions execute in containers) - R papers use rocker base images; no host R installation required **PDF Processing:** - PyMuPDF (fitz) - Text extraction - pdfplumber - Table extraction - PyYAML - Configuration - python-slugify - Filename handling ```bash pip install -r requirements.txt ``` --- ## Project Status ### ✅ Completed (v3.0.1) - Extraction system complete and validated (8-pass workflow, Pass 0-7; schema v2.6; 11-paper corpus) - Assessment framework pilot-tested (Pass 8-9: classification, quality gating, Seven Signals) - Reproduction system v1.1 (Docker-based, four-session workflow, adversarial review) - Phase 1 pilot study complete: 5 JAS papers through the full pipeline (4 SUCCESSFUL / 1 PARTIAL reproductions) - Reliability established (25-run variability test: assessments stable across runs) - FAIR scoring standardised (15 binary GO-FAIR sub-principles, data and code scored independently) - Repository organisation, FAIR4RS preparation, and wiki-layout process record ### 🚧 In Progress - Agentic modernisation: subagent/workflow implementation of the reproduction and FAIR lanes (see [wiki/planning/agentic-modernisation-plan.md](https://github.com/saross/llm-reproducibility/blob/HEAD/wiki/planning/agentic-modernisation-plan.md)) - OSF preregistration of Phase 2 hypotheses (five candidates drafted from pilot findings) ### 📋 Planned (Next Phase) - JAS 2023-2026 corpus: descriptive FAIR census + preregistered confirmatory reproduction study (census doubles as the sampling frame) - Python/Julia reproduction support (extension points documented) - Zenodo deposition and DOI assignment --- ## Contributing We welcome contributions in several areas: - **Domain expansion**: Adapt extraction to new fieldwork disciplines - **Vocabulary development**: Refine controlled vocabularies empirically - **Testing**: Run extractions on new papers, report issues - **Examples**: Contribute worked extractions from other domains - **Assessment**: Help develop credibility assessment rubrics - **Documentation**: Improve guides and fix errors See [CONTRIBUTING.md](https://github.com/saross/llm-reproducibility/blob/HEAD/CONTRIBUTING.md) for detailed guidelines. --- ## Citation If you use this work in research, please cite: ```bibtex @software{llm-reproducibility, title = {LLM-Based Research Extraction and Assessment}, author = {[Author information in CITATION.cff]}, version = {3.0.1}, year = {2026}, url = {[Repository URL in CITATION.cff]} } ``` See [CITATION.cff](https://github.com/saross/llm-reproducibility/blob/HEAD/CITATION.cff) for machine-readable citation. --- ## Acknowledgments - Developed with Anthropic's Claude (Sonnet 4.5 through the Claude 5 family) - Tested on archaeological research papers - Informed by TIDieR, CONSORT-Outcomes, and SPIRIT frameworks - Assessment framework adapted from repliCATS methodology - Inspired by research transparency and reproducibility initiatives --- ## Support - **Documentation**: See [docs/](https://github.com/saross/llm-reproducibility/tree/HEAD/docs/) for comprehensive guides - **Issues**: Report issues via GitHub Issues - **Questions**: See [docs/user-guide/getting-started.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/getting-started.md) --- ## Development History This project has evolved through multiple versions: - **v2.0-2.1** (Oct 16, 2025): Initial claims/evidence extraction - **v2.2-2.3** (Oct 17-18, 2025): Two-pass workflow with consolidation - **v2.4** (Oct 19-20, 2025): RDMAP extraction framework added - **v2.5** (Oct 23, 2025): Repository rationalisation and FAIR4RS preparation - **v2.6** (Nov 2025): Infrastructure extraction (Pass 6), 8-pass workflow complete (Pass 0-7), assessment framework development - **v2.7** (Jan 2026): Session-per-pass execution mode (+75% claims, +100% research designs vs autonomous) - **v2.8-2.9** (Feb 2026): Reproduction system v1.0 → v1.1, validated on 5 pilot papers - **v3.0** (Feb 2026): Schema v2.6 alignment; assessment system promoted to pilot-tested; JAS pilot study complete - **v3.0.1** (Jul 2026): Metadata reconciliation; wiki-layout migration; compound-aware de-hyphenation; agentic modernisation plan See [archive/README.md](https://github.com/saross/llm-reproducibility/blob/HEAD/archive/README.md) and [manifest.yaml](https://github.com/saross/llm-reproducibility/blob/HEAD/manifest.yaml) for complete development history. --- ## Licence This project uses dual licensing: - **Code** (scripts, validation tools): [Apache-2.0](https://github.com/saross/llm-reproducibility/tree/HEAD/LICENSE-CODE) - **Documentation** (guides, prompts, examples): [CC-BY-4.0](https://github.com/saross/llm-reproducibility/tree/HEAD/LICENSE-DOCS) See [LICENSE-CODE](https://github.com/saross/llm-reproducibility/tree/HEAD/LICENSE-CODE) and [LICENSE-DOCS](https://github.com/saross/llm-reproducibility/tree/HEAD/LICENSE-DOCS) for full licence texts. --- **Ready to extract!** See [docs/user-guide/getting-started.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/getting-started.md) to begin.