llm-reproducibility

内容来源：README.md（说明文档） · 原始地址 · 查看安装指南

原始内容

# LLM-Based Research Extraction and Assessment

[![License: Apache-2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![License: CC BY 4.0](https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/)

Automated extraction of claims, evidence, and methodology from research papers using Large Language Models, enabling systematic assessment of research transparency, reproducibility, and credibility.

**Version:** 3.0.1 | **Schema:** v2.6 | **Workflow:** v5.0.0 (8-pass, session-per-pass)
**Status:** Extraction system complete, assessment framework pilot-tested, reproduction system v1.1
**Target Domains:** Fieldwork-based research (archaeology, ecology, ethnography, field geology, etc.)
**Manifest:** See [manifest.yaml](https://github.com/saross/llm-reproducibility/blob/HEAD/manifest.yaml) for component versions

---

## Overview

This repository contains tools and workflows for:

1. **Extraction** (Complete): Systematically extract claims, evidence, implicit arguments, research designs, methods, and protocols from research papers
2. **Assessment** (Pilot-tested): Evaluate research transparency, replicability, and credibility via research-approach classification, quality gating, and the repliCATS Seven Signals adapted for Humanities, Arts, and Social Sciences (HASS)
3. **Reproduction** (v1.1): Docker-based reproduction of published computational analyses, with quantitative verification and adversarial review

The pipeline uses multi-pass workflows with Claude (Anthropic) to create structured JSON representations of research papers, enabling downstream analysis, credibility assessment, and reproduction attempts. All three systems were validated in a completed pilot study of five Journal of Archaeological Science papers (see `studies/open-science-compliance/`).

---

## Quick Start

### 1. Install the Extraction Skill

```bash
# Install the research-assessor skill in Claude Code
# The skill is located at: .claude/skills/research-assessor/
```

See [docs/research-assessor-guide/installation.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/installation.md) for detailed installation instructions.

### 2. Prepare Your Paper

```bash
# Extract text from PDF
python extraction-system/scripts/extract_pdf_text.py "your-paper.pdf"

# Output will be in: sources/processed-md/your-paper.md
```

See [docs/user-guide/pdf-extraction.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/pdf-extraction.md) for more details.

### 3. Run Extraction

Use the eight-pass workflow (Pass 0-7):

0. **Metadata Pass**: Extract publication metadata and paper structure
1. **Claims/Evidence Pass 1**: Liberal extraction of evidence, claims, implicit arguments
2. **Claims/Evidence Pass 2**: Rationalisation and consolidation
3. **RDMAP Pass 1a**: Liberal extraction of research designs, methods, protocols
4. **RDMAP Pass 1b**: Implicit RDMAP extraction
5. **RDMAP Pass 2**: Rationalisation and tier verification
6. **Infrastructure Pass**: Extract PIDs, FAIR assessment, funding, permits
7. **Validation Pass**: Structural integrity checks

See [docs/user-guide/extraction-workflow.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/extraction-workflow.md) for complete workflow guide.

---

## Repository Structure

```text
llm-reproducibility/
├── input/                 # Workflow entry points (queue, launch prompt, workflow)
├── extraction-system/     # All extraction tools (skill, prompts, schema, scripts)
├── assessment-system/     # Assessment tools (pilot-tested)
├── reproduction-system/   # Docker-based reproduction workflow (prompts, templates)
├── docs/                  # User-facing documentation (guides, skill docs, background)
├── wiki/                  # Research process record (continuity, lab notebook, planning)
├── studies/               # Research studies using the pipeline (open-science-compliance)
├── examples/              # Curated extraction examples
├── outputs/               # Extraction outputs organised by paper
├── reports/               # Key testing and QA reports
├── archive/               # Development history organised by version
├── manifest.yaml          # Component version manifest (source of truth)
├── LICENSE-CODE           # Apache 2.0 (code)
├── LICENSE-DOCS           # CC-BY-4.0 International (documentation)
└── CITATION.cff           # Machine-readable citation
```

### Key Directories

- **[input/](https://github.com/saross/llm-reproducibility/tree/HEAD/input/)** - Workflow entry points
  - `workflow.md` - Authoritative 8-pass extraction workflow
  - `extraction-launch.md` - Quick-start primer for new extractions
  - `queue.yaml` - Paper processing queue with status tracking

- **[extraction-system/](https://github.com/saross/llm-reproducibility/tree/HEAD/extraction-system/)** - Complete extraction toolkit
  - `prompts/` - Eight extraction prompts (Pass 0-7)
  - `schema/` - JSON schema for extraction output (v2.6)
  - `scripts/` - PDF text extraction and validation utilities
  - `extraction-plan-unified-model.md` - Flexible planning guidance

- **[docs/](https://github.com/saross/llm-reproducibility/tree/HEAD/docs/)** - User-facing documentation: how to install and use the systems
  - `user-guide/` - Getting started, workflow, schema reference
  - `research-assessor-guide/` - Comprehensive skill documentation
  - `schema/` - Schema versioning and crosswalks
  - `background-research/` - Deep research reports

- **[wiki/](https://github.com/saross/llm-reproducibility/tree/HEAD/wiki/)** - Versioned research process record: how the research is being done
  - `continuity.md` - Cross-session state and session log
  - `working-notes.md` - Empirical lab notebook
  - `reflections/` - Meta-research observations
  - `planning/` - Implementation plans and roadmaps
  - Kept in-repo (not a GitHub Wiki) so the process record is versioned and
    archived with the code — see `wiki/index.md`

> **docs/ vs wiki/:** `docs/` documents the *product* (for users of the
> extraction/assessment/reproduction systems); `wiki/` documents the *process*
> (for provenance and project continuity). If you want to use the tools, start
> in `docs/`; if you want to see how this project works day to day, start in
> `wiki/`.

- **[examples/](https://github.com/saross/llm-reproducibility/tree/HEAD/examples/)** - Example extractions
  - Complete extraction samples with annotations
  - Blank templates for new extractions
  - Quick reference guides

- **[archive/](https://github.com/saross/llm-reproducibility/tree/HEAD/archive/)** - Development history
  - Organized by version (v2.0-v2.1 → v2.5)
  - Planning, documentation, reports, test outputs

---

## Features

### Extraction System (v2.6)

**Six Object Types:**
- Evidence - Observations, measurements, data points
- Claims - Interpretations, assertions, generalizations
- Implicit Arguments - Unstated assumptions, logical implications
- Research Designs - Strategic framing and rationale (WHY)
- Methods - Tactical approaches (WHAT was done)
- Protocols - Operational procedures (HOW specifically)

**Multi-Pass Workflow (8 passes, 0-7):**
- Pass 0: Metadata extraction
- Passes 1-2: Claims/Evidence (liberal extraction → rationalisation)
- Passes 3-5: RDMAP (liberal → implicit → rationalisation)
- Pass 6: Infrastructure (PIDs, FAIR, funding, permits)
- Pass 7: Validation ensuring structural integrity

**Provenance Tracking:**
- Every item linked to source text via verbatim quotes
- Location tracking (section, page, paragraph)
- Cross-references between all object types
- Complete consolidation metadata for traceability

**Fieldwork-Specific:**
- Respects abductive/inductive reasoning
- Tracks opportunistic decisions and adaptations
- Distinguishes planned vs. actual procedures
- Documents emergent discoveries

### Assessment Framework (Pilot-tested, v1.0)

Three Pillars framework (Transparency → Credibility → Reproducibility) with the
repliCATS Seven Signals adapted for HASS disciplines:

- **Pass 8 — Classification:** research approach (inductive/deductive/abductive)
  with HARKing detection
- **Track A — Quality gating:** extraction quality checks before assessment
- **Pass 9 — Seven Signals in three clusters:** Foundational Clarity
  (Comprehensibility, Transparency); Evidential Strength (Plausibility,
  Validity, Robustness, Generalisability); Reproducibility

Reliability evidence from a 25-run variability test (5 papers × 5 runs):
100% classification stability, 96% verdict stability, aggregate-score
coefficients of variation 1.9–3.4% (see
[outputs/variability-test/](https://github.com/saross/llm-reproducibility/tree/HEAD/outputs/variability-test/)).

See [wiki/planning/credibility-implementation-plan-v2.0.md](https://github.com/saross/llm-reproducibility/blob/HEAD/wiki/planning/credibility-implementation-plan-v2.0.md) for the framework design.

### Reproduction System (v1.1)

Docker-based reproduction of published computational analyses (R-based papers;
Python/Julia extension points documented):

- **Four-session workflow:** planning (R-Plan), preparation (R-A), execution and
  verification (R-B), adversarial review (R-C, run in a fresh context)
- **Paper-type classification** (batch-ready/Dockerfile through
  interactive/proprietary) driving effort estimation
- **Quantitative verification** against every table, figure, and inline statistic,
  with a six-category discrepancy taxonomy including PAPER_ERROR (reproduction
  can surface errors in the published paper itself)
- **Verdicts:** SUCCESSFUL / PARTIAL / FAILED / BLOCKED

Pilot results (5 JAS papers): 4 SUCCESSFUL, 1 PARTIAL (sole cause: data
inaccessibility), including exact reproduction of deterministic analyses and
discovery of two calculation errors in a published table. See
[reproduction-system/](https://github.com/saross/llm-reproducibility/tree/HEAD/reproduction-system/) and the pilot findings report in
[studies/open-science-compliance/reports/](https://github.com/saross/llm-reproducibility/tree/HEAD/studies/open-science-compliance/reports/).

---

## Documentation

### User Guides
- [Getting Started](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/getting-started.md) - Installation and first extraction
- [Extraction Workflow](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/extraction-workflow.md) - Complete 8-pass workflow (Pass 0-7)
- [PDF Extraction](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/pdf-extraction.md) - Preparing papers for analysis
- [Schema Reference](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/schema-reference.md) - Understanding the extraction schema

### Research Assessor Skill Guide
- [Overview](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/README.md) - What the skill does and when to use it
- [Installation](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/installation-guide.md) - Installing and verifying the skill
- [Usage Guide](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/usage-guide.md) - Complete usage reference
- [Architecture](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/architecture.md) - How the skill works
- [Quick Reference](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/quick-reference.md) - Cheat sheet for common tasks

### Reproduction System
- [Overview](https://github.com/saross/llm-reproducibility/blob/HEAD/reproduction-system/README.md) - Docker-based reproduction workflow
- [Planning Guide](https://github.com/saross/llm-reproducibility/blob/HEAD/reproduction-system/reproduction-plan-guide.md) - Paper-specific adaptation
- [Launch Primer](https://github.com/saross/llm-reproducibility/blob/HEAD/reproduction-system/reproduction-launch.md) - Quick-start for reproductions

### Schema and Development
- [Schema Reference](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/schema-reference.md) - Current schema documentation
- [Version History](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/research-assessor-guide/version.md) - Skill version information

---

## Validation

The pipeline has been validated at three levels:

- **Extraction corpus:** 11 papers fully extracted (archaeology, ancient
  history, environmental science; see [input/queue.yaml](https://github.com/saross/llm-reproducibility/blob/HEAD/input/queue.yaml)).
  A run comparison demonstrated the session-per-pass workflow captures 75%
  more claims and 100% more research designs than single-session extraction.
- **Reliability:** 25-run variability test (5 papers × 5 context-cleared
  runs) — extraction counts vary between runs, but assessments are stable:
  100% classification stability, 96% verdict stability, aggregate-score CV
  1.9–3.4%. Conclusion: a single run suffices for credibility assessment.
  See [outputs/variability-test/](https://github.com/saross/llm-reproducibility/tree/HEAD/outputs/variability-test/).
- **End-to-end pilot study:** 5 Journal of Archaeological Science papers
  through the complete pipeline (extraction → assessment → FAIR scoring →
  Docker reproduction), 4 SUCCESSFUL / 1 PARTIAL reproductions. See
  [studies/open-science-compliance/](https://github.com/saross/llm-reproducibility/tree/HEAD/studies/open-science-compliance/).

Earlier v2.5-era extraction-quality reports remain in
[reports/extraction-testing/](https://github.com/saross/llm-reproducibility/tree/HEAD/reports/extraction-testing/).

---

## Dependencies

**Extraction and Assessment:**
- Claude with Skills support (via Claude Code)
- Python 3.8+ for PDF extraction scripts

**Reproduction System:**
- Docker (all reproductions execute in containers)
- R papers use rocker base images; no host R installation required

**PDF Processing:**
- PyMuPDF (fitz) - Text extraction
- pdfplumber - Table extraction
- PyYAML - Configuration
- python-slugify - Filename handling

```bash
pip install -r requirements.txt
```

---

## Project Status

### ✅ Completed (v3.0.1)
- Extraction system complete and validated (8-pass workflow, Pass 0-7; schema v2.6; 11-paper corpus)
- Assessment framework pilot-tested (Pass 8-9: classification, quality gating, Seven Signals)
- Reproduction system v1.1 (Docker-based, four-session workflow, adversarial review)
- Phase 1 pilot study complete: 5 JAS papers through the full pipeline (4 SUCCESSFUL / 1 PARTIAL reproductions)
- Reliability established (25-run variability test: assessments stable across runs)
- FAIR scoring standardised (15 binary GO-FAIR sub-principles, data and code scored independently)
- Repository organisation, FAIR4RS preparation, and wiki-layout process record

### 🚧 In Progress
- Agentic modernisation: subagent/workflow implementation of the reproduction
  and FAIR lanes (see [wiki/planning/agentic-modernisation-plan.md](https://github.com/saross/llm-reproducibility/blob/HEAD/wiki/planning/agentic-modernisation-plan.md))
- OSF preregistration of Phase 2 hypotheses (five candidates drafted from pilot findings)

### 📋 Planned (Next Phase)
- JAS 2023-2026 corpus: descriptive FAIR census + preregistered confirmatory
  reproduction study (census doubles as the sampling frame)
- Python/Julia reproduction support (extension points documented)
- Zenodo deposition and DOI assignment

---

## Contributing

We welcome contributions in several areas:

- **Domain expansion**: Adapt extraction to new fieldwork disciplines
- **Vocabulary development**: Refine controlled vocabularies empirically
- **Testing**: Run extractions on new papers, report issues
- **Examples**: Contribute worked extractions from other domains
- **Assessment**: Help develop credibility assessment rubrics
- **Documentation**: Improve guides and fix errors

See [CONTRIBUTING.md](https://github.com/saross/llm-reproducibility/blob/HEAD/CONTRIBUTING.md) for detailed guidelines.

---

## Citation

If you use this work in research, please cite:

```bibtex
@software{llm-reproducibility,
  title = {LLM-Based Research Extraction and Assessment},
  author = {[Author information in CITATION.cff]},
  version = {3.0.1},
  year = {2026},
  url = {[Repository URL in CITATION.cff]}
}
```

See [CITATION.cff](https://github.com/saross/llm-reproducibility/blob/HEAD/CITATION.cff) for machine-readable citation.

---

## Acknowledgments

- Developed with Anthropic's Claude (Sonnet 4.5 through the Claude 5 family)
- Tested on archaeological research papers
- Informed by TIDieR, CONSORT-Outcomes, and SPIRIT frameworks
- Assessment framework adapted from repliCATS methodology
- Inspired by research transparency and reproducibility initiatives

---

## Support

- **Documentation**: See [docs/](https://github.com/saross/llm-reproducibility/tree/HEAD/docs/) for comprehensive guides
- **Issues**: Report issues via GitHub Issues
- **Questions**: See [docs/user-guide/getting-started.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/getting-started.md)

---

## Development History

This project has evolved through multiple versions:

- **v2.0-2.1** (Oct 16, 2025): Initial claims/evidence extraction
- **v2.2-2.3** (Oct 17-18, 2025): Two-pass workflow with consolidation
- **v2.4** (Oct 19-20, 2025): RDMAP extraction framework added
- **v2.5** (Oct 23, 2025): Repository rationalisation and FAIR4RS preparation
- **v2.6** (Nov 2025): Infrastructure extraction (Pass 6), 8-pass workflow complete (Pass 0-7), assessment framework development
- **v2.7** (Jan 2026): Session-per-pass execution mode (+75% claims, +100% research designs vs autonomous)
- **v2.8-2.9** (Feb 2026): Reproduction system v1.0 → v1.1, validated on 5 pilot papers
- **v3.0** (Feb 2026): Schema v2.6 alignment; assessment system promoted to pilot-tested; JAS pilot study complete
- **v3.0.1** (Jul 2026): Metadata reconciliation; wiki-layout migration; compound-aware de-hyphenation; agentic modernisation plan

See [archive/README.md](https://github.com/saross/llm-reproducibility/blob/HEAD/archive/README.md) and [manifest.yaml](https://github.com/saross/llm-reproducibility/blob/HEAD/manifest.yaml) for complete development history.

---

## Licence

This project uses dual licensing:

- **Code** (scripts, validation tools): [Apache-2.0](https://github.com/saross/llm-reproducibility/tree/HEAD/LICENSE-CODE)
- **Documentation** (guides, prompts, examples): [CC-BY-4.0](https://github.com/saross/llm-reproducibility/tree/HEAD/LICENSE-DOCS)

See [LICENSE-CODE](https://github.com/saross/llm-reproducibility/tree/HEAD/LICENSE-CODE) and [LICENSE-DOCS](https://github.com/saross/llm-reproducibility/tree/HEAD/LICENSE-DOCS) for full licence texts.

---

**Ready to extract!** See [docs/user-guide/getting-started.md](https://github.com/saross/llm-reproducibility/blob/HEAD/docs/user-guide/getting-started.md) to begin.

LLM-Based Research Extraction and Assessment

Automated extraction of claims, evidence, and methodology from research papers using Large Language Models, enabling systematic assessment of research transparency, reproducibility, and credibility.

Version: 3.0.1 | Schema: v2.6 | Workflow: v5.0.0 (8-pass, session-per-pass) Status: Extraction system complete, assessment framework pilot-tested, reproduction system v1.1 Target Domains: Fieldwork-based research (archaeology, ecology, ethnography, field geology, etc.) Manifest: See manifest.yaml for component versions

Overview

This repository contains tools and workflows for:

Extraction (Complete): Systematically extract claims, evidence, implicit arguments, research designs, methods, and protocols from research papers
Assessment (Pilot-tested): Evaluate research transparency, replicability, and credibility via research-approach classification, quality gating, and the repliCATS Seven Signals adapted for Humanities, Arts, and Social Sciences (HASS)
Reproduction (v1.1): Docker-based reproduction of published computational analyses, with quantitative verification and adversarial review

The pipeline uses multi-pass workflows with Claude (Anthropic) to create structured JSON representations of research papers, enabling downstream analysis, credibility assessment, and reproduction attempts. All three systems were validated in a completed pilot study of five Journal of Archaeological Science papers (see studies/open-science-compliance/).

Quick Start

1. Install the Extraction Skill

# Install the research-assessor skill in Claude Code
# The skill is located at: .claude/skills/research-assessor/

See docs/research-assessor-guide/installation.md for detailed installation instructions.

2. Prepare Your Paper

# Extract text from PDF
python extraction-system/scripts/extract_pdf_text.py "your-paper.pdf"

# Output will be in: sources/processed-md/your-paper.md

See docs/user-guide/pdf-extraction.md for more details.

3. Run Extraction

Use the eight-pass workflow (Pass 0-7):

Metadata Pass: Extract publication metadata and paper structure
Claims/Evidence Pass 1: Liberal extraction of evidence, claims, implicit arguments
Claims/Evidence Pass 2: Rationalisation and consolidation
RDMAP Pass 1a: Liberal extraction of research designs, methods, protocols
RDMAP Pass 1b: Implicit RDMAP extraction
RDMAP Pass 2: Rationalisation and tier verification
Infrastructure Pass: Extract PIDs, FAIR assessment, funding, permits
Validation Pass: Structural integrity checks

See docs/user-guide/extraction-workflow.md for complete workflow guide.

Repository Structure

llm-reproducibility/
├── input/                 # Workflow entry points (queue, launch prompt, workflow)
├── extraction-system/     # All extraction tools (skill, prompts, schema, scripts)
├── assessment-system/     # Assessment tools (pilot-tested)
├── reproduction-system/   # Docker-based reproduction workflow (prompts, templates)
├── docs/                  # User-facing documentation (guides, skill docs, background)
├── wiki/                  # Research process record (continuity, lab notebook, planning)
├── studies/               # Research studies using the pipeline (open-science-compliance)
├── examples/              # Curated extraction examples
├── outputs/               # Extraction outputs organised by paper
├── reports/               # Key testing and QA reports
├── archive/               # Development history organised by version
├── manifest.yaml          # Component version manifest (source of truth)
├── LICENSE-CODE           # Apache 2.0 (code)
├── LICENSE-DOCS           # CC-BY-4.0 International (documentation)
└── CITATION.cff           # Machine-readable citation

Key Directories

input/ - Workflow entry points
- workflow.md - Authoritative 8-pass extraction workflow
- extraction-launch.md - Quick-start primer for new extractions
- queue.yaml - Paper processing queue with status tracking
extraction-system/ - Complete extraction toolkit
- prompts/ - Eight extraction prompts (Pass 0-7)
- schema/ - JSON schema for extraction output (v2.6)
- scripts/ - PDF text extraction and validation utilities
- extraction-plan-unified-model.md - Flexible planning guidance
docs/ - User-facing documentation: how to install and use the systems
- user-guide/ - Getting started, workflow, schema reference
- research-assessor-guide/ - Comprehensive skill documentation
- schema/ - Schema versioning and crosswalks
- background-research/ - Deep research reports
wiki/ - Versioned research process record: how the research is being done
- continuity.md - Cross-session state and session log
- working-notes.md - Empirical lab notebook
- reflections/ - Meta-research observations
- planning/ - Implementation plans and roadmaps
- Kept in-repo (not a GitHub Wiki) so the process record is versioned and archived with the code — see wiki/index.md

docs/ vs wiki/: docs/ documents the product (for users of the extraction/assessment/reproduction systems); wiki/ documents the process (for provenance and project continuity). If you want to use the tools, start in docs/; if you want to see how this project works day to day, start in wiki/.

examples/ - Example extractions
- Complete extraction samples with annotations
- Blank templates for new extractions
- Quick reference guides
archive/ - Development history
- Organized by version (v2.0-v2.1 → v2.5)
- Planning, documentation, reports, test outputs

Features

Extraction System (v2.6)

Six Object Types:

Evidence - Observations, measurements, data points
Claims - Interpretations, assertions, generalizations
Implicit Arguments - Unstated assumptions, logical implications
Research Designs - Strategic framing and rationale (WHY)
Methods - Tactical approaches (WHAT was done)
Protocols - Operational procedures (HOW specifically)

Multi-Pass Workflow (8 passes, 0-7):

Pass 0: Metadata extraction
Passes 1-2: Claims/Evidence (liberal extraction → rationalisation)
Passes 3-5: RDMAP (liberal → implicit → rationalisation)
Pass 6: Infrastructure (PIDs, FAIR, funding, permits)
Pass 7: Validation ensuring structural integrity

Provenance Tracking:

Every item linked to source text via verbatim quotes
Location tracking (section, page, paragraph)
Cross-references between all object types
Complete consolidation metadata for traceability

Fieldwork-Specific:

Respects abductive/inductive reasoning
Tracks opportunistic decisions and adaptations
Distinguishes planned vs. actual procedures
Documents emergent discoveries

Assessment Framework (Pilot-tested, v1.0)

Three Pillars framework (Transparency → Credibility → Reproducibility) with the repliCATS Seven Signals adapted for HASS disciplines:

Pass 8 — Classification: research approach (inductive/deductive/abductive) with HARKing detection
Track A — Quality gating: extraction quality checks before assessment
Pass 9 — Seven Signals in three clusters: Foundational Clarity (Comprehensibility, Transparency); Evidential Strength (Plausibility, Validity, Robustness, Generalisability); Reproducibility

Reliability evidence from a 25-run variability test (5 papers × 5 runs): 100% classification stability, 96% verdict stability, aggregate-score coefficients of variation 1.9–3.4% (see outputs/variability-test/).

See wiki/planning/credibility-implementation-plan-v2.0.md for the framework design.

Reproduction System (v1.1)

Docker-based reproduction of published computational analyses (R-based papers; Python/Julia extension points documented):

Four-session workflow: planning (R-Plan), preparation (R-A), execution and verification (R-B), adversarial review (R-C, run in a fresh context)
Paper-type classification (batch-ready/Dockerfile through interactive/proprietary) driving effort estimation
Quantitative verification against every table, figure, and inline statistic, with a six-category discrepancy taxonomy including PAPER_ERROR (reproduction can surface errors in the published paper itself)
Verdicts: SUCCESSFUL / PARTIAL / FAILED / BLOCKED

Pilot results (5 JAS papers): 4 SUCCESSFUL, 1 PARTIAL (sole cause: data inaccessibility), including exact reproduction of deterministic analyses and discovery of two calculation errors in a published table. See reproduction-system/ and the pilot findings report in studies/open-science-compliance/reports/.

Documentation

User Guides

Getting Started - Installation and first extraction
Extraction Workflow - Complete 8-pass workflow (Pass 0-7)
PDF Extraction - Preparing papers for analysis
Schema Reference - Understanding the extraction schema

Research Assessor Skill Guide

Overview - What the skill does and when to use it
Installation - Installing and verifying the skill
Usage Guide - Complete usage reference
Architecture - How the skill works
Quick Reference - Cheat sheet for common tasks

Reproduction System

Overview - Docker-based reproduction workflow
Planning Guide - Paper-specific adaptation
Launch Primer - Quick-start for reproductions

Schema and Development

Schema Reference - Current schema documentation
Version History - Skill version information

Validation

The pipeline has been validated at three levels:

Extraction corpus: 11 papers fully extracted (archaeology, ancient history, environmental science; see input/queue.yaml). A run comparison demonstrated the session-per-pass workflow captures 75% more claims and 100% more research designs than single-session extraction.
Reliability: 25-run variability test (5 papers × 5 context-cleared runs) — extraction counts vary between runs, but assessments are stable: 100% classification stability, 96% verdict stability, aggregate-score CV 1.9–3.4%. Conclusion: a single run suffices for credibility assessment. See outputs/variability-test/.
End-to-end pilot study: 5 Journal of Archaeological Science papers through the complete pipeline (extraction → assessment → FAIR scoring → Docker reproduction), 4 SUCCESSFUL / 1 PARTIAL reproductions. See studies/open-science-compliance/.

Earlier v2.5-era extraction-quality reports remain in reports/extraction-testing/.

Dependencies

Extraction and Assessment:

Claude with Skills support (via Claude Code)
Python 3.8+ for PDF extraction scripts

Reproduction System:

Docker (all reproductions execute in containers)
R papers use rocker base images; no host R installation required

PDF Processing:

PyMuPDF (fitz) - Text extraction
pdfplumber - Table extraction
PyYAML - Configuration
python-slugify - Filename handling

pip install -r requirements.txt

Project Status

✅ Completed (v3.0.1)

Extraction system complete and validated (8-pass workflow, Pass 0-7; schema v2.6; 11-paper corpus)
Assessment framework pilot-tested (Pass 8-9: classification, quality gating, Seven Signals)
Reproduction system v1.1 (Docker-based, four-session workflow, adversarial review)
Phase 1 pilot study complete: 5 JAS papers through the full pipeline (4 SUCCESSFUL / 1 PARTIAL reproductions)
Reliability established (25-run variability test: assessments stable across runs)
FAIR scoring standardised (15 binary GO-FAIR sub-principles, data and code scored independently)
Repository organisation, FAIR4RS preparation, and wiki-layout process record

🚧 In Progress

Agentic modernisation: subagent/workflow implementation of the reproduction and FAIR lanes (see wiki/planning/agentic-modernisation-plan.md)
OSF preregistration of Phase 2 hypotheses (five candidates drafted from pilot findings)

📋 Planned (Next Phase)

JAS 2023-2026 corpus: descriptive FAIR census + preregistered confirmatory reproduction study (census doubles as the sampling frame)
Python/Julia reproduction support (extension points documented)
Zenodo deposition and DOI assignment

Contributing

We welcome contributions in several areas:

Domain expansion: Adapt extraction to new fieldwork disciplines
Vocabulary development: Refine controlled vocabularies empirically
Testing: Run extractions on new papers, report issues
Examples: Contribute worked extractions from other domains
Assessment: Help develop credibility assessment rubrics
Documentation: Improve guides and fix errors

See CONTRIBUTING.md for detailed guidelines.

Citation

If you use this work in research, please cite:

@software{llm-reproducibility,
  title = {LLM-Based Research Extraction and Assessment},
  author = {[Author information in CITATION.cff]},
  version = {3.0.1},
  year = {2026},
  url = {[Repository URL in CITATION.cff]}
}

See CITATION.cff for machine-readable citation.

Acknowledgments

Developed with Anthropic's Claude (Sonnet 4.5 through the Claude 5 family)
Tested on archaeological research papers
Informed by TIDieR, CONSORT-Outcomes, and SPIRIT frameworks
Assessment framework adapted from repliCATS methodology
Inspired by research transparency and reproducibility initiatives

Support

Documentation: See docs/ for comprehensive guides
Issues: Report issues via GitHub Issues
Questions: See docs/user-guide/getting-started.md

Development History

This project has evolved through multiple versions:

v2.0-2.1 (Oct 16, 2025): Initial claims/evidence extraction
v2.2-2.3 (Oct 17-18, 2025): Two-pass workflow with consolidation
v2.4 (Oct 19-20, 2025): RDMAP extraction framework added
v2.5 (Oct 23, 2025): Repository rationalisation and FAIR4RS preparation
v2.6 (Nov 2025): Infrastructure extraction (Pass 6), 8-pass workflow complete (Pass 0-7), assessment framework development
v2.7 (Jan 2026): Session-per-pass execution mode (+75% claims, +100% research designs vs autonomous)
v2.8-2.9 (Feb 2026): Reproduction system v1.0 → v1.1, validated on 5 pilot papers
v3.0 (Feb 2026): Schema v2.6 alignment; assessment system promoted to pilot-tested; JAS pilot study complete
v3.0.1 (Jul 2026): Metadata reconciliation; wiki-layout migration; compound-aware de-hyphenation; agentic modernisation plan

See archive/README.md and manifest.yaml for complete development history.

Licence

This project uses dual licensing:

Code (scripts, validation tools): Apache-2.0
Documentation (guides, prompts, examples): CC-BY-4.0

See LICENSE-CODE and LICENSE-DOCS for full licence texts.

Ready to extract! See docs/user-guide/getting-started.md to begin.