---
slug: "mapreduce-skill"
source_type: "readme"
source_url: "https://cdn.jsdelivr.net/gh/hyophyop/mapreduce-skill@main/README.md"
repo: "https://github.com/hyophyop/mapreduce-skill"
source_file: "README.md"
branch: "main"
---
# mapreduce-skill (Codex skills)

This repository contains a paired set of skills for OpenAI Codex CLI:

- `reduce-orchestrator`: a MapReduce (+ Verify) orchestrator that coordinates many workers and synthesizes results.
- `map-worker`: the worker contract used for all Map and Verify tasks (writes narrative reports to `report_path`).

These skills are designed to run as a **pair**. `reduce-orchestrator` depends on `map-worker` and will not work correctly if `map-worker` is not installed.

## What these skills do

### reduce-orchestrator

The orchestrator enforces a loop:

1) **PLAN**: write a run contract + deterministic `plan.json`
2) **MAP**: spawn multiple workers (must use `map-worker`) to do the actual work
3) **REDUCE**: summarize worker reports into consolidated findings + open questions
4) **VERIFY (mandatory)**: spawn parallel verify workers to confirm key claims / contradictions
5) **DECIDE**: either iterate (schedule more work) or finish (write `final.json`/`final.md`)

All state is persisted under `.rlm/` so the orchestrator can rehydrate run state deterministically rather than relying on chat history.

### map-worker

The worker skill provides a strict invocation contract:

- do one task,
- keep chat output minimal,
- write a narrative report to `report_path` (required),
- include contract compliance notes when a run contract is provided.

## Warning: token usage can be huge

This workflow can consume **a very large number of tokens** because it intentionally:

- runs **many parallel workers**,
- runs an additional **Verify phase** (parallel again),
- and often repeats across **multiple iterations**.

If you are running with strong models and high reasoning effort, costs can balloon quickly. Treat this skill as a “batch research / batch debugging” workflow, not a casual helper.

Practical safety tips:

- Keep `plan.json` granular (micro/meso tasks) and cap `max_workers` / `max_iterations`.
- Use smaller compute tiers for exploration; upgrade only the tasks that truly need it.
- Keep Verify focused (top-K issues + contradictions). Avoid “verify everything”.
- Prefer evidence by file paths and short quotes instead of re-summarizing huge texts.

## Acknowledgements / inspiration

This skill was inspired by (and is grateful to):

- RLM: https://alexzhang13.github.io/blog/2025/rlm/
- AI MapReduce: https://danielsada.tech/blog/ai-map-reduce/

## Install

1) Clone this repo.

2) Copy both skills into your Codex skills directory:

```bash
for skill in map-worker reduce-orchestrator; do
  mkdir -p "$CODEX_HOME/skills/$skill"
  rsync -a --delete "$skill/" "$CODEX_HOME/skills/$skill/"
done
```

If `CODEX_HOME` is not set, it is typically `~/.codex`.

## How it works (high level)

### Persistent run state (`.rlm/`)

The orchestrator uses a deterministic filesystem layout to keep runs reproducible and concurrency-safe:

- `.rlm/runs/<run_id>/` for the active run (plan, reports, final output, etc.)
- `.rlm/archive/<archive_id>/` for immutable snapshots of completed runs
- `.rlm/locks/` for per-run and cleanup locks

This enables:

- **State rehydration** (reload from disk instead of chat)
- **Concurrency safety** (avoid two orchestrators clobbering the same run)
- **Retention / TTL cleanup** (clean old archives safely)

### “Orchestrator doesn’t do the work”

The orchestrator is a planner/dispatcher/integrator. It should not do substantive domain work itself.

All real work is delegated to `map-worker` tasks, whose outputs are written as narrative reports to deterministic `report_path` locations. Reduction and verification consume those report files as the source of truth.

### Deferred opportunities (don’t lose good ideas)

When you run many workers in parallel, they often surface valuable improvements that are **out of scope** for the current run (refactors, cleanup, follow-up experiments, better diagnostics, etc.). This system is designed to *capture those opportunities* instead of losing them in chat scrollback.

`reduce-orchestrator` supports a “Deferred opportunities” mechanism:

- workers record out-of-scope ideas under a dedicated **Deferred opportunities** section in their reports,
- the orchestrator aggregates them into `.rlm/runs/<run_id>/deferred.json` using `rlm_collect_deferred.py`,
- and final outputs are expected to reference those deferred items so you can schedule follow-up runs intentionally.

### Task sizing: `granularity` and `compute_tier`

The orchestrator expects each worker task to be defined along two axes:

- `granularity`: how large the task unit is (controls parallelism and scope risk)
- `compute_tier`: how strong the model/effort should be (controls reasoning capacity and cost)

Recommended `granularity` vocabulary:

- `micro`: one narrow outcome; ideally 1 file / 1 function / 1 command; trivially reviewable
- `meso`: a small cohesive change across a few files or a small subsystem
- `macro`: cross-cutting / large-context work (generally avoid assigning macro directly; split into micro/meso + a join worker)

Recommended planning rule:

- If you think you need `macro`, you usually want **3–8 micro/meso workers + 1 join** instead.

Typical `compute_tier` usage:

- Start with lower tiers for broad exploration, then upgrade only the tasks that truly need deeper synthesis.
- If results are consistently low-quality, either **split the task** (reduce granularity) or **upgrade compute_tier**.

#### Current `compute_tier` model mapping (defaults)

These are the current defaults used by `rlm_run_ready.py` when it spawns workers:

- `standard` → `gpt-5.1-codex-mini` with reasoning effort `medium`
- `standard-plus` → `gpt-5.2-codex` with reasoning effort `medium`
- `heavy` → `gpt-5.2-codex` with reasoning effort `xhigh`

#### Changing the model settings (when needed)

You can and should change model settings when it improves quality/cost:

- **Per task**: set `compute_tier` on that task in `.rlm/runs/<run_id>/plan.json`.
- **Per run default**: pass `--default-compute-tier` to `rlm_run_ready.py` (used when a task does not specify `compute_tier`).
- **Change the mapping itself**: edit `_COMPUTE_MAP` in `reduce-orchestrator/scripts/rlm_run_ready.py` and re-install/re-package the skill.

### Mandatory Verify phase

Even when Map reports look good, Verify is required. It focuses on:

- top-K claims by impact,
- contradictions,
- weakly-supported statements.

This is intentionally conservative: it trades extra cost for fewer “hallucinated conclusions”.

## Quickstart (typical run sequence)

The authoritative workflow and command templates live in `reduce-orchestrator/SKILL.md`. The worker contract lives in `map-worker/SKILL.md`.

At a high level:

1) Initialize a run (locks + TTL cleanup) with `rlm_admin.py`
2) Write `.rlm/runs/<run_id>/plan.json` and `.rlm/runs/<run_id>/artifacts/context/run_contract.md`
3) Spawn READY map tasks (with inflight cap) using `rlm_run_ready.py`
4) Monitor progress via report freshness with `rlm_watch_reports.py`
5) Build a deterministic inventory with `rlm_reduce.py`
6) Reduce + decide next steps
7) Spawn verify tasks
8) Write `final.json`/`final.md`, archive, and cleanup using `rlm_admin.py`

## Repo layout

- `reduce-orchestrator/`: the Codex skill folder (orchestrator)
- `map-worker/`: the Codex skill folder (worker)
- `dist/reduce-orchestrator.skill`: packaged `.skill` file (zip)
- `dist/map-worker.skill`: packaged `.skill` file (zip)