pi-spark

Reliable distributed execution for agent workloads — without forking Pi.

A PostgreSQL control plane, pull-based workers, and a thin Pi Extension
for durable remote runs, capability routing, leases, fencing, retries, and cancellation.

Quick start · Architecture · Embed a worker · Pi Extension · 中文使用指南

[!IMPORTANT] pi-spark is an experimental distributed run engine. APIs may change before the first stable release. The control plane enforces scoped client credentials and per-Worker identities, but it does not terminate TLS or provide rate limiting. Put every cross-machine deployment behind a trusted TLS/mTLS gateway or service mesh.

Why pi-spark

Thin integration	Durable coordination	Safe reassignment
Integrate through REST, the worker library, or Pi's public Extension API.	PostgreSQL is the source of truth for runs, idempotency, events, placement, and retries.	Workers pull outbound, heartbeat a lease, and settle only with the current token and fencing epoch.

The control plane is independent from any agent UI. Applications can submit work over REST, embed workers in their own process, place workers on separate machines, and route each run by executor ID and labels.

Available today

• Durable submit, inspect, retry, cancel, and event APIs, with idempotency-key conflict detection.
• Pull-based worker registration and routing by executor ID plus arbitrary string labels.
• Database-backed idempotent claim replay, lease heartbeats, fencing epochs, bounded retry attempts, jittered reconnect backoff, cooperative cancellation, and graceful shutdown.
• Built-in deterministic mock and optional Pi SDK executor drivers.
• A driver SDK for registering application-specific executors without changing the control plane.
• Scoped client credentials, owner-isolated runs, and per-Worker credentials bound to capability ceilings.
• A Pi Extension that restores unfinished polling and injects terminal results into the originating session.
• Docker Compose verification for routing, idempotency, claim-response loss and replay, retry, fencing, concurrency, and cancellation.

Quick start

Requirements: Node.js 24+, npm, Docker, and Docker Compose.

Install the published client, Worker runtime, or embeddable Control Plane:

npm install --global @anrans001/pi-spark-cli
npm install @anrans001/pi-spark-worker @anrans001/pi-spark-driver-sdk
npm install @anrans001/pi-spark-control-plane

The public packages are ESM-only. Run pi-spark --help, pi-spark-worker --help, or pi-spark-control-plane --help after installation.

To develop from source:

npm ci
npm run build
npm test

Run the complete Docker E2E stack:

npm run test:docker
npm run docker:down

test:docker first recreates the dedicated Compose database volume, creates random local-only credentials under .cache/docker-auth when they are missing, reuses a complete valid bundle when present, then verifies both a fresh authenticated stack and a real upgrade from the legacy v1 schema to the current schema. The verifier drops an already-committed claim response at the TCP boundary, replays the same request concurrently, and checks conflict and stale fencing behavior. docker:down removes the checkout-scoped PostgreSQL and credential volumes, run history, and generated local credentials.

Keep the stack running for manual exploration:

npm run docker:up

export PI_SPARK_CLIENT_TOKEN_FILE="$PWD/.cache/docker-auth/client-primary.token"
node packages/cli/dist/index.js workers
node packages/cli/dist/index.js submit \
  "hello from pi-spark" \
  --kind mock \
  --label slot=a \
  --idempotency-key demo:hello:001 \
  --json
node packages/cli/dist/index.js status <run-id>

All CLI commands require exactly one of PI_SPARK_CLIENT_TOKEN or PI_SPARK_CLIENT_TOKEN_FILE. They accept --url; otherwise the URL comes from PI_SPARK_URL, then CONTROL_PLANE_URL, and finally http://127.0.0.1:3000. Tokens are intentionally not accepted as command-line arguments.

docker:up builds before invoking the idempotent docker:auth generator or reading credentials. It stops the existing application containers and creates service-scoped credential volumes only for the requested services and their dependencies, then streams the required bytes over stdin to a short-lived, network-disabled materializer. The materializer has only CAP_CHOWN; it seals each volume directory as UID/GID 1000 mode 0500 with regular mode-0400 files. Application containers mount only their own volume read-only and continue to run as the non-root node user with a read-only root filesystem and no capabilities.

Canonical host files remain mode 0600. Plaintext values are not placed in Compose YAML, container environments, image layers, command arguments, or logs; during a running local stack, a second plaintext copy exists in checkout-scoped Docker volumes and is removed by npm run docker:down. On one host, the lifecycle commands serialize concurrent operations, validate volume ownership labels, and pin a checked local Unix/npipe Docker endpoint. Remote Docker daemons are rejected: run lifecycle commands on each Docker host and connect remote Workers through the control-plane protocol. Custom credential paths must resolve outside the Docker build context. To rotate deliberately, run npm run docker:stop, then npm run docker:auth -- --force, followed by npm run docker:up. This local bridge caps the registry at 64 KiB. Production Swarm, Kubernetes, or another scheduler should use its native secret facility instead.

For complete CLI, REST, worker, Pi SDK, remote-machine, and environment-variable instructions, see the 中文使用指南.

Architecture

flowchart LR
    APP["Your application"] -->|REST / CLI| CP["Control plane"]
    PI["Pi session"] --> EXT["pi-spark Extension"]
    EXT --> CP
    CP <--> DB[("PostgreSQL")]
    WA["Worker A"] -->|register · claim · heartbeat · settle| CP
    WB["Worker B"] -->|register · claim · heartbeat · settle| CP
    WA --> EA["Registered executor drivers"]
    WB --> EB["Registered executor drivers"]

The worker connection is outbound: a remote worker only needs network access to the control plane. A claimed run carries a lease token and a monotonically increasing epoch. After lease expiry, another worker may reclaim the run, while the stale worker is fenced from committing.

The driver boundary deliberately excludes lease credentials and settlement APIs. A driver receives validated task input, worker context, and an AbortSignal; the worker runtime retains ownership of heartbeats, fencing, and result settlement.

See Architecture and failure model for the current invariants.

Runnable examples

The two-worker routing example calls the REST API directly, waits for both workers, submits two tasks concurrently, and verifies label placement: slot=a runs on worker-a, while slot=b runs on worker-b.

npm run docker:up
PI_SPARK_CLIENT_TOKEN_FILE="$PWD/.cache/docker-auth/client-primary.token" \
npm run example:two-worker-routing
npm run docker:down

The example exercises the same pull, placement, lease, and settlement protocol used by workers on separate hosts.

The Pi native Extension example starts Pi with the exact published npm package, disables every unrelated local resource, and exposes only spark_delegate, spark_status, and spark_cancel. Against the default Docker stack, ask it to submit kind=mock with slot=a and wait for the automatic follow-up:

npm install --global --ignore-scripts \
  @earendil-works/pi-coding-agent@0.80.10
npm run docker:up
export PI_SPARK_CLIENT_TOKEN_FILE="$PWD/.cache/docker-auth/client-primary.token"
npm run example:pi-native-extension

Embed through REST

Applications do not need to load the Pi Extension. Submit a durable run directly:

(
  set -eu
  umask 077
  curl_header="$(mktemp)"
  trap 'rm -f "$curl_header"' EXIT
  {
    printf '%s' 'authorization: Bearer '
    cat "$PWD/.cache/docker-auth/client-primary.token"
    printf '\n'
  } >"$curl_header"

  curl -sS -X POST http://127.0.0.1:3000/v1/runs \
    --header "@$curl_header" \
    -H 'content-type: application/json' \
    -d '{
      "idempotencyKey": "demo:api:001",
      "task": {
        "kind": "mock",
        "payload": { "prompt": "hello", "delayMs": 100 }
      },
      "placement": { "labels": { "slot": "a" } },
      "maxAttempts": 3
    }'
)

The subshell keeps the bearer out of exported environment variables and curl arguments; its EXIT trap removes the mode-0600 temporary header file.

Use GET /v1/runs/:id for state, GET /v1/runs/:id/events for durable events, and POST /v1/runs/:id/cancel for cooperative cancellation.

Authentication registry

Tokens use ps1.<kid>.<base64url-secret>. Generate one without printing the plaintext secret to standard output:

npm run auth:create -- app-a-2026-07 /absolute/path/outside/repository/app-a.token

The command creates the token file exclusively with mode 0600 and prints its id plus secretSha256 registry fragment. Prefer a token path outside the repository and every Docker build context; .dockerignore is defense in depth, not a secret store. PI_SPARK_AUTH_CONFIG_FILE must point to an absolute, versioned JSON registry. Client principals carry a subject, scopes, and a mandatory submit policy when runs:submit is present. Worker principals bind one Worker ID to allowed executors, exact labels, and maximum concurrency. See the Chinese guide for a complete registry example.

Owner-scoped permissions are runs:read, runs:cancel, and runs:retry; the corresponding :any scopes are operator privileges. workers:read reveals Worker topology. Different credentials may share a principal during key rotation, while their key IDs and secret digests remain unique.

Upgrading a v1 database to the current schema is an offline Control Plane operation. Stop every Control Plane instance before starting exactly one current instance: the owner migration locks and rewrites runs while replacing global idempotency uniqueness. After it completes, start the remaining current instances. Historical v1 runs receive an internal empty owner sentinel that cannot be configured as an authenticated subject. They are accessible only to operator credentials with the relevant runs:read:any, runs:cancel:any, or runs:retry:any scope; an ordinary client cannot impersonate that sentinel.

For an existing v2 deployment, roll out v3 in Control-Plane-first order. Upgrade and verify every Control Plane replica while existing Workers continue to use the legacy POST /v1/workers/claim; only then roll the Workers to the idempotent PUT /v1/workers/claim-requests/:claimRequestId protocol. A new Worker that reaches an old Control Plane fails closed on 404 and never falls back to the non-idempotent route. Roll back in the reverse order: Workers first, then Control Plane replicas.

Embed a worker

An executor ID is an open, operator-controlled routing key such as mock, pi, codex, or acme/code-review.v1. Runs can only be claimed by workers that both register a driver for the ID and advertise it in WORKER_EXECUTORS.

Install the Worker runtime and driver contract:

npm install @anrans001/pi-spark-worker @anrans001/pi-spark-driver-sdk

See the runnable custom executor worker for a complete source-checkout example with shutdown handling.

Implement ExecutorDriver from @anrans001/pi-spark-driver-sdk, then inject the driver into SparkWorker:

import { ExecutorError, type ExecutorDriver } from "@anrans001/pi-spark-driver-sdk";
import { loadConfig, SparkWorker } from "@anrans001/pi-spark-worker";

type EchoPayload = { prompt: string };

const echoDriver: ExecutorDriver<EchoPayload> = {
  manifest: {
    id: "echo",
    apiVersion: 1,
    displayName: "Echo",
  },

  parsePayload(payload) {
    if (
      payload === null ||
      typeof payload !== "object" ||
      !("prompt" in payload) ||
      typeof payload.prompt !== "string"
    ) {
      throw new ExecutorError({
        code: "INVALID_PAYLOAD",
        message: "payload.prompt must be a string",
        retryable: false,
      });
    }
    return { prompt: payload.prompt };
  },

  async execute(input, context, signal) {
    if (signal.aborted) throw signal.reason;
    return {
      executor: "echo",
      workerId: context.workerId,
      prompt: input.payload.prompt,
    };
  },
};

const config = loadConfig({
  ...process.env,
  WORKER_EXECUTORS: "echo",
});
const shutdown = new AbortController();

await new SparkWorker(config, {
  drivers: [echoDriver],
  includeBuiltIns: false,
}).run(shutdown.signal);

loadConfig requires exactly one of PI_SPARK_WORKER_TOKEN or PI_SPARK_WORKER_TOKEN_FILE. Each credential is bound server-side to one Worker ID, an executor allowlist, exact routing labels, and a maximum concurrency. A Worker cannot self-promote those capabilities in its registration payload.

Driver IDs contain at most 128 characters, use lowercase letters, digits, ., _, /, and -, and each slash-delimited segment must start and end with a letter or digit. Duplicate IDs and advertised-but-unregistered drivers fail during worker startup.

Driver results must be plain JSON values with at most 1,000,000 UTF-8 bytes, depth 64, and 100,000 visited values. Circular references, custom prototypes, hidden properties, symbols, BigInt, and non-finite numbers are rejected before settlement. Driver lifecycle hooks receive an AbortSignal and are bounded by DRIVER_LIFECYCLE_TIMEOUT_MS.

ExecutorError.details is persisted only when it is JSON-safe and within 64,000 bytes, depth 32, and 10,000 visited values; otherwise the worker replaces it with an omission marker.

Keep driver selection in trusted deployment configuration. A submitted task chooses among already registered IDs; it cannot cause a worker to import arbitrary code.

Use from Pi

Install the published native Pi package, start a compatible worker, and then launch Pi with the Control Plane client credential:

pi install npm:@anrans001/pi-spark-pi-extension@0.1.0

PI_SPARK_URL=http://127.0.0.1:3000 \
PI_SPARK_CLIENT_TOKEN_FILE=/absolute/path/client.token \
  pi

Use pi install npm:@anrans001/pi-spark-pi-extension@0.1.0 -l --approve for a project-local package, or try it for one process without changing settings:

PI_SPARK_URL=http://127.0.0.1:3000 \
PI_SPARK_CLIENT_TOKEN_FILE=/absolute/path/client.token \
  pi -e npm:@anrans001/pi-spark-pi-extension@0.1.0

For an isolated, runnable local-stack walkthrough, see the Pi native Extension example. The package's pi.extensions manifest points Pi at the compiled entry automatically; the manual npm install plus filesystem -e form is only needed for source-level development.

The Extension registers three model tools:

spark_delegate({ task, kind?, payload?, labels?, maxAttempts? })
spark_status({ runId })
spark_cancel({ runId })

spark_delegate returns a durable run ID with terminate: true. Pi can become idle while the worker runs; the Extension polls in the background and injects the terminal result as a follow-up turn. It derives a stable idempotency key from Pi's session ID and tool-call ID, persists tracked runs, and restores unfinished polling on session_start.

The Extension snapshots its client credential during a session and rebuilds the client on every session_start. After rotating a client token file or environment value, start or restart the Pi session so unfinished polling resumes with the new credential.

[!NOTE] The CLI defaults to kind=mock, while spark_delegate defaults to kind=pi. The Compose workers advertise only mock, so use kind: "mock" when testing the Extension against the default stack.

For real Pi execution, install the optional SDK peer in the dedicated worker image or deployment directory:

npm install --no-save --package-lock=false \
  @earendil-works/pi-coding-agent@0.80.10

Then configure WORKER_EXECUTORS=pi, the required isolated PI_WORKSPACE_ROOT, an explicit PI_ALLOWED_TOOLS allowlist, and the provider credentials used by the Pi SDK. A task can only choose a subset of the worker allowlist; the default allowlist is empty. Do not put model credentials in the control-plane image.

Packages

Package	Responsibility
@anrans001/pi-spark-protocol	Schemas, API records, lease identities, and shared client errors
@anrans001/pi-spark-driver-sdk	Executor-driver contracts and JSON-safe result validation
@anrans001/pi-spark-control-plane	Fastify API, PostgreSQL state machine, placement, and lease reaper
@anrans001/pi-spark-worker	Pull loop, driver registry, concurrency, heartbeats, cancellation, and settlement
@anrans001/pi-spark-pi-extension	Public Pi API bridge, durable run tracking, and result injection
@anrans001/pi-spark-cli	Operator commands: submit, status, cancel, and workers
@anrans001/pi-spark-e2e (private)	Repository-only Docker Compose verifier for routing, retry, fencing, and cancellation

Security

• The Control Plane fails closed unless PI_SPARK_AUTH_CONFIG_FILE points to a valid static registry. It stores SHA-256 credential digests, applies client scopes, isolates runs by subject, and binds each Worker identity to a capability ceiling.
• Bearer credentials do not encrypt traffic. Never expose the HTTP port directly to the internet; use TLS/mTLS at a trusted gateway or service mesh and redact Authorization there as well.
• Static registry changes take effect after a Control Plane restart. Rotate credentials by overlapping two key IDs for the same principal, move clients or Workers, then remove the old key.
• Authentication is not rate limiting. Enforce request and connection limits at the trusted ingress.
• An agent executor can invoke tools with the worker's filesystem, network access, and credentials. PI_WORKSPACE_ROOT bounds the Pi executor's initial cwd; it is not a complete sandbox.
• Delivery is at-least-once with stable IDs. External effects such as deploys, messages, and writes still require their own idempotency key or reconciliation.
• Cancellation is cooperative. Fencing prevents a stale worker from committing, but cannot undo an external side effect that already occurred.

Please report vulnerabilities privately as described in SECURITY.md.

Documentation

• 中文使用指南
• Architecture and failure model
• Two-worker routing example
• Contributing
• Security policy

Contributions are welcome. Start with CONTRIBUTING.md and keep pull requests focused, tested, and explicit about compatibility changes.