---
slug: "justynclark-project-aware-qa"
source_type: "readme"
source_url: "https://cdn.jsdelivr.net/gh/justyn-clark/project-aware-qa@main/README.md"
repo: "https://github.com/justyn-clark/project-aware-qa"
source_file: "README.md"
branch: "main"
---
# project-aware-qa

[![npm](https://img.shields.io/npm/v/@justynclark/project-aware-qa)](https://www.npmjs.com/package/@justynclark/project-aware-qa)
[![license](https://img.shields.io/npm/l/@justynclark/project-aware-qa)](./LICENSE)

A **Pi-first QA pack** that scans a real codebase, infers its test surfaces, ranks risk,
runs evidence-based checks, captures screenshots, and writes `.qa/*` reports.

It is *project-aware*: instead of running generic checks, it first builds a **surface
registry** by correlating frontend routes, backend endpoints, forms, DTOs, cache usage and
existing tests by **entity**, then plans and verifies the riskiest surfaces — admin/CRUD
flows, mutations without cache invalidation, forms whose fields don't match their DTO, and
untested mutating areas.

It is **multi-stack** via a plugin architecture: deep adapters parse real web surfaces for
specific ecosystems, and a generic adapter gives every other language a useful baseline.
A monorepo (e.g. a Remix frontend + a Phoenix backend) is detected as multiple stacks and
verified per-stack.

### Stacks

| Adapter | Coverage |
| --- | --- |
| **node** (deep) | React Router / Remix routes, NestJS controllers/DTOs, forms, cache, Vitest/Jest/Playwright |
| **phoenix** (deep) | Phoenix router (scopes/pipelines, `resources`, LiveView), controllers, Ecto schemas, HEEx forms, Cachex, ExUnit |
| **python** (deep) | FastAPI/Flask decorator routes (APIRouter/Blueprint prefixes, Pydantic bodies), Django `urls.py` + DRF `router.register` + `@admin.register` CRUD, Pydantic/DRF/Django-model/form DTOs, templates, cache, pytest |
| **go** (deep) | chi / gin / echo / fiber / gorilla / net-http routes (chi `Route` closures + `Group` prefixes, gorilla `.Methods`, Go 1.22 `"GET /path"`), request/response structs as DTOs, `go test`, `go build/vet`/golangci-lint |
| **rust** (deep) | axum `.route(...)` chains, actix/Rocket `#[get]` macros, serde `Deserialize` structs as DTOs, `cargo build/test/clippy` |
| **java** (deep) | Spring `@RestController`/`@*Mapping` (class prefix + `@RequestBody`), JAX-RS `@Path`/`@GET`, records + `*Request`/`*Dto` classes, maven/gradle |
| **csharp** (deep) | ASP.NET Core minimal APIs (`MapGet`/`MapGroup`) + MVC controllers (`[HttpGet]`, `[controller]`, `[FromBody]`), record/`*Request` DTOs, `dotnet build/test` |
| **generic** (any language) | Detects the project type — **Zig, Odin, Godot (GUT), C/C++, Swift, Ruby, PHP, …** (and any language above when no web framework is present) — maps build/test/lint commands, and emits a test-gap baseline, so even systems/game stacks get something useful instead of "0 files" |

Adding a deep adapter for a new ecosystem is incremental — implement one `StackAdapter`
(file extensions, toolchain, detectors) and the shared inference/risk/report/verify core
works unchanged.

## Install into Pi

This is a Pi package: its `package.json` `pi` manifest points at the extension and the
skill, so Pi discovers both.

```bash
# Install the published package:
pi install npm:@justynclark/project-aware-qa

# …or from this repo (git source or a local clone):
pi install git:github.com/justyn-clark/project-aware-qa
pi install .
```

After installing, `/reload` in Pi and the four commands appear.

### Developing on the pack

```bash
npm install
npm run check   # strict typecheck
npm run lint    # biome
npm test        # node:test suite
pi -e extensions/qa.ts   # load the local extension for a session
```

## Commands

| Command | Writes |
| --- | --- |
| `/qa-scan` | `.qa/surface-registry.json`, `scan-report.md`, `crud-matrix.md`, `risk-report.md` |
| `/qa-plan [scope]` | `.qa/test-plan.md` |
| `/qa-verify [scope] [flags]` | `.qa/verification-report.md`, `.qa/findings/*`, `.qa/screenshots/*` |
| `/qa-report` | `.qa/qa-report.md` |

`/qa-verify` flags (all **off by default** — they're brakes):
`--install`, `--tests`, `--screenshots`, `--start-dev-server`, `--base-url <url>`,
`--max-routes <n>`, `--timeout <ms>`.

### As standalone scripts

```bash
tsx scripts/scan.ts   /path/to/repo
tsx scripts/plan.ts   /path/to/repo --scope admin
tsx scripts/verify.ts /path/to/repo --screenshots --base-url http://localhost:5173
tsx scripts/report.ts /path/to/repo
```

## The repair loop

```
scan → plan → verify → fix → verify → report
```

Nothing is "done" without evidence. Every verification check links its command log under
`.qa/findings/`, and every failing check becomes a repair-loop finding carrying:
**issue · evidence · suspected cause · fix status · rerun evidence · remaining gap.**
Skipped checks are never counted as passing — they record *why* they were skipped.

### Route checks are calibrated, not naïve

An HTTP status is only a trustworthy route signal when the server distinguishes real
routes from unknown ones. So `/qa-verify` first **calibrates** the server (probes the root
and a deliberately-nonexistent path):

- **SSR / framework apps** (unknown path → 404): status is meaningful. Routes that 2xx/3xx
  pass; **any 404/5xx fails the check** (partial = failed), with the broken routes named by
  source file. 404s that are mere normalization artifacts (trailing slash) are **rescued**
  by a variant re-probe before being reported. 401/403 are treated as auth-gated (a soft
  note, not a failure).
- **SPA / catch-all apps** (unknown path → 200): HTTP status can't validate routes, so the
  route check is reported **inconclusive** (never a false pass), and the authoritative
  verdict comes from the **rendered-DOM check** in `--screenshots` mode — which flags pages
  that render a not-found/error boundary and counts console errors.

`inconclusive` is a first-class status: not a pass, not a fail, with the reason recorded.

## What it detects

- **Routes** — React Router (RR6 `<Route>`, RR7 `route()`/`index()` config) and
  Remix/flat-routes file conventions (`app/routes/**`, `$param`, `_index`, pathless `_`
  layouts, `.route.tsx`, hybrid folders), plus light Next.js app-router support.
- **Backend** — NestJS controllers/endpoints (`@Controller`, `@Get/@Post/@Put/@Patch/@Delete`,
  `@Body() Dto`) and DTOs with class-validator decorators.
- **Forms** — Remix `<Form>`, raw `<form>`, react-hook-form, Formik, with field names.
- **Cache/invalidation** — TanStack Query (`useQuery`/`useMutation`/`invalidateQueries`),
  SWR (`useSWR`/`mutate`), React Router revalidation, `cachified` + `clearCacheKey`,
  NestJS `cache-manager`, and `localStorage`/`sessionStorage`.
- **Tests** — Vitest / Jest / Playwright, mapped to the entities they cover.

## Architecture

```
extensions/qa.ts          Pi extension — registers /qa-scan /qa-plan /qa-verify /qa-report
scripts/*.ts              CLI wrappers around the same core
src/
  types.ts                Surface + registry model (Phase 3 core object)
  scanner/                files walker, package/toolchain detection, scan orchestration
  detectors/              react-router, nestjs, forms, cache, playwright, tests, entities
  inference/              crud (entity correlation), risk (scoring)
  registry/               build (assembly), schema (validation)
  verify/                 commands (safe runner + brakes), playwright (probes+screenshots)
  report/                 markdown (scan/crud/risk/plan), verification, summary
  core.ts                 runScan / runPlan / runVerify / runReport
templates/                surface-registry.schema.json + report templates
skills/project-aware-qa/  SKILL.md (agent guidance)
```

Runtime dependencies: **none** — the scanner uses only Node built-ins. Playwright is used
opportunistically (resolved from the target repo or globally) and is never required.

## Surface registry

The central artifact (`.qa/surface-registry.json`) is an array of `Surface` objects:

```ts
{ id, type, entity?, frontendRoute?, backendEndpoint?, files,
  risk, riskScore, riskReasons, inferredTests, evidence, gaps }
```

`type ∈ { crud-flow, form, api-endpoint, route, cache-risk, visual-page, test-gap }`.
Validated against `templates/surface-registry.schema.json`.

## Limitations (v1)

- Route detection is heuristic across React Router versions; flat-routes edge cases may
  mis-derive a path.
- NestJS decorator parsing is regex-based (computed paths / unusual decorators may be missed).
- Form detection requires real fields to avoid over-detecting search/layout forms.
- Auth-gated admin routes need credentials/config to verify at runtime.
- Visual QA captures, it does **not** diff against baselines.
- Cache-invalidation inference is best-effort and can produce false positives/negatives.