---
slug: "arcadia64-pi-ddgs"
source_type: "readme"
source_url: "https://cdn.jsdelivr.net/gh/Arcadia64/ddgs-api@main/README.md"
repo: "https://github.com/Arcadia64/ddgs-api"
source_file: "README.md"
branch: "main"
---
# ddgs-api

Local web search and page fetch for [Pi](https://www.npmjs.com/package/@mariozechner/pi-coding-agent). Runs in Docker, no API keys, no rate limits.

The fetch path has three layers: a fast static HTTP client (`curl_cffi`), real headless Chrome, and [Camoufox](https://github.com/daijro/camoufox) (a Firefox fork that's painful for anti-bot stacks to detect). The extension layers them automatically so the agent gets useful text out of most pages without thinking about it.

## Setup

Clone the repo and start the backend:

```bash
docker compose up -d --build
```

First build takes 5–10 minutes and the image is around 3GB. It downloads Chrome, Camoufox, and a GeoIP database, so be patient. Once it's up:

```bash
curl http://localhost:8091/health
# {"status":"ok"}
```

Then install the extension into Pi:

```bash
pi install npm:@arcadia64/pi-ddgs
```

Reload Pi. You'll have four tools available: `web_search`, `web_search_news`, `get_search_results`, and `fetch_url`.

## Config

Drop a JSON file at `~/.pi/ddgs.json` to override defaults. Every field is optional.

```json
{
  "ddgsApiUrl": "http://localhost:8091",
  "timeoutMs": 30000,
  "fetchTimeoutMs": 20000,
  "maxResults": 10,
  "safesearch": "off",
  "region": "us-en",
  "defaultEngine": "chrome",
  "searchCacheSize": 50
}
```

| Field | Default | Notes |
|---|---|---|
| `ddgsApiUrl` | `http://localhost:8091` | Backend URL. Point this at a remote box if you split machines. |
| `timeoutMs` | 30000 | Search request timeout. |
| `fetchTimeoutMs` | 20000 | Static fetch timeout. Render paths multiply this by 3 (Chrome) or 5 (Camoufox). |
| `maxResults` | 10 | Default result count. |
| `safesearch` | `"off"` | `"off"`, `"moderate"`, `"on"`. |
| `region` | `"us-en"` | DDG region code. |
| `defaultEngine` | `"chrome"` | `"chrome"` is fast, `"camoufox"` is slower but beats more anti-bot. |
| `searchCacheSize` | 50 | LRU bound on cached search results. |

## What it can and can't fetch

Static HTML, server-rendered SPAs, and most blogs/docs work fine through the fast static path. Pure client-rendered SPAs fall back to Chrome. Cloudflare's standard managed challenge falls through to Camoufox and usually gets the page. Reddit fetches route through [Redlib](https://github.com/redlib-org/redlib) (a mobile-OAuth-spoofing frontend) instead of reddit.com, which bypasses Reddit's VPN-IP blocklist — see [Host rewrites](#host-rewrites) to change or disable this.

Things it doesn't do: Cloudflare Turnstile, interactive challenges, and sites that block on IP reputation or behavioral signals. Those need residential proxies or real auth, which is out of scope.

---

## Beyond the basics

### Container hygiene

Each fetch creates a fresh browser context and destroys it before returning, so no cookies, history, or session state persists between requests.

Memory is bounded in layers, because headless browsers under burst load will otherwise eat everything:

- **Concurrency caps.** Rendered fetches queue behind per-engine semaphores (Chrome 4, Camoufox 2). A burst of render requests waits as cheap coroutines instead of stacking browser processes, so peak memory is a constant rather than a function of load.
- **Browser rotation.** Long-lived browsers leak slowly even when idle, so each engine is relaunched periodically (Chrome every 500 requests / 2h, Camoufox every 50 / 1h) without interrupting in-flight requests.
- **`mem_limit: 2g` with `memswap_limit: 3g`.** Swap headroom above the cap means a breach degrades to slow-but-alive instead of the kernel thrashing page cache until every request times out.
- **`init: true`** reaps orphaned browser child processes so zombies don't accumulate.
- **Healthcheck + autoheal.** A `/health` probe marks the container unhealthy if it stops responding, and the `autoheal` sidecar restarts it (plain Docker only *labels* unhealthy containers; restart-on-unhealthy is Swarm-only). Worst case a wedged container is back in ~2.5 minutes without intervention. `restart: unless-stopped` still covers plain crashes.

### Backend endpoints

| Path | Purpose |
|---|---|
| `/search?q=&max_results=&safesearch=&region=` | DDG web search |
| `/search/news?q=&max_results=&safesearch=&region=` | DDG news search |
| `/fetch?url=&render=&engine=` | Fetch + clean text. `render=true` uses a headless browser. `engine` is `chrome` (default) or `camoufox`. |
| `/health` | Liveness |

### Host rewrites

`app/main.py` has a small map that rewrites hosts before fetching:

```python
HOST_REWRITES = {
    "reddit.com":      "redlib.privacyredirect.com",  # Redlib proxy (spoofs mobile OAuth)
    "www.reddit.com":  "redlib.privacyredirect.com",
}
```

The default Redlib instance is set via `REDDIT_PROXY_HOST` env var (`redlib.privacyredirect.com`). Set it to a different Redlib instance URL (see [github.com/redlib-org/redlib-instances](https://github.com/redlib-org/redlib-instances)), or to `reddit.com` to disable the rewrite entirely. Add others if useful — `medium.com` → `scribe.rip`, `twitter.com` → `nitter.net`, etc.

### Installing the extension from a local path

If you're hacking on the extension and don't want to bump npm versions every time, point Pi at the cloned directory instead:

```json
{
  "packages": ["C:/path/to/ddgs-api/pi-ddgs"]
}
```

Reload Pi. Edit `pi-ddgs/index.ts` freely.

### Split setup (extension on machine A, backend on machine B)

Run `docker compose up -d --build` on machine B and make sure port 8091 is reachable from machine A. On machine A, set `"ddgsApiUrl": "http://<machine-B-ip>:8091"` in `~/.pi/ddgs.json`, install the extension, reload Pi.

### Layout

```
ddgs-api/
├── docker-compose.yml
├── Dockerfile
├── app/
│   └── main.py             # FastAPI backend
└── pi-ddgs/                # Pi extension
    ├── package.json
    ├── index.ts
    └── src/
        ├── ddgs.ts
        └── config.ts
```