--- slug: "pi-smart-fetch" source_type: "readme" source_url: "https://cdn.jsdelivr.net/gh/Thinkscape/agent-smart-fetch@main/README.md" repo: "https://github.com/Thinkscape/agent-smart-fetch" source_file: "README.md" branch: "main" --- # Agent Smart Fetch Better web fetching for agents. ## Features - ๐Ÿ” **Browser-like TLS/SSL + HTTP fingerprints** โ€” better success on bot-defended pages - ๐Ÿงน **Defuddle extraction** โ€” clean readable content instead of noisy HTML - ๐Ÿง  **Useful metadata** โ€” title, author, site, language, published date when available - ๐Ÿ“ฆ **Downloads + large file support** โ€” stream attachments and binaries to temp files - ๐Ÿ” **Client-side `` redirects** โ€” follows sane meta refresh redirects with loop limits - ๐Ÿ”— **Alternate content fallback** โ€” when extraction produces no/thin content, follows qualified `` entries in `` that match the requested output format - โšก **Batch fetch** โ€” fetch many URLs with bounded concurrency - ๐Ÿ“ **Multiple output formats** โ€” `markdown`, `html`, `text`, `json`, `raw` ## [@thinkscape/smart-fetch](https://github.com/Thinkscape/agent-smart-fetch/blob/HEAD/packages/smart-fetch/README.md) Smart Fetch CLI. Install globally and use `smart-fetch` (or `sf`) from the terminal. ```bash npm install -g @thinkscape/smart-fetch sf https://example.com ``` ## [pi-smart-fetch](https://github.com/Thinkscape/agent-smart-fetch/blob/HEAD/packages/pi-smart-fetch/README.md) Smart Fetch for pi.dev. Registers: - `web_fetch` - `batch_web_fetch` ## [openclaw-smart-fetch](https://github.com/Thinkscape/agent-smart-fetch/blob/HEAD/packages/openclaw-smart-fetch/README.md) Smart Fetch for OpenClaw. Registers: - `smart_fetch` - `batch_smart_fetch` ![pi Smart Fetch](https://raw.githubusercontent.com/Thinkscape/agent-smart-fetch/main/packages/pi-smart-fetch/demo.gif) ## Development This repo is a Bun monorepo. Install dependencies: ```bash bun install ``` Run the workspace: ```bash bun run test bun run build bun run check ``` Run package-specific commands: ```bash bun run test:core bun run test:pi bun run test:openclaw bun run test:cli bun run build:core bun run build:pi bun run build:openclaw bun run build:cli ``` Integration tests: ```bash bun run test:integration ``` Install the local pre-commit hook: ```bash bun run hooks:install ``` ## Versioning and publishing Versioning is global across the monorepo. Every merged PR to `main` is released automatically. The default release type is `patch`. To request a `minor` or `major` release, add a `.changeset/*.md` file to the PR: ```md minor Add batch request timeout controls. ``` Dependency update PRs are opened by Renovate. They also release as `patch` by default unless a changeset overrides the bump level. Bump all package versions together manually: ```bash bun run version:patch bun run version:minor bun run version:major ``` Create a local release commit and tag from changesets: ```bash bun run release ``` Local manual publish commands: ```bash bun run publish:pi bun run publish:openclaw bun run publish:cli bun run publish:all ``` Note: development uses Bun, but CI publishing still uses `npm publish` so npm Trusted Publishing works correctly. ## Repository - GitHub: `https://github.com/Thinkscape/agent-smart-fetch`