agent-fetch

agent-fetch addresses a common issue faced by AI agents when scraping web content: servers often detect client fingerprints, causing standard HTTP tools (like curl or an agent's built-in fetching capabilities) to receive truncated or incomplete responses. This prevents agents from obtaining full article information, impacting subsequent analysis and processing.

This tool spoofs servers into returning complete page content by emulating the network fingerprint of a real browser. It then employs multiple extraction strategies (such as Readability, JSON-LD, Next.js, RSC) to obtain structured, complete articles, including all paragraphs, headings, and links. It runs locally, requiring no API keys or cloud dependencies, and can be installed directly via npm or executed using the npx command.

agent-fetch is ideal for AI agent developers who need to process web content, RAG (Retrieval-Augmented Generation) pipeline builders, and any users who need to inject complete, clean web content into LLM conversations or knowledge bases. For example, when NotebookLM cannot directly add a URL as a source, you can use agent-fetch to extract the content and paste it as text.

The recommended usage is to prioritize agent-fetch over built-in fetching functions for automated workflows requiring high-fidelity, structured web content. Please note that while it supports custom timeouts and cookies, it may still need to be combined with other tools when handling extremely complex or dynamically loaded pages.