openclaw-ultra-scraping

内容来源：SKILL.md（标准 Skill 格式） · 原始地址 · 查看安装指南

原始内容

---
name: openclaw-ultra-scraping
description: >
  Powerful web scraping, crawling, and data extraction with stealth anti-bot bypass.
  Bypasses anti-bot systems (Cloudflare Turnstile, CAPTCHAs) out of the box.
  Use when: (1) scraping websites that block normal requests, (2) extracting structured
  data from web pages, (3) crawling multiple pages with concurrency, (4) taking
  screenshots of web pages, (5) extracting links, (6) any web scraping task that
  needs stealth/anti-detection, (7) user asks to scrape/crawl/extract from URLs,
  (8) need to bypass Cloudflare or other bot protection. Supports CSS/XPath selectors,
  adaptive element tracking (survives site redesigns), multi-session spiders,
  pause/resume crawls, proxy rotation, and async operations. Powered by MyClaw.ai.
---

# OpenClaw Ultra Scraping

Adaptive web scraping framework for OpenClaw agents. Handles everything from single-page extraction to full-scale concurrent crawls with anti-bot bypass.

## Setup

Run once before first use:

```bash
bash scripts/setup.sh
```

This installs all dependencies + browser engines into `/opt/scrapling-venv`.

## Quick Start — CLI Script

The bundled `scripts/scrape.py` provides a unified CLI:

```bash
PYTHON=/opt/scrapling-venv/bin/python3

# Simple fetch (JSON output)
$PYTHON scripts/scrape.py fetch "https://example.com" --css ".content"

# Extract text
$PYTHON scripts/scrape.py extract "https://example.com" --css "h1"

# Stealth mode (bypass Cloudflare)
$PYTHON scripts/scrape.py fetch "https://protected-site.com" --stealth --solve-cloudflare --css ".data"

# Ultra-stealth (Kasada / FingerprintJS / reCAPTCHA Enterprise)
# Requires: pip install cloakbrowser
$PYTHON scripts/scrape.py fetch "https://enterprise-protected.com" --ultra-stealth --css ".data"

# Ultra-stealth with human-like behavior + residential proxy
$PYTHON scripts/scrape.py fetch "https://heavily-protected.com" --ultra-stealth --humanize --proxy "http://user:pass@residential:8080" --css ".data"

# Dynamic (full browser rendering)
$PYTHON scripts/scrape.py fetch "https://spa-site.com" --dynamic --css ".product"

# Extract links
$PYTHON scripts/scrape.py links "https://example.com" --filter "\.pdf$"

# Multi-page crawl
$PYTHON scripts/scrape.py crawl "https://example.com" --depth 2 --concurrency 10 --css ".item" -o results.json

# Output formats: json, jsonl, csv, text, markdown, html
$PYTHON scripts/scrape.py fetch "https://example.com" -f markdown -o page.md
```

## Quick Start — Python

For complex tasks, write Python directly using the venv:

```python
#!/opt/scrapling-venv/bin/python3
from scrapling.fetchers import Fetcher, StealthyFetcher

# Simple HTTP
page = Fetcher.get('https://example.com', impersonate='chrome')
titles = page.css('h1::text').getall()

# Bypass Cloudflare
page = StealthyFetcher.fetch('https://protected.com', headless=True, solve_cloudflare=True)
data = page.css('.product').getall()
```

## Fetcher Selection Guide

| Tier | Scenario | Fetcher | Flag |
|------|----------|---------|------|
| 1 | Normal sites, fast scraping | `Fetcher` | (default) |
| 2 | JS-rendered SPAs | `DynamicFetcher` | `--dynamic` |
| 3 | Cloudflare / anti-bot protected | `StealthyFetcher` | `--stealth` |
| 3 | Cloudflare Turnstile challenge | `StealthyFetcher` | `--stealth --solve-cloudflare` |
| 4 | Kasada / FingerprintJS / reCAPTCHA Enterprise | `CloakBrowser` | `--ultra-stealth` |

Tiers 1-3 auto-escalate on failure. Tier 4 (CloakBrowser) is optional — install once when needed:

```bash
pip install cloakbrowser   # ~200MB, one-time download
```

When `--stealth` fails against enterprise-grade protection, the agent will automatically
show the install prompt. After installing, retry with `--ultra-stealth`.

## Selector Cheat Sheet

```python
page.css('.class')                    # CSS
page.css('.class::text').getall()     # Text extraction
page.xpath('//div[@id="main"]')      # XPath
page.find_all('div', class_='item')  # BS4-style
page.find_by_text('keyword')         # Text search
page.css('.item', adaptive=True)     # Adaptive (survives redesigns)
```

## Advanced Features

- **Adaptive tracking**: `auto_save=True` on first run, `adaptive=True` later — elements are found even after site redesign
- **Proxy rotation**: Pass `proxy="http://host:port"` or use `ProxyRotator`
- **Sessions**: `FetcherSession`, `StealthySession`, `DynamicSession` for cookie/state persistence
- **Spider framework**: Scrapy-like concurrent crawling with pause/resume
- **Async support**: All fetchers have async variants

For full API details: read `references/api-reference.md`

name: openclaw-ultra-scraping description: > Powerful web scraping, crawling, and data extraction with stealth anti-bot bypass. Bypasses anti-bot systems (Cloudflare Turnstile, CAPTCHAs) out of the box. Use when: (1) scraping websites that block normal requests, (2) extracting structured data from web pages, (3) crawling multiple pages with concurrency, (4) taking screenshots of web pages, (5) extracting links, (6) any web scraping task that needs stealth/anti-detection, (7) user asks to scrape/crawl/extract from URLs, (8) need to bypass Cloudflare or other bot protection. Supports CSS/XPath selectors, adaptive element tracking (survives site redesigns), multi-session spiders, pause/resume crawls, proxy rotation, and async operations. Powered by MyClaw.ai.

OpenClaw Ultra Scraping

Adaptive web scraping framework for OpenClaw agents. Handles everything from single-page extraction to full-scale concurrent crawls with anti-bot bypass.

Setup

Run once before first use:

bash scripts/setup.sh

This installs all dependencies + browser engines into /opt/scrapling-venv.

Quick Start — CLI Script

The bundled scripts/scrape.py provides a unified CLI:

PYTHON=/opt/scrapling-venv/bin/python3

# Simple fetch (JSON output)
$PYTHON scripts/scrape.py fetch "https://example.com" --css ".content"

# Extract text
$PYTHON scripts/scrape.py extract "https://example.com" --css "h1"

# Stealth mode (bypass Cloudflare)
$PYTHON scripts/scrape.py fetch "https://protected-site.com" --stealth --solve-cloudflare --css ".data"

# Ultra-stealth (Kasada / FingerprintJS / reCAPTCHA Enterprise)
# Requires: pip install cloakbrowser
$PYTHON scripts/scrape.py fetch "https://enterprise-protected.com" --ultra-stealth --css ".data"

# Ultra-stealth with human-like behavior + residential proxy
$PYTHON scripts/scrape.py fetch "https://heavily-protected.com" --ultra-stealth --humanize --proxy "http://user:pass@residential:8080" --css ".data"

# Dynamic (full browser rendering)
$PYTHON scripts/scrape.py fetch "https://spa-site.com" --dynamic --css ".product"

# Extract links
$PYTHON scripts/scrape.py links "https://example.com" --filter "\.pdf$"

# Multi-page crawl
$PYTHON scripts/scrape.py crawl "https://example.com" --depth 2 --concurrency 10 --css ".item" -o results.json

# Output formats: json, jsonl, csv, text, markdown, html
$PYTHON scripts/scrape.py fetch "https://example.com" -f markdown -o page.md

Quick Start — Python

For complex tasks, write Python directly using the venv:

#!/opt/scrapling-venv/bin/python3
from scrapling.fetchers import Fetcher, StealthyFetcher

# Simple HTTP
page = Fetcher.get('https://example.com', impersonate='chrome')
titles = page.css('h1::text').getall()

# Bypass Cloudflare
page = StealthyFetcher.fetch('https://protected.com', headless=True, solve_cloudflare=True)
data = page.css('.product').getall()

Fetcher Selection Guide

Tier	Scenario	Fetcher	Flag
1	Normal sites, fast scraping	`Fetcher`	(default)
2	JS-rendered SPAs	`DynamicFetcher`	`--dynamic`
3	Cloudflare / anti-bot protected	`StealthyFetcher`	`--stealth`
3	Cloudflare Turnstile challenge	`StealthyFetcher`	`--stealth --solve-cloudflare`
4	Kasada / FingerprintJS / reCAPTCHA Enterprise	`CloakBrowser`	`--ultra-stealth`

Tiers 1-3 auto-escalate on failure. Tier 4 (CloakBrowser) is optional — install once when needed:

pip install cloakbrowser   # ~200MB, one-time download

When --stealth fails against enterprise-grade protection, the agent will automatically show the install prompt. After installing, retry with --ultra-stealth.

Selector Cheat Sheet

page.css('.class')                    # CSS
page.css('.class::text').getall()     # Text extraction
page.xpath('//div[@id="main"]')      # XPath
page.find_all('div', class_='item')  # BS4-style
page.find_by_text('keyword')         # Text search
page.css('.item', adaptive=True)     # Adaptive (survives redesigns)

Advanced Features

Adaptive tracking: auto_save=True on first run, adaptive=True later — elements are found even after site redesign
Proxy rotation: Pass proxy="http://host:port" or use ProxyRotator
Sessions: FetcherSession, StealthySession, DynamicSession for cookie/state persistence
Spider framework: Scrapy-like concurrent crawling with pause/resume
Async support: All fetchers have async variants

For full API details: read references/api-reference.md