Universal Web API Capability

Crawl Site

Run a bounded breadth-first crawl across same-site pages and public files with robots rules, filters, rate limits, controlled concurrency, per-page failures, and stateless continuation tokens.

Capability ID web.site.crawl

Cost 1 credit per request

Maximum runtime 120 seconds

View raw

# Crawl Site

Run a bounded breadth-first crawl across same-site pages and public files with robots rules, filters, rate limits, controlled concurrency, per-page failures, and stateless continuation tokens.

- Platform: [Universal Web](https://docs.upscrape.com/docs/platforms/web)
- Capability ID: `web.site.crawl`
- Cost: 1 credit per request
- Maximum runtime: 120 seconds
- Execute endpoint: `POST https://data.upscrape.com/execute`

## Request

Use the exact public capability ID in the shared execute envelope.

```bash
curl --request POST \
  --url https://data.upscrape.com/execute \
  --header "Authorization: Bearer YOUR_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Prefer: wait=30" \
  --data '{
  "input": {
    "concurrency": 2,
    "detail": "standard",
    "max_depth": 2,
    "max_pages": 5,
    "url": "https://example.com/"
  },
  "capability": "web.site.crawl"
}'
```

## Input

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `ai` | `string` | No | Optional relevance filtering for discovered documents. auto keeps the full deterministic inventory if AI is unavailable; always requires AI to succeed. |
| `concurrency` | `integer` | No |  |
| `continuation_token` | `string` | No | Opaque stateless frontier returned by a preceding crawl slice with the same options. |
| `delay_ms` | `integer` | No |  |
| `detail` | `string` | No |  |
| `document` | `object` | No |  |
| `document.chunk_chars` | `integer` | No |  |
| `document.continuation_token` | `string` | No |  |
| `document.outputs` | `array<string>` | No | Requested native document outputs. This release supports structured data, text, and Markdown without an external document sidecar. |
| `document.page_end` | `integer` | No |  |
| `document.page_start` | `integer` | No |  |
| `exclude_patterns` | `array<string>` | No |  |
| `include_patterns` | `array<string>` | No |  |
| `include_subdomains` | `boolean` | No |  |
| `instructions` | `string` | No | Required when ai is auto or always. Describe which documents/files are relevant; unrelated PDFs and other files are removed from the returned crawl slice. |
| `max_depth` | `integer` | No |  |
| `max_pages` | `integer` | No |  |
| `respect_robots` | `boolean` | No |  |
| `url` | `string` | Yes | Public http(s) site URL to crawl. |

### Example input

```json
{
  "concurrency": 2,
  "detail": "standard",
  "max_depth": 2,
  "max_pages": 5,
  "url": "https://example.com/"
}
```

## Response

Successful output is returned in `results[0].data`. Raw platform output is intentionally open-ended and may evolve with the upstream source.

### Illustrative sample output

This redacted fixture is an example, not a fixed response schema.

```json
{
  "files": [],
  "final_url": "https://example.com/",
  "pages": [
    {
      "capture_backend": "http",
      "content": {
        "bytes_read": 559,
        "category": "page",
        "format": "html",
        "media_type": "text/html",
        "sha256": "[redacted:token]",
        "truncated": false,
        "type_source": "mime"
      },
      "content_type": "text/html",
      "depth": 0,
      "elapsed_ms": 0,
      "files": [],
      "final_url": "https://example.com/",
      "language": "en",
      "links": [
        {
          "text": "Learn more",
          "url": "https://iana.org/domains/example",
          "xpath": "/html[1]/body[1]/div[1]/p[2]/a[1]"
        }
      ],
      "rendered": false,
      "status_code": 200,
      "text": "Example Domain Example Domain This domain is for use in documentation examples without needing permission. Avoid use in operations. Learn more",
      "title": "Example Domain",
      "url": "https://example.com/",
      "warnings": []
    }
  ],
  "stats": {
    "ai_attempts": 0,
    "ai_completion_tokens": 0,
    "ai_filtered_files": 0,
    "ai_mode": "never",
    "ai_prompt_tokens": 0,
    "ai_used": false,
    "concurrency": 2,
    "crawled_pages": 1,
    "effective_delay_ms": 0,
    "elapsed_ms": 2203,
    "estimated_result_bytes": 1151,
    "failed_pages": 0,
    "files_found": 0,
    "frontier_dropped": 0,
    "max_depth": 2,
    "remaining_frontier": 0,
    "response_compacted": false,
    "returned_pages": 1,
    "robots_applied": true
  },
  "url": "https://example.com/",
  "warnings": []
}
```
### Illustrative output fields

Derived from the sample above for orientation only. These fields are not a fixed response schema.

| Path | Observed type | Example |
| --- | --- | --- |
| `files` | `array` | 0 items |
| `final_url` | `string` | https://example.com/ |
| `pages` | `array` | 1 items |
| `pages` | `array` | 1 items |
| `stats` | `object` | 19 fields |
| `stats.ai_attempts` | `integer` | 0 |
| `stats.ai_completion_tokens` | `integer` | 0 |
| `stats.ai_filtered_files` | `integer` | 0 |
| `stats.ai_mode` | `string` | never |
| `stats.ai_prompt_tokens` | `integer` | 0 |
| `stats.ai_used` | `boolean` | false |
| `stats.concurrency` | `integer` | 2 |
| `stats.crawled_pages` | `integer` | 1 |
| `stats.effective_delay_ms` | `integer` | 0 |
| `stats.elapsed_ms` | `integer` | 2203 |
| `stats.estimated_result_bytes` | `integer` | 1151 |
| `stats.failed_pages` | `integer` | 0 |
| `stats.files_found` | `integer` | 0 |
| `stats.frontier_dropped` | `integer` | 0 |
| `stats.max_depth` | `integer` | 2 |
| `stats.remaining_frontier` | `integer` | 0 |
| `stats.response_compacted` | `boolean` | false |
| `stats.returned_pages` | `integer` | 1 |
| `stats.robots_applied` | `boolean` | true |
| `url` | `string` | https://example.com/ |
| `warnings` | `array` | 0 items |

## Execution behavior

A `200` response completed inline. A `202` response was queued; poll `GET /jobs/{id}` until the job reaches `completed` or `failed`. Use an `Idempotency-Key` when retrying must not create a duplicate logical job.

## Related documentation

- [Authentication](https://docs.upscrape.com/docs/api/authentication)
- [Jobs and results](https://docs.upscrape.com/docs/api/jobs)
- [Errors and retries](https://docs.upscrape.com/docs/api/errors)
- [Idempotency](https://docs.upscrape.com/docs/api/idempotency)

POST /execute Shared execution endpoint

Input parameters

ai string optional

Optional relevance filtering for discovered documents. auto keeps the full deterministic inventory if AI is unavailable; always requires AI to succeed.

Available values neverautoalways

concurrency integer optional