Universal Web API Capability

Crawl Site

Run a bounded breadth-first crawl across same-site pages and public files with robots rules, explicit scope patterns, controlled concurrency, stateless continuation, source-completeness reconciliation, and optional non-destructive AI relevance ranking.

Capability ID web.site.crawl

Cost 1 credit per request

Maximum runtime 120 seconds

View raw

# Crawl Site

Run a bounded breadth-first crawl across same-site pages and public files with robots rules, explicit scope patterns, controlled concurrency, stateless continuation, source-completeness reconciliation, and optional non-destructive AI relevance ranking.

- Platform: [Universal Web](https://docs.upscrape.com/docs/platforms/web)
- Capability ID: `web.site.crawl`
- Cost: 1 credit per request
- Maximum runtime: 120 seconds
- Execute endpoint: `POST https://data.upscrape.com/execute`

## Request

Use the exact public capability ID in the shared execute envelope.

```bash
curl --request POST \
  --url https://data.upscrape.com/execute \
  --header "Authorization: Bearer YOUR_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Prefer: wait=30" \
  --data '{
  "input": {
    "concurrency": 2,
    "detail": "standard",
    "max_depth": 2,
    "max_pages": 5,
    "url": "https://example.com/"
  },
  "capability": "web.site.crawl"
}'
```

## Input

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `accept_statuses` | `array<integer>` | No | Accept statuses supplied for this request. |
| `ai` | `string` | No | Optional non-destructive relevance ranking for discovered documents. Every deterministic candidate remains in the result; auto leaves candidates unranked if AI is unavailable, while always requires ranking to succeed. |
| `allow_browser_fallback` | `boolean` | No | Allow browser fallback supplied for this request. |
| `allow_truncated` | `boolean` | No | Allow truncated supplied for this request. |
| `concurrency` | `integer` | No | Concurrency supplied for this request. |
| `continuation_token` | `string` | No | Opaque stateless frontier returned by a preceding crawl slice with the same options. A transiently failed URL is retained for at most one bounded retry; permanent failures are not amplified. |
| `delay_ms` | `integer` | No | Delay ms supplied for this request. |
| `detail` | `string` | No | Detail supplied for this request. Allowed values: `summary`, `standard`, `full`. |
| `document` | `object` | No | Document supplied for this request. |
| `document.chunk_chars` | `integer` | No | Chunk chars supplied for this request. |
| `document.continuation_token` | `string` | No | Continuation token supplied for this request. |
| `document.ocr` | `string` | No | Ocr supplied for this request. Allowed values: `auto`, `never`, `always`. |
| `document.outputs` | `array<string>` | No | Requested document outputs. source returns the originally captured bytes: inline through 5.5 MB, otherwise through a hash-verified durable store, and fails if neither delivery path is available. When source is omitted, remote_source is provenance only and contains no bytes. |
| `document.page_end` | `integer` | No | Page end supplied for this request. |
| `document.page_start` | `integer` | No | Page start supplied for this request. |
| `exclude_patterns` | `array<string>` | No | Exclude patterns supplied for this request. |
| `include_patterns` | `array<string>` | No | Include patterns supplied for this request. |
| `include_subdomains` | `boolean` | No | When false, only the seed host is allowed. When true, hosts are still limited to the seed's registrable domain; unrelated hosts are never added. |
| `instructions` | `string` | No | Required when ai is auto or always. Describe which documents/files are relevant. Ranking is non-destructive: every deterministic candidate remains in the result. |
| `max_body_bytes` | `integer` | No | Max body bytes supplied for this request. |
| `max_depth` | `integer` | No | Max depth supplied for this request. |
| `max_pages` | `integer` | No | Max pages supplied for this request. |
| `max_total_bytes` | `integer` | No | Max total bytes supplied for this request. |
| `respect_robots` | `boolean` | No | Respect robots supplied for this request. |
| `url` | `string` | Yes | Public http(s) site URL to crawl. |

### Example input

```json
{
  "concurrency": 2,
  "detail": "standard",
  "max_depth": 2,
  "max_pages": 5,
  "url": "https://example.com/"
}
```

## Response

Successful output is returned in `results[0].data`. Raw platform output is intentionally open-ended and may evolve with the upstream source.

### Illustrative sample output

This redacted fixture is an example, not a fixed response schema.

```json
{
  "files": [],
  "final_url": "https://example.com/",
  "pages": [
    {
      "capture_backend": "http",
      "content": {
        "bytes_read": 559,
        "category": "page",
        "format": "html",
        "media_type": "text/html",
        "sha256": "[redacted:token]",
        "truncated": false,
        "type_source": "mime"
      },
      "content_type": "text/html",
      "depth": 0,
      "elapsed_ms": 0,
      "files": [],
      "final_url": "https://example.com/",
      "language": "en",
      "links": [
        {
          "text": "Learn more",
          "url": "https://iana.org/domains/example",
          "xpath": "/html[1]/body[1]/div[1]/p[2]/a[1]"
        }
      ],
      "rendered": false,
      "status_code": 200,
      "text": "Example Domain Example Domain This domain is for use in documentation examples without needing permission. Avoid use in operations. Learn more",
      "title": "Example Domain",
      "url": "https://example.com/",
      "warnings": []
    }
  ],
  "stats": {
    "ai_attempts": 0,
    "ai_completion_tokens": 0,
    "ai_mode": "never",
    "ai_prompt_tokens": 0,
    "ai_ranking_mode": "non_destructive",
    "ai_rejected_files": 0,
    "ai_relevant_files": 0,
    "ai_uncertain_files": 0,
    "ai_unselected_files": 0,
    "ai_used": false,
    "concurrency": 2,
    "crawled_pages": 1,
    "effective_delay_ms": 0,
    "elapsed_ms": 2203,
    "estimated_result_bytes": 1151,
    "failed_pages": 0,
    "files_before_compaction": 0,
    "files_found": 0,
    "files_omitted_by_compaction": 0,
    "files_returned": 0,
    "frontier_dropped": 0,
    "max_depth": 2,
    "remaining_frontier": 0,
    "response_compacted": false,
    "returned_pages": 1,
    "robots_applied": true
  },
  "url": "https://example.com/",
  "warnings": []
}
```
### Illustrative output fields

Derived from the sample above for orientation only. These fields are not a fixed response schema.

| Path | Observed type | Example |
| --- | --- | --- |
| `files` | `array` | 0 items |
| `final_url` | `string` | https://example.com/ |
| `pages` | `array` | 1 items |
| `pages` | `array` | 1 items |
| `stats` | `object` | 26 fields |
| `stats.ai_attempts` | `integer` | 0 |
| `stats.ai_completion_tokens` | `integer` | 0 |
| `stats.ai_mode` | `string` | never |
| `stats.ai_prompt_tokens` | `integer` | 0 |
| `stats.ai_ranking_mode` | `string` | non_destructive |
| `stats.ai_rejected_files` | `integer` | 0 |
| `stats.ai_relevant_files` | `integer` | 0 |
| `stats.ai_uncertain_files` | `integer` | 0 |
| `stats.ai_unselected_files` | `integer` | 0 |
| `stats.ai_used` | `boolean` | false |
| `stats.concurrency` | `integer` | 2 |
| `stats.crawled_pages` | `integer` | 1 |
| `stats.effective_delay_ms` | `integer` | 0 |
| `stats.elapsed_ms` | `integer` | 2203 |
| `stats.estimated_result_bytes` | `integer` | 1151 |
| `stats.failed_pages` | `integer` | 0 |
| `stats.files_before_compaction` | `integer` | 0 |
| `stats.files_found` | `integer` | 0 |
| `stats.files_omitted_by_compaction` | `integer` | 0 |
| `stats.files_returned` | `integer` | 0 |
| `stats.frontier_dropped` | `integer` | 0 |
| `stats.max_depth` | `integer` | 2 |
| `stats.remaining_frontier` | `integer` | 0 |
| `stats.response_compacted` | `boolean` | false |
| `stats.returned_pages` | `integer` | 1 |
| `stats.robots_applied` | `boolean` | true |
| `url` | `string` | https://example.com/ |
| `warnings` | `array` | 0 items |

## Execution behavior

A `200` response completed inline. A `202` response was queued; poll `GET /jobs/{id}` until the job reaches `completed` or `failed`. Use an `Idempotency-Key` when retrying must not create a duplicate logical job.

## Related documentation

- [Authentication](https://docs.upscrape.com/docs/api/authentication)
- [Jobs and results](https://docs.upscrape.com/docs/api/jobs)
- [Errors and retries](https://docs.upscrape.com/docs/api/errors)
- [Idempotency](https://docs.upscrape.com/docs/api/idempotency)

POST /execute Shared execution endpoint

Input parameters

accept_statuses array<integer> optional

Accept statuses supplied for this request.

ai string optional

Optional non-destructive relevance ranking for discovered documents. Every deterministic candidate remains in the result; auto leaves candidates unranked if AI is unavailable, while always requires ranking to succeed.

Available values neverautoalways

allow_browser_fallback boolean optional

Allow browser fallback supplied for this request.

allow_truncated boolean optional

Allow truncated supplied for this request.

concurrency integer optional

Concurrency supplied for this request.

Example 2

continuation_token string optional

Opaque stateless frontier returned by a preceding crawl slice with the same options. A transiently failed URL is retained for at most one bounded retry; permanent failures are not amplified.

delay_ms integer optional

Delay ms supplied for this request.

detail string optional

Detail supplied for this request. Allowed values: `summary`, `standard`, `full`.

Example standard

Available values summarystandardfull

document object optional

Document supplied for this request.

document.chunk_chars integer optional

Chunk chars supplied for this request.

document.continuation_token string optional

Continuation token supplied for this request.

document.ocr string optional

Ocr supplied for this request. Allowed values: `auto`, `never`, `always`.

Available values autoneveralways

document.outputs array<string> optional

Requested document outputs. source returns the originally captured bytes: inline through 5.5 MB, otherwise through a hash-verified durable store, and fails if neither delivery path is available. When source is omitted, remote_source is provenance only and contains no bytes.

document.page_end integer optional

Page end supplied for this request.

document.page_start integer optional

Page start supplied for this request.

exclude_patterns array<string> optional

Exclude patterns supplied for this request.

include_patterns array<string> optional

Include patterns supplied for this request.

include_subdomains boolean optional

When false, only the seed host is allowed. When true, hosts are still limited to the seed's registrable domain; unrelated hosts are never added.

instructions string optional

Required when ai is auto or always. Describe which documents/files are relevant. Ranking is non-destructive: every deterministic candidate remains in the result.

max_body_bytes integer optional

Max body bytes supplied for this request.

max_depth integer optional

Max depth supplied for this request.

Example 2

max_pages integer optional

Max pages supplied for this request.

Example 5

max_total_bytes integer optional

Max total bytes supplied for this request.

respect_robots boolean optional

Respect robots supplied for this request.

url string required

Public http(s) site URL to crawl.

Example https://example.com/

Response

Successful output is returned in results[0].data. Raw platform output is intentionally open-ended and may evolve with the upstream source.

Illustrative sample The full redacted fixture is shown in the response panel.

Illustrative output fields

Derived from the sample above for orientation only. These observed fields are not a fixed response schema.

Path	Observed type	Example
`files`	`array`	0 items
`final_url`	`string`	https://example.com/
`pages`	`array`	1 items
`pages`	`array`	1 items
`stats`	`object`	26 fields
`stats.ai_attempts`	`integer`	0
`stats.ai_completion_tokens`	`integer`	0
`stats.ai_mode`	`string`	never
`stats.ai_prompt_tokens`	`integer`	0
`stats.ai_ranking_mode`	`string`	non_destructive
`stats.ai_rejected_files`	`integer`	0
`stats.ai_relevant_files`	`integer`	0
`stats.ai_uncertain_files`	`integer`	0
`stats.ai_unselected_files`	`integer`	0
`stats.ai_used`	`boolean`	false
`stats.concurrency`	`integer`	2
`stats.crawled_pages`	`integer`	1
`stats.effective_delay_ms`	`integer`	0
`stats.elapsed_ms`	`integer`	2203
`stats.estimated_result_bytes`	`integer`	1151
`stats.failed_pages`	`integer`	0
`stats.files_before_compaction`	`integer`	0
`stats.files_found`	`integer`	0
`stats.files_omitted_by_compaction`	`integer`	0
`stats.files_returned`	`integer`	0
`stats.frontier_dropped`	`integer`	0
`stats.max_depth`	`integer`	2
`stats.remaining_frontier`	`integer`	0
`stats.response_compacted`	`boolean`	false
`stats.returned_pages`	`integer`	1
`stats.robots_applied`	`boolean`	true
`url`	`string`	https://example.com/
`warnings`	`array`	0 items

Execution behavior

A 200 response completed inline. A 202 response was queued; poll GET /jobs/:id until the job reaches completed or failed.