Universal Web API Capability

Map Site

Build a deterministic, robots-aware public URL inventory from the starting page, robots.txt, and bounded recursive sitemap indexes. Classifies pages and files without deeply fetching every discovered URL and supports opaque pagination cursors.

Capability ID web.site.map

Cost 1 credit per request

Maximum runtime 120 seconds

View raw

# Map Site

Build a deterministic, robots-aware public URL inventory from the starting page, robots.txt, and bounded recursive sitemap indexes. Classifies pages and files without deeply fetching every discovered URL and supports opaque pagination cursors.

- Platform: [Universal Web](https://docs.upscrape.com/docs/platforms/web)
- Capability ID: `web.site.map`
- Cost: 1 credit per request
- Maximum runtime: 120 seconds
- Execute endpoint: `POST https://data.upscrape.com/execute`

## Request

Use the exact public capability ID in the shared execute envelope.

```bash
curl --request POST \
  --url https://data.upscrape.com/execute \
  --header "Authorization: Bearer YOUR_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Prefer: wait=30" \
  --data '{
  "input": {
    "discover_sitemaps": true,
    "max_urls": 100,
    "respect_robots": true,
    "url": "https://example.com/"
  },
  "capability": "web.site.map"
}'
```

## Input

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `cursor` | `string` | No | Opaque cursor from the preceding map response using the same options. |
| `discover_sitemaps` | `boolean` | No |  |
| `exclude_patterns` | `array<string>` | No |  |
| `include_patterns` | `array<string>` | No |  |
| `include_subdomains` | `boolean` | No |  |
| `max_sitemaps` | `integer` | No |  |
| `max_urls` | `integer` | No | Maximum URLs returned in this response page. |
| `respect_robots` | `boolean` | No |  |
| `url` | `string` | Yes | Public http(s) site URL to inventory. |

### Example input

```json
{
  "discover_sitemaps": true,
  "max_urls": 100,
  "respect_robots": true,
  "url": "https://example.com/"
}
```

## Response

Successful output is returned in `results[0].data`. Raw platform output is intentionally open-ended and may evolve with the upstream source.

### Illustrative sample output

This redacted fixture is an example, not a fixed response schema.

```json
{
  "final_url": "https://example.com/",
  "origin": "https://example.com",
  "robots": {
    "applied": true,
    "fetched": false,
    "sitemaps": [],
    "status_code": 404,
    "url": "https://example.com/robots.txt",
    "warnings": []
  },
  "sitemaps": [
    {
      "bytes": 559,
      "content_type": "text/html",
      "error": "unexpected status 404",
      "sha256": "[redacted:token]",
      "status_code": 404,
      "url": "https://example.com/sitemap.xml"
    }
  ],
  "stats": {
    "elapsed_ms": 1484,
    "estimated_result_bytes": 657,
    "inventory_count": 1,
    "offset": 0,
    "response_compacted": false,
    "returned_count": 1,
    "sitemap_count": 1
  },
  "url": "https://example.com/",
  "urls": [
    {
      "allowed": true,
      "format": "unknown",
      "kind": "page",
      "source": "seed",
      "url": "https://example.com/"
    }
  ],
  "warnings": []
}
```
### Illustrative output fields

Derived from the sample above for orientation only. These fields are not a fixed response schema.

| Path | Observed type | Example |
| --- | --- | --- |
| `final_url` | `string` | https://example.com/ |
| `origin` | `string` | https://example.com |
| `robots` | `object` | 6 fields |
| `robots.applied` | `boolean` | true |
| `robots.fetched` | `boolean` | false |
| `robots.sitemaps` | `array` | 0 items |
| `robots.status_code` | `integer` | 404 |
| `robots.url` | `string` | https://example.com/robots.txt |
| `robots.warnings` | `array` | 0 items |
| `sitemaps` | `array` | 1 items |
| `sitemaps` | `array` | 1 items |
| `stats` | `object` | 7 fields |
| `stats.elapsed_ms` | `integer` | 1484 |
| `stats.estimated_result_bytes` | `integer` | 657 |
| `stats.inventory_count` | `integer` | 1 |
| `stats.offset` | `integer` | 0 |
| `stats.response_compacted` | `boolean` | false |
| `stats.returned_count` | `integer` | 1 |
| `stats.sitemap_count` | `integer` | 1 |
| `url` | `string` | https://example.com/ |
| `urls` | `array` | 1 items |
| `urls` | `array` | 1 items |
| `warnings` | `array` | 0 items |

## Execution behavior

A `200` response completed inline. A `202` response was queued; poll `GET /jobs/{id}` until the job reaches `completed` or `failed`. Use an `Idempotency-Key` when retrying must not create a duplicate logical job.

## Related documentation

- [Authentication](https://docs.upscrape.com/docs/api/authentication)
- [Jobs and results](https://docs.upscrape.com/docs/api/jobs)
- [Errors and retries](https://docs.upscrape.com/docs/api/errors)
- [Idempotency](https://docs.upscrape.com/docs/api/idempotency)

POST /execute Shared execution endpoint