# Map Site

Build a deterministic, robots-aware public URL inventory from the starting page, robots.txt, and bounded recursive sitemap indexes. Classifies pages and files without deeply fetching every discovered URL and supports opaque pagination cursors.

- Platform: [Universal Web](https://docs.upscrape.com/docs/platforms/web)
- Capability ID: `web.site.map`
- Cost: 1 credit per request
- Maximum runtime: 120 seconds
- Execute endpoint: `POST https://data.upscrape.com/execute`

## Request

Use the exact public capability ID in the shared execute envelope.

```bash
curl --request POST \
  --url https://data.upscrape.com/execute \
  --header "Authorization: Bearer YOUR_API_KEY" \
  --header "Content-Type: application/json" \
  --header "Prefer: wait=30" \
  --data '{
  "input": {
    "discover_sitemaps": true,
    "max_urls": 100,
    "respect_robots": true,
    "url": "https://example.com/"
  },
  "capability": "web.site.map"
}'
```

## Input

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `cursor` | `string` | No | Opaque cursor from the preceding map response using the same options. |
| `discover_sitemaps` | `boolean` | No |  |
| `exclude_patterns` | `array<string>` | No |  |
| `include_patterns` | `array<string>` | No |  |
| `include_subdomains` | `boolean` | No |  |
| `max_sitemaps` | `integer` | No |  |
| `max_urls` | `integer` | No | Maximum URLs returned in this response page. |
| `respect_robots` | `boolean` | No |  |
| `url` | `string` | Yes | Public http(s) site URL to inventory. |

### Example input

```json
{
  "discover_sitemaps": true,
  "max_urls": 100,
  "respect_robots": true,
  "url": "https://example.com/"
}
```

## Response

Successful output is returned in `results[0].data`. Raw platform output is intentionally open-ended and may evolve with the upstream source.

### Illustrative sample output

This redacted fixture is an example, not a fixed response schema.

```json
{
  "final_url": "https://example.com/",
  "origin": "https://example.com",
  "robots": {
    "applied": true,
    "fetched": false,
    "sitemaps": [],
    "status_code": 404,
    "url": "https://example.com/robots.txt",
    "warnings": []
  },
  "sitemaps": [
    {
      "bytes": 559,
      "content_type": "text/html",
      "error": "unexpected status 404",
      "sha256": "[redacted:token]",
      "status_code": 404,
      "url": "https://example.com/sitemap.xml"
    }
  ],
  "stats": {
    "elapsed_ms": 1484,
    "estimated_result_bytes": 657,
    "inventory_count": 1,
    "offset": 0,
    "response_compacted": false,
    "returned_count": 1,
    "sitemap_count": 1
  },
  "url": "https://example.com/",
  "urls": [
    {
      "allowed": true,
      "format": "unknown",
      "kind": "page",
      "source": "seed",
      "url": "https://example.com/"
    }
  ],
  "warnings": []
}
```
### Illustrative output fields

Derived from the sample above for orientation only. These fields are not a fixed response schema.

| Path | Observed type | Example |
| --- | --- | --- |
| `final_url` | `string` | https://example.com/ |
| `origin` | `string` | https://example.com |
| `robots` | `object` | 6 fields |
| `robots.applied` | `boolean` | true |
| `robots.fetched` | `boolean` | false |
| `robots.sitemaps` | `array` | 0 items |
| `robots.status_code` | `integer` | 404 |
| `robots.url` | `string` | https://example.com/robots.txt |
| `robots.warnings` | `array` | 0 items |
| `sitemaps` | `array` | 1 items |
| `sitemaps` | `array` | 1 items |
| `stats` | `object` | 7 fields |
| `stats.elapsed_ms` | `integer` | 1484 |
| `stats.estimated_result_bytes` | `integer` | 657 |
| `stats.inventory_count` | `integer` | 1 |
| `stats.offset` | `integer` | 0 |
| `stats.response_compacted` | `boolean` | false |
| `stats.returned_count` | `integer` | 1 |
| `stats.sitemap_count` | `integer` | 1 |
| `url` | `string` | https://example.com/ |
| `urls` | `array` | 1 items |
| `urls` | `array` | 1 items |
| `warnings` | `array` | 0 items |

## Execution behavior

A `200` response completed inline. A `202` response was queued; poll `GET /jobs/{id}` until the job reaches `completed` or `failed`. Use an `Idempotency-Key` when retrying must not create a duplicate logical job.

## Related documentation

- [Authentication](https://docs.upscrape.com/docs/api/authentication)
- [Jobs and results](https://docs.upscrape.com/docs/api/jobs)
- [Errors and retries](https://docs.upscrape.com/docs/api/errors)
- [Idempotency](https://docs.upscrape.com/docs/api/idempotency)