Sitemap Extractor API

Every URL a website has, from its sitemaps and pages.

The Sitemap Extractor API lists every URL a website has. It reads the site's sitemaps and robots.txt, and follows links when those are not enough. Use it to plan a crawl, check what a site publishes, or watch for new pages.

Call it

curl https://data.upscrape.com/sitemap \  -H "Authorization: Bearer $UPSCRAPE_API_KEY" \  -H "Content-Type: application/json" \  -H "Prefer: wait=30" \  -d '{"url": "https://stripe.com", "max_urls": 20}'

import osimport requestsjob = requests.post(    "https://data.upscrape.com/sitemap",    headers={"Authorization": f"Bearer {os.environ['UPSCRAPE_API_KEY']}", "Prefer": "wait=30"},    json={"url": "https://stripe.com", "max_urls": 20},).json()for entry in job["results"][0]["data"]["urls"]:    print(entry["url"])

const res = await fetch("https://data.upscrape.com/sitemap", {  method: "POST",  headers: {    Authorization: `Bearer ${process.env.UPSCRAPE_API_KEY}`,    "Content-Type": "application/json",    Prefer: "wait=30",  },  body: JSON.stringify({ url: "https://stripe.com", max_urls: 20 }),});const job = await res.json();for (const entry of job.results[0].data.urls) console.log(entry.url);

The list is in results[0].data.urls, each with its url, and its title and description where the site gives them. From a run on staging on 16 October 2026:

Output

https://stripe.com/https://stripe.com/abouthttps://stripe.com/aehttps://stripe.com/ae/atlas/legalhttps://stripe.com/ae/climate/legal…

Narrow the list

Input	What it does
`max_urls`	URLs in this response, up to 5,000. 1,000 by default.
`include_patterns`, `exclude_patterns`	Only URLs that match, or none that do, such as `"path:/blog/*"` or `"contains:pricing"`.
`include_assets`	Also list scripts, styles, images and files. Off by default: pages only.
`include_subdomains`	Also list other subdomains of the same site.
`fetch_titles`	Open each page to fill in its title and description. Slower; off by default.

The next batch

A site with more URLs than max_urls returns continuation.token. Send it back as cursor, with the same other input, for the next batch: Paging through results shows the loop.

Something missing or unclear? Ask the team that builds the endpoints.