Web Crawling API

Crawl a whole site and get every page as Markdown.

The Web Crawling API starts at one page, follows its links across the same site and returns every page it reaches as Markdown. Use it to load a documentation site into a search index, or to read a whole site at once.

Call it

curl https://data.upscrape.com/crawl \  -H "Authorization: Bearer $UPSCRAPE_API_KEY" \  -H "Content-Type: application/json" \  -H "Prefer: wait=30" \  -d '{"url": "https://docs.stripe.com/payments", "max_pages": 10}'

import osimport requestsjob = requests.post(    "https://data.upscrape.com/crawl",    headers={"Authorization": f"Bearer {os.environ['UPSCRAPE_API_KEY']}", "Prefer": "wait=30"},    json={"url": "https://docs.stripe.com/payments", "max_pages": 10},).json()for page in job["results"][0]["data"]["pages"]:    print(page["url"], page["title"])

const res = await fetch("https://data.upscrape.com/crawl", {  method: "POST",  headers: {    Authorization: `Bearer ${process.env.UPSCRAPE_API_KEY}`,    "Content-Type": "application/json",    Prefer: "wait=30",  },  body: JSON.stringify({ url: "https://docs.stripe.com/payments", max_pages: 10 }),});const job = await res.json();for (const page of job.results[0].data.pages) console.log(page.url, page.title);

Each page in results[0].data.pages has its url, title, depth and full markdown. From a run on staging on 16 October 2026:

Output

https://docs.stripe.com/payments Payments | Stripe Documentationhttps://docs.stripe.com/ Stripe Documentationhttps://docs.stripe.com/get-started Get started | Stripe Documentationhttps://docs.stripe.com/revenue Revenue | Stripe Documentationhttps://docs.stripe.com/risk Risk | Stripe Documentation…

A crawl of many pages can take more than 30 seconds. The call then answers 202 and you fetch the job when it's done; Long-running calls has a loop that uses this same crawl.

Shape the crawl

Input	What it does
`max_pages`	Pages to fetch in this call, up to 100. 10 by default.
`max_depth`	How many links away from the start page to go, up to 10. 2 by default.
`include_patterns`, `exclude_patterns`	Only follow, or never follow, URLs that match, such as `"path:/docs/*"`.
`include_subdomains`	Also crawl other subdomains of the same site, such as `docs.` and `blog.`.
`respect_robots`	On by default: follow the site's robots.txt.
`delay_ms`	A pause between requests to the site, up to 5 seconds.

Crawl in parts

completion_reason says why it stopped: every page reached, or the max_pages you set. When there is more, the response includes a continuation_token: send it back with the same options and the crawl carries on where it stopped, without repeating pages.

A very large crawl

delivery: "hosted" returns the pages as a file to download, instead of inside the JSON, when there is too much for one response. auto, the default, chooses for you.

Something missing or unclear? Ask the team that builds the endpoints.