Universal Web API
Capture, extract, map, crawl, and screenshot public web content.
web
Capabilities
Every capability uses the same POST /execute envelope. Open an endpoint for
its exact ID, input contract, code samples, cost, runtime, and illustrative output.
web.page.archive
Compatibility alias for saved-page capture with automatic inline or hosted delivery up to 50 MB per asset and 500 MB total. Returns explicit outcome and fidelity, requested versus effective asset budgets, per-asset failures, and an embedded upscrape-manifest.json in ZIP artifacts. New integrations should use web.page.capture.
web.page.capture
Capture one public webpage or content URL with fail-closed HTTP, truncation and OCR semantics, physical-attempt evidence, and optional exact-source, single-HTML or ZIP artifacts.
web.page.extract
Capture one public webpage or supported content URL and return data shaped by a caller-provided JSON Schema or fields shorthand. Deterministic extraction uses page metadata, the evidence graph, parsed files, and filtered URL enumeration first; internal AI resolves remaining fields per the ai mode.
web.page.screenshot
Capture a host-rendered JPEG screenshot of a public viewport or full page through the typed internal browser acquisition boundary.
web.site.crawl
Run a bounded breadth-first crawl across same-site pages and public files with robots rules, explicit scope patterns, controlled concurrency, stateless continuation, source-completeness reconciliation, and optional non-destructive AI relevance ranking.
web.site.map
Build a deterministic, robots-aware public URL inventory from the starting page, robots.txt, and bounded recursive sitemap indexes. Classifies pages and files without deeply fetching every discovered URL and supports opaque pagination cursors.
Common uses
- Turn webpages and public files into traceable structured content, Markdown, and searchable chunks
- Discover downloadable documents, datasets, media, and resources
- Map sitemaps and robots-aware site inventories
- Run bounded, resumable crawls across pages and public files
- Create JPEG screenshots and durable HTML or ZIP snapshots
- Extract caller-defined fields with deterministic and optional AI assistance
Integration contract
A 200 response completed inline. A 202 response was queued; poll
GET /jobs/:id until it reaches completed or failed.