Why we built Upscrape: one data layer for apps, agents and the web
What changes when a team depends on scraped data every day instead of once, and the four parts of Upscrape that follow from it: capabilities, durable execution, REST and MCP over one catalog, and schedules.
Upscrape provides real-time data APIs for more than 50 platforms across commerce, grocery and delivery, social, maps and advertising. Every capability has a published schema and returns structured JSON from the live source, through one REST API or an OAuth-secured MCP server for AI clients.
We founded Upscrape in 2024 and spent two years building data pipelines for clients. This post explains what those pipelines taught us and how it shaped the product.
One scrape is easy; depending on it is the work
"Get product data from this website" starts as one request: find the page or endpoint, fetch it, parse the fields. Then it has to run again tomorrow, for hundreds of locations, recover when a request fails, avoid duplicate work when a retry follows a timeout, survive the source changing, add a second platform with a different data model, serve a backend, and finally let an AI agent find and use it. Most of the effort goes into that list, and Upscrape is built around it.
Take Blinkit. A useful request asks for protein bars at one delivery pincode and returns the products shown there now, with product ID, price, MRP, pack size and inventory. Upscrape exposes it as blinkit.search: a query plus a six-digit pincode or coordinates in, product records out. Depending on it every day raises the questions a one-off script never meets: what happens when a run is slow, a connection drops, the site changes, a retry could run twice, or an input stops matching the source.
Capabilities keep what makes each source useful
Blinkit search has location rules that a Reddit post search does not. googlemaps.search takes a location bias with its query, and blinkit.product needs a product ID that means nothing for a generic web page. So Upscrape models each piece of platform work as a capability: a named operation such as blinkit.search or reddit.posts.search with its own documented input. Each one has a published contract holding its input schema, examples, timeout and credit cost, and every public platform page is generated from that contract. Raw output stays open-ended because upstream sources change.
For the rest of the public web, Universal Web captures pages, extracts fields you define, maps and crawls sites, and takes screenshots.
Execution is part of the product
Every REST request goes to one entry point, POST /execute, naming a capability and input that matches its JSON Schema. Quick work completes inline; longer work returns a durable job to poll. That is where the concerns that get rebuilt around every script are handled once:
Retries without duplicates. An idempotency key makes an uncertain retry resolve to the same job.
Rate limits with timing. A rate-limited request says how long to wait.
Stable error codes. Failures carry machine-readable codes (errors and retries).
Pending work you can find again. A job ID survives a timeout on your side.
On 23 September 2026 we sent the same blinkit.location request twice with one operation key, the MCP equivalent of Idempotency-Key:
| First call | Same call again, same key | |
|---|---|---|
| Job ID |
3dc815b0-99f5-…
|
3dc815b0-99f5-…
|
| New execution | Yes |
No: idempotent_replay: true
|
| Collected at | 06:21:06 UTC | 06:21:06 UTC, the original result |
| Result |
12.9698066, 77.7499632, merchant 45631
| The same result |
The retry returned the original job, and the docs state that a replay "does not add another capability charge".
It also moves maintenance. Source access, retries and upstream changes stay behind the endpoint, and the status page shows how each platform is running. The consuming team keeps its application logic, storage, analysis and domain checks, and stops maintaining the same source separately in every consumer.
One execution layer, different consumers
REST suits software that already knows the capability it needs: the application calls blinkit.search and keeps control of retries, storage and scheduling. MCP suits an agent that has to find the capability, read its inputs and call it, which Upscrape's MCP server does with four tools over the same live catalog. Both share the account, input schema, execution, credit cost and result provenance. MCP vs API for AI agents runs one search both ways.
A browser is the right tool when the work is interactive: stateful screens, multi-step flows, actions that exist only in the rendered app. For a backend that needs a product list every day, rebuilding it from a screen made for people ties the application to details it should not depend on. Web scraping for AI agents works through that choice for agents. An official API with the data, acceptable terms and suitable limits may be the most direct route, and a team with one small, stable internal target can reasonably run its own scraper.
Schedules add time and state
A price snapshot every morning, an alert when fields on a public source change, or a dataset in a warehouse on a schedule all need runs to be scheduled, failures told apart from real source changes, and earlier results kept as a baseline. You can build that around the API; the Blinkit price-tracker guide does it with Python, SQLite and cron. Or Monitors, in private beta (join the waitlist with POST /monitors/waitlist), run any published Platform API or Universal Web capability every hour to every 30 days and compare selected fields between runs. Each run is a normal Upscrape job.
A 200 response still needs checking
A technically successful extraction can hold unexpected or incomplete upstream data. Upscrape's provenance guidance treats every field of capability output as untrusted input: validate the values you act on, keep retrieved text out of a model's trusted instructions, and ignore unknown fields as sources evolve. MCP results carry a _provenance marker that says so. The consumer decides what the data means and how much checking a consequential action needs.
Breadth runs through one system
This is how the public catalog looked on 23 September 2026:
- Commerce and delivery 26 platforms 145
- Social and content 14 platforms 140
- Web and maps 9 platforms, including Universal Web 48
- Events and parking 2 platforms 19
- Ad libraries 3 platforms 8
- Business records 1 platform 4
Each capability has its own input schema, and all 364 listed on 23 September 2026 run through the same POST /execute and job system over REST, or upscrape_execute over MCP.
Source: Upscrape public catalog (upscrape.com/scrapers and its machine-readable index), 23 September 2026.
A new platform still needs its own source knowledge, but it arrives through the same catalog, POST /execute and MCP tools as every other, so the applications and agents that use it need no new integration.
What we are building
A single layer for live external data: Platform APIs for more than 50 platforms, MCP access for AI agents such as Claude, ChatGPT and Cursor, and universal web extraction for everything else, all running through one execution model. Applications call capabilities over REST, agents discover them over MCP, and recurring work adds a schedule. Some specialised workloads still deserve their own infrastructure.
Pricing is prepaid and shown per request: each capability shows its credit cost before it runs, credit packs start at $49 for 25,000 credits, purchased credits do not expire, and failed requests are not charged. Custom data projects and enterprise volumes are available on request.
