Jobs and results

Over MCP, a call waits for its result up to about 55 seconds. Most finish well before. This page covers the rest: slow calls, large results, and asking for only the part you need.

A slow call

When a call is still running at the end of its wait, upscrape_execute returns the job instead of the result: its job_id, its state, and poll_after_ms, how long to wait before checking.

Fetch it with upscrape_get_job_result:

{"job_id": "d7ef67f5-8cb8-4989-8b55-8ba67a1aab4f"}

Check again, waiting a little longer each time, until the job is completed or failed. Never run the call again because it is slow: that starts, and pays for, a second job. Stopping checking doesn't cancel the job; it finishes, and you can fetch it later.

A large result

An agent sees up to 24 KiB of a result at once. A larger result arrives as a preview, marked as cut. There are three ways to get the rest, and none of them runs the call again or costs anything.

Page through a list

If the result is a list, ask for part of it:

{"job_id": "d7ef67f5-8cb8-4989-8b55-8ba67a1aab4f", "offset": 20, "limit": 20}

result_pagination gives the total and has_more.

Pick the part you need

collection_path picks a list inside the result, and fields keeps only the fields you name. Both are JSON Pointers:

{  "job_id": "d7ef67f5-8cb8-4989-8b55-8ba67a1aab4f",  "collection_path": "/items",  "offset": 0,  "limit": 5,  "fields": ["/name", "/rating"]}

Each item comes back keyed by the pointers, such as {"/name": "The Coffee Movement", "/rating": 4.8}. A field an item doesn't have is left out, never invented. In a pointer, write / as ~1 and ~ as ~0.

Read it whole

To get the complete result exactly as stored, read it in pieces:

{"job_id": "d7ef67f5-8cb8-4989-8b55-8ba67a1aab4f", "format": "json_chunk", "byte_offset": 0, "chunk_bytes": 8192}

1. Append result_chunk.text exactly as it comes.

2. Ask again from result_chunk.next_byte_offset, always the offset returned, until has_more is false.

3. Check the total length against total_bytes and the hash against sha256, then parse the JSON.

chunk_bytes is 4 to 16,384 bytes; a piece can come back shorter. For a whole dataset, a script is better than an agent reading it piece by piece: the same job is available over REST with GET /jobs/{id} and an API key.

What it costs

A call is charged once, at its price, when it succeeds. Waiting, checking, fetching results and failed calls are free.

Retrying safely

Give each call an operation_key. If the answer is lost, because the connection dropped or the app reconnected, send the same call with the same key: you get the first job, not a new one. A new call needs a new key.

Without a key, the same call with the same request ID and arguments within about 15 minutes returns the first job; after that it starts a new one.

A failed job stays failed when you send it again with its key. retryable: true means trying again later may work; do that with a new operation_key. When upscrape_execute answers temporarily_unavailable with retryable: false, no job was created and nothing was charged: the endpoint can't run right now, so try another one or try again later.

Good to know

• collected_at is when the job finished. result_complete: false means you are seeing part of the result.

• Disconnecting doesn't cancel calls already started.

• The server doesn't send progress updates or cancel a call when the app disconnects; the job_id is how you follow it.