> ## Documentation Index > Fetch the complete documentation index at: https://docs.hanji.dev/llms.txt > Use this file to discover all available pages before exploring further. # Batch processing > Async batch extraction. Upload many files, submit one batch, poll for results. Process thousands of documents without managing thousands of requests. You upload your files, hand us the list as one batch, and poll a single endpoint until it's done. Same engine as sync. Same response schema. Same credit billing as the sync API. The only difference is *how* you submit and *how* you fetch results. ## When to use batch Use **async batch** when: * You're processing more than \~50 documents in a single workflow. * You'd otherwise have to write a retry loop around `POST /v1/parse/file`. * The documents are large (long PDFs) and you'd rather poll than hold a connection open. Stick with **[sync](/introduction#quickstart)** when: * You need the result inline with the request (interactive agent loops, screenshots). * The document is small (under 10 pages) and a multi-second response is fine. * You're already at human-perceptible latency on the user side. ## How it works Two ways to hand us documents. Use whichever fits how your data already lives. **By upload** — you control the bytes end to end: ``` 1. POST /v1/files (per file) → file_id + upload URL 2. PUT (per file) → upload the file bytes 3. POST /v1/batches (once) → batch_id, status="pending" 4. GET /v1/batches/{id} (poll loop) → per-item status as items finish 5. GET /v1/batches/{id}/items/{id}/result → 302 redirect to the result JSON ``` **By URL** — skip steps 1-2 entirely; we fetch: ``` 1. POST /v1/batches (once) → {"source": {"type": "urls", "urls": [...]}} 2. GET /v1/batches/{id} (poll loop) → per-item status as items finish 3. GET /v1/batches/{id}/items/{id}/result → 302 redirect to the result JSON ``` See **[Submitting by URL](#submitting-by-url)** below — best when the files already live behind a URL (your storage, a customer's presigned link). For PHI and other sensitive workloads, this is also the path where **source documents are never written to our storage**; only the extraction result is retained (same 3-day window as uploads). A few properties worth knowing before you build: * **Same input formats as sync.** PDF, PPTX, DOCX, and raster images (PNG, JPEG, WebP, TIFF, HEIC/HEIF, BMP). Images bill one page each; multi-frame TIFFs bill one page per frame. * **Uploads go straight to storage.** `POST /v1/files` gives you a presigned URL and you PUT the file bytes to it directly, so uploads run at your connection's full speed. * **No separate "confirm upload" step.** Hand the `file_id`s straight to `POST /v1/batches` once your PUTs return; a finished upload is always accepted, even if you submit immediately. (Want to confirm a single upload landed first? `GET /v1/files/{file_id}` returns its `status`.) * **3-day retention.** Uploaded inputs and result blobs both auto-expire after **3 days**. The clock starts at upload (for inputs) or at item completion (for results). Need longer? Email [hello@hanji.dev](mailto:hello@hanji.dev). We're happy to bump it to a week or more on request. URL-sourced items skip the input side of this entirely — see below. * **Retries are safe.** Send an `Idempotency-Key` header with `POST /v1/batches`. Retrying with the same key within 3 days returns the same batch instead of creating a duplicate, so you're never double-billed. ## End-to-end example This loops over a local directory, uploads everything in parallel, submits one batch, polls until all items reach a terminal state, and writes each result to disk. ```python python theme={null} import asyncio, os, time from pathlib import Path import httpx API = "https://api.hanji.dev" HEADERS = {"X-API-KEY": os.environ["HANJI_API_KEY"]} async def upload(client: httpx.AsyncClient, path: Path) -> str: meta = (await client.post( f"{API}/v1/files", json={"filename": path.name, "size_bytes": path.stat().st_size}, )).json() async with httpx.AsyncClient() as raw: await raw.put( meta["upload"]["url"], content=path.read_bytes(), headers={"Content-Type": "application/octet-stream"}, timeout=600, ) return meta["id"] async def main(input_dir: str, output_dir: str) -> None: files = sorted(p for p in Path(input_dir).rglob("*.pdf") if p.is_file()) Path(output_dir).mkdir(parents=True, exist_ok=True) async with httpx.AsyncClient(headers=HEADERS, timeout=60) as client: sem = asyncio.Semaphore(10) async def _bound(p): async with sem: return await upload(client, p) file_ids = await asyncio.gather(*[_bound(p) for p in files]) batch = (await client.post( f"{API}/v1/batches", headers={"Idempotency-Key": f"my-run-{int(time.time())}"}, json={"source": {"type": "files", "file_ids": file_ids}}, )).json() print("submitted batch", batch["id"], "with", batch["total_items"], "items") while True: state = (await client.get(f"{API}/v1/batches/{batch['id']}")).json() print(state["status"], state["counts"]) if state["status"] in {"completed", "partially_failed", "failed", "cancelled", "expired"}: break await asyncio.sleep(3) for item in state["items"]: if item["status"] != "succeeded": continue r = await client.get( f"{API}{item['result_url']}", follow_redirects=True ) (Path(output_dir) / f"{item['id']}.json").write_bytes(r.content) asyncio.run(main("./pdfs", "./extracted")) ``` ```javascript javascript theme={null} import { readFile, stat, mkdir, writeFile, readdir } from "node:fs/promises"; import { join } from "node:path"; const API = "https://api.hanji.dev"; const HEADERS = { "X-API-KEY": process.env.HANJI_API_KEY }; async function upload(path) { const size = (await stat(path)).size; const meta = await (await fetch(`${API}/v1/files`, { method: "POST", headers: { ...HEADERS, "Content-Type": "application/json" }, body: JSON.stringify({ filename: path.split("/").pop(), size_bytes: size }), })).json(); await fetch(meta.upload.url, { method: "PUT", body: await readFile(path) }); return meta.id; } async function main(inputDir, outputDir) { await mkdir(outputDir, { recursive: true }); const entries = await readdir(inputDir); const pdfs = entries.filter(f => f.endsWith(".pdf")).map(f => join(inputDir, f)); const fileIds = await Promise.all(pdfs.map(upload)); const batch = await (await fetch(`${API}/v1/batches`, { method: "POST", headers: { ...HEADERS, "Content-Type": "application/json", "Idempotency-Key": `my-run-${Date.now()}` }, body: JSON.stringify({ source: { type: "files", file_ids: fileIds } }), })).json(); console.log("submitted batch", batch.id, "with", batch.total_items, "items"); let state = batch; const terminal = new Set(["completed", "partially_failed", "failed", "cancelled", "expired"]); while (!terminal.has(state.status)) { await new Promise(r => setTimeout(r, 3000)); state = await (await fetch(`${API}/v1/batches/${batch.id}`, { headers: HEADERS })).json(); console.log(state.status, state.counts); } for (const item of state.items) { if (item.status !== "succeeded") continue; const r = await fetch(`${API}${item.result_url}`, { headers: HEADERS, redirect: "follow" }); await writeFile(join(outputDir, `${item.id}.json`), Buffer.from(await r.arrayBuffer())); } } main("./pdfs", "./extracted"); ``` ```bash curl theme={null} # 1. Reserve a slot, get a presigned PUT URL. curl -sS -X POST https://api.hanji.dev/v1/files \ -H "X-API-KEY: $HANJI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"filename": "chart.pdf", "size_bytes": 18342}' # → { "id": "file_xxx", "upload": { "url": "https://...?X-Amz-Signature=...", ... } } # 2. Upload the file bytes to the presigned URL. curl -X PUT --data-binary @chart.pdf "$UPLOAD_URL" # 3. Submit the batch (one or many file_ids). curl -sS -X POST https://api.hanji.dev/v1/batches \ -H "X-API-KEY: $HANJI_API_KEY" \ -H "Idempotency-Key: my-run-2026-05-07" \ -H "Content-Type: application/json" \ -d '{"source": {"type": "files", "file_ids": ["file_xxx"]}}' # → { "id": "batch_yyy", "status": "pending", "total_items": 1 } # 4. Poll until terminal. curl -sS "https://api.hanji.dev/v1/batches/batch_yyy" -H "X-API-KEY: $HANJI_API_KEY" # 5. Fetch a successful item's result JSON. curl -L "https://api.hanji.dev/v1/batches/batch_yyy/items/item_zzz/result" \ -H "X-API-KEY: $HANJI_API_KEY" ``` An item's result JSON is exactly what the sync endpoint would have returned for that document, including the [`usage`](/introduction#pricing-and-credits) object (`pages`, `credits`, `credits_per_page`) on successfully billed items. DOCX/PPTX items also include [`pdf_rendition_url`](/introduction#pdf-rendition); that link lasts as long as the result JSON (3 days), so copy the PDF when you fetch the item. ## Submitting by URL Pass `source: {"type": "urls", "urls": [...]}` instead of `file_ids` and skip the upload step — no `POST /v1/files`, no `PUT`. Each URL becomes one item; we fetch it when that item starts processing, not when you submit the batch, so a slow or large download doesn't block the submit call. (Per-file limits are the same as elsewhere: 150 MB / 2,000 pages.) ```bash curl theme={null} curl -sS -X POST https://api.hanji.dev/v1/batches \ -H "X-API-KEY: $HANJI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "source": { "type": "urls", "urls": [ "https://example.com/report.pdf", "https://your-bucket.s3.amazonaws.com/inbox/report.pdf?X-Amz-Signature=..." ] } }' # → { "id": "batch_yyy", "status": "pending", "total_items": 2 } ``` Poll and fetch results exactly as you would for uploaded files — everything downstream of submission (status lifecycle, webhooks, pagination, item errors) is identical. A few things specific to URL input: * **Public and presigned URLs both work.** A presigned URL's signature lives in the query string; we use the URL exactly as given, so pass it through unmodified. Give it enough TTL to survive queue time — if the signature expires before we fetch it, the fetch fails the same way an already-expired URL would (see `url_fetch_failed` below). * **Source bytes are never written to our storage.** The document is held only for the length of the extraction; only the *result* JSON is persisted (3-day retention, same as the upload path). If keeping PHI or other sensitive source documents off our storage is a requirement, this is the path to use. * **HTTP(S) only**, up to 100 URLs per batch (upload batches allow up to 10,000 `file_ids`). * **`GET /v1/batches/{id}` echoes back `url`** on each item (query string stripped, so a presigned signature never shows up in a poll response) instead of `file_id`, which is `null` for these items. `items[].error.message` gets the same treatment for `url_fetch_failed` items. * **Failed fetches aren't retried when the failure is permanent** — a bad URL, a 403/404, or an expired presigned signature all fail the item immediately as `url_fetch_failed` rather than retrying a failure that can't succeed. Transient failures (5xx, timeouts) retry exactly like any other item. ## Batch options `POST /v1/batches` accepts the same extraction options as sync, applied to every item in the batch, plus a `metadata` object for tying the batch back to your own system: | Field | Type | Default | Description | | --------------------- | --------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `source` | object | required | `{"type": "files", "file_ids": [...]}` (up to 10,000 ids), or `{"type": "urls", "urls": [...]}` (up to 100) — see [Submitting by URL](#submitting-by-url). | | `extract_text` | boolean | `true` | Set `false` to skip text spans in every item. | | `extract_images` | boolean | `true` | Set `false` to skip figure extraction in every item. | | `table_output_format` | `"markdown" \| "cell_grid"` | `"markdown"` | Same as sync: `"cell_grid"` returns a true bounding box for every table cell in each item's result. | | `chunking` | `"none" \| "semantic"` | `"none"` | Same as sync: `"semantic"` adds RAG-ready [`segments`](/introduction#chunking) to every item's result. | | `chunk_size` | integer | `1000` | Same as sync: target segment size in characters (±25% band); validated 200–8000 only when chunking is enabled. | | `metadata` | object | `null` | Arbitrary JSON stored with the batch and echoed back on every poll and in the batch list. Put your own run ids, tenant ids, or job references here. | On the upload side, `POST /v1/files` takes a `filename` and a required `size_bytes`, plus an optional `content_type`. After the bytes land, `GET /v1/files/{file_id}` reports a `sha256` of what arrived, so you can verify the upload byte for byte. ## Polling cursor `GET /v1/batches/{id}` returns items in pages. To only fetch what changed since your last call, pass back the `next_cursor` from the previous response as `?cursor=...`. Cursors are opaque; treat them as strings. ```bash theme={null} # First page curl ".../v1/batches/batch_yyy?limit=100" # → { "items": [...], "next_cursor": "MjAyNi0wNS0wN1QxMjowMDowMC..." } # Next page (or "what's changed since I last polled") curl ".../v1/batches/batch_yyy?limit=100&cursor=MjAyNi0wNS0wN1QxMjowMDowMC..." ``` A typical client polls every 2-5 seconds without a cursor (always seeing the full current state) until terminal, then walks the cursor to drain the final list. For very large batches (10k+ items), pass a cursor so you only get the items that changed. `limit` defaults to 100 items per page and caps at 500. Beyond `status`, `result_url`, and `error`, each item carries `file_id` or `url` (whichever source type you submitted — the other is `null`), `position` (its index in the source list you submitted), `page_count` (the pages it billed, once parsed), `attempts`, and `started_at` / `completed_at` / `updated_at` timestamps. To see all your batches, `GET /v1/batches` lists them newest first with the same cursor pattern, an optional `?status=` filter, and a `limit` of up to 200 (default 50). ## Completion webhooks Instead of polling, you can have us **POST you a signed event the moment a batch finishes**. This is the recommended production integration path — polling `GET /v1/batches/{id}` is limited to 200 requests per second per organization and may return `429 Too Many Requests` when that limit is exceeded. Register an endpoint in the [dashboard](https://hanji.dev/dashboard/webhooks) (you get a `whsec_…` signing secret), then opt any batch in with a `webhook` field: ```bash theme={null} curl -sS -X POST https://api.hanji.dev/v1/batches \ -H "X-API-KEY: $HANJI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "source": { "type": "files", "file_ids": ["file_xxx"] }, "webhook": { "mode": "svix" } }' ``` When the batch reaches a terminal status we POST a thin `batch.update` event — ids, `status`, `counts`, your echoed `metadata`, no results inline: ```json theme={null} { "type": "batch.update", "batch_id": "batch_yyy", "status": "partially_failed", "counts": { "pending": 0, "running": 0, "succeeded": 4, "failed": 1, "cancelled": 0 }, "total_items": 5, "metadata": { "prior_auth_id": "PA-1234" }, "completed_at": "2026-08-05T12:34:56Z", "results_expires_at": "2026-08-08T12:34:56Z" } ``` Then fetch results exactly as you would after polling (`GET /v1/batches/{batch_id}` → per-item `result_url`). The signature is [Svix](https://www.svix.com/)-wire-compatible, so you verify it with the standard `svix` library. `status` is one of `completed`, `partially_failed`, `failed`, `cancelled`. Treat **both** `completed` and `partially_failed` as "results are ready" — `partially_failed` means some items succeeded and some did **not** (failed **or** were cancelled; `counts.failed` may be `0`), not a total failure. A handler that only branches on `completed` will silently drop the successful results of partially-failed batches. Webhooks are **opt-in per batch** (omit the field and none fire), deliver **at-least-once** (dedup on the `svix-id` header), and retry for \~27 hours before giving up. The full setup — handler examples in Python and TypeScript, secret rotation, the test ping, direct (unsigned) mode, and the deliveries dashboard — is in the **[Webhooks guide](/guides/webhooks)**. ## Status lifecycle A **batch** moves through: ``` pending → running → completed | partially_failed | failed | cancelled | expired ``` An **item** moves through: ``` pending → running → succeeded | failed | cancelled ``` `partially_failed` means at least one item succeeded and at least one did **not** (failed or was cancelled); treat it the same as `completed` and inspect `items[].error.code` for any failures. Items aren't retried for terminal errors: if a document is unsupported (`unsupported_input`), too large (`page_limit_exceeded` / `document_too_large`), or a URL couldn't be fetched (`url_fetch_failed`), the same item won't succeed on a re-poll. Fix the inputs and submit a new batch. ## Errors Batch errors come in two shapes, both documented here. ### Request errors The call itself was rejected: bad `file_id`, file not uploaded, result not ready. `error` is a flat **string** code you can switch on: ```json theme={null} { "error": "file_not_uploaded", "file_ids": ["file_abc123"] } ``` | `error` | Status | Endpoint | Meaning + fix | | ------------------- | ------ | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `file_not_found` | `404` | `POST /v1/batches` | A `file_id` doesn't exist for your account, or its 3-day TTL lapsed. The offending ids come back in `file_ids`. | | `file_not_uploaded` | `409` | `POST /v1/batches` | A `file_id`'s bytes aren't in storage. A *finished* upload won't hit this: it means the PUT never completed or the upload expired. Re-upload the ids in `file_ids` and resubmit. | | `result_not_ready` | `409` | `GET …/items/{id}/result` | The item hasn't reached `succeeded` yet (the `status` field tells you the current state). Keep polling `GET /v1/batches/{id}`. | ### Item errors These are *item* failures: the batch call itself succeeded, but a document failed during processing. The shape is **nested**. `item.error` is an object, distinct from the flat request-error string above: ```json theme={null} { "error": { "code": "unsupported_input", "message": "…" } } ``` When `item.status == "failed"`, `item.error.code` is one of: | Code | Meaning | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `payment_required` | Customer is out of credits. The whole batch will hit this once it triggers. Top up and re-submit a fresh batch. | | `unsupported_input` | The file is empty, or isn't a supported format. We check the file itself, not its name. | | `document_too_large` | Source bigger than 150 MB. | | `page_limit_exceeded` | Source has more than 2,000 pages. | | `extraction_failed` | Generic extraction error (corrupted PDF, missing fonts, etc.). | | `ocr_provider_error` | Transient extraction outage. The item was retried automatically before failing; re-submit it in a new batch. | | `upload_missing` | The file was never uploaded, or the 3-day retention window passed before processing started. Re-upload and re-submit. | | `url_fetch_failed` | URL-sourced item only. The URL was invalid, forbidden, not found, or a presigned signature had expired before we fetched it. Not retried — fix the URL (or reissue the presigned link with more TTL) and re-submit. | | `internal_error` | Unexpected server error. Re-submit the item; contact support if it repeats. | ## Cancelling `POST /v1/batches/{id}/cancel` flips remaining `pending` items to `cancelled`. Items already `running` finish on their own; we don't kill in-flight work. Cancelled items are not billed. The batch's terminal status will be `cancelled` if no items succeeded; `partially_failed` or `completed` if some did. ## Concurrency By default we process up to **8 items concurrently** per customer, so one large batch can't starve other customers. Email us if you need higher concurrency for sustained workloads. The cap is server-side and nothing you manage: items simply wait in `pending` until capacity frees up. ## Limits | Limit | Default | | ------------------------------------------------------------ | ----------------------------------------------------------------------------- | | Max files per batch (`file_ids`) | 10,000 | | Max URLs per batch (`urls`) | 100 | | Max page count per file | 1,000 | | Max file size | 150 MB | | Result + upload retention | 3 days (input clock starts at upload; output clock starts at item completion) | | Idempotency key dedup window | 3 days | | Submission rate limit (`POST /v1/batches`, `POST /v1/files`) | 60/min per key | ## Next steps Stop polling. Get a signed `batch.update` event the moment a batch finishes. Need structured fields instead of full-document chunks? Define a JSON schema and get cited values back.