# Scrape

> Fetch a page and extract structured data from it against a JSON schema.

Source: https://cloud.gomry.com/services/scrape

Give Scrape a URL and a JSON schema and it returns that shape, filled in from the page. It renders JavaScript first, so single-page apps and pages that build their content client-side work the same as static HTML.

## Overview

Two operations, priced separately because they cost differently. POST /v1/scrape renders the page and runs an LLM extraction against your schema, and is billed per page. POST /v1/fetch renders the page and returns its raw HTML plus metadata with no extraction, at a quarter the price — reach for it when you only need og:image, JSON-LD, or the raw HTML and intend to parse it yourself.

Only calls that returned a page are billed. A failed scrape, a timeout, or an unreachable host costs nothing. The response tells you what was metered in its `pages` or `fetches` field, so what you were charged is visible in the same payload as the result.

What it does not do: there is no search endpoint. Scrape answers questions about a URL you already have — it will not find one for you, and a caller that needs 'the best image for this event name' has to bring its own candidate URLs. It also does not crawl: one call is one page, and following links is yours to orchestrate.

## Endpoints

**POST /v1/scrape** — Fetch a page and extract structured data from it against a JSON schema.

```bash
curl -X POST https://api.cloud.gomry.com/v1/scrape \
  -H "Authorization: Bearer $GOMRY_CLOUD_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/events/summer-fest",
    "prompt": "Extract the event",
    "schema": { "type": "object", "properties": {
      "name": { "type": "string" }, "startDate": { "type": "string" }, "venue": { "type": "string" } } }
  }'
```

The response reports `pages` — the quantity you were metered for this call.

**POST /v1/fetch** — Fetch a page and return its raw HTML and metadata, without LLM extraction. A quarter the price.

```bash
curl -X POST https://api.cloud.gomry.com/v1/fetch \
  -H "Authorization: Bearer $GOMRY_CLOUD_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com/events/summer-fest" }'
```

The response reports `fetches` — the quantity you were metered for this call.

## Request options

- `url` — Required. The page to scrape. Max 2048 characters. Private addresses and internal hostnames are refused.
- `schema` — Required for /v1/scrape. A JSON Schema object describing the shape you want back. Returned in `data`.
- `prompt` — Required for /v1/scrape. 1–2000 characters of instruction for the extraction, e.g. "Extract the event".
- `waitFor` — Milliseconds to wait after load before reading the page, for content that arrives late. 0–15000.
- `timeout` — Milliseconds before the vendor call is abandoned. 1000–90000. The whole request is capped at 120s server-side, so budget for two calls if you use `retry`.
- `onlyMainContent` — Strip navigation, footers and boilerplate before extraction. Defaults to true; set false when the data you want lives in the chrome.
- `includeRawHtml` — Also return the page's raw HTML in `rawHtml`. Use when you parse JSON-LD or embedded blobs yourself.
- `retry` — { requiredField, waitFor }. If the first pass returns without that field, the page is fetched once more with the longer wait. Bills both calls, and only when each returned a page.

## Pricing

| Unit | Price |
| --- | --- |
| per page | $0.02 |
| per fetch | 0.5¢ |

Only calls that returned a result are billed. Failures cost nothing. Billed monthly in arrears — see [Billing](https://cloud.gomry.com/docs/billing).

## Authentication

Send a key as a bearer token. This service's operations require the `scrape:run` and `scrape:fetch` scopes, granted independently — and the project must have Scrape enabled. See [Authentication](https://cloud.gomry.com/docs/authentication).