# ScrapeWhale API > Web data for marketing agents: traffic, ads, social profiles, and clean markdown from any URL or file — every endpoint returns structured JSON and/or agent-ready markdown. Every response is built for agents: structured `data` for post-processing, clean `markdown` for dropping straight into a context window — your choice per request. Machine-readable spec: [/openapi.json](/openapi.json) · agent summary: [/llms.txt](/llms.txt). # Getting started ## Quick start Create an API key at [/settings/apikeys](/settings/apikeys) — new accounts get **10 free credits**, no card required. Then make your first call: ```bash curl 'https://scrapewhale.dev/api/v1/traffic?domain=nike.com' \ -H 'Authorization: Bearer ' ``` Or pull a full brand dossier — logo, traffic, authority, ads, and the homepage as markdown, in one call: ```bash curl 'https://scrapewhale.dev/api/v1/brand?domain=nike.com' \ -H 'Authorization: Bearer ' ``` ## Authentication Create an API key at **/settings/apikeys**, then send it on every request: ``` Authorization: Bearer ``` (`x-api-key: ` also works.) Keys are rate-limited to 60 requests per minute; credits are the real spend cap. Logged-in users can also run every endpoint from the **Playground** at [/playground](/playground) — same endpoints, same credits. ## Credits Every successful request consumes credits from your balance (new accounts start with 10 free credits; credit packs never expire): | Endpoints | Credits | | --- | --- | | URL / file → markdown, search, maps, trends, social profiles, X/Twitter posts, YouTube video/channel/transcript/search, TikTok Shop lookups, Shopify catalogue, Google ad creative/advertiser, traffic & SEO metrics, domain facts, Chrome Web Store | 1 | | Ads endpoints, the TikTok Shop catalog, and AI Overview (heavy upstream crawls/renders) | 2 | **Failed requests are refunded automatically** — you are never charged for an error. **Cache hits are free**: data endpoints serve recently-fetched public data (6–24h freshness windows; logos 7 days) from a shared cache at no credit cost — the response carries `"cached": true` and `"creditsCharged": 0`. ## Output formats Extraction endpoints accept `?format=json|markdown|both` (default `both`): - `json` — structured payload only (`data`) - `markdown` — clean markdown only (`markdown`) - `both` — both fields in one response Both artifacts are stored with the conversion and can be re-fetched later via `GET /api/v1/conversions/{id}`. ## Errors Errors are JSON `{ "error": "", "code": "" }`: | Status | Meaning | | --- | --- | | 400 | Invalid parameters | | 401 | Missing or invalid API key | | 402 | Insufficient credits | | 404 | Not found | | 422 | Extraction/conversion failed (`code` carries the failure category; credits refunded) | | 429 | Rate limit exceeded | File uploads can additionally return 413 (over 10 MB) or 415 (unsupported or blocked type — HTML files are rejected; convert web pages by URL instead). # Integrations ## MCP server A hosted MCP server at `https://scrapewhale.dev/mcp` (Streamable HTTP) exposes every endpoint as tools, grouped by platform (`youtube`, `ads`, `domain_intel`, `google_trends`, …), plus `brand_dossier` and `get_result`. Nothing to install: sign in with your ScrapeWhale account, or send the same API key as REST. Tool calls cost the same credits as the matching endpoint. **Claude (claude.ai / Desktop)** — Settings → Connectors → *Add custom connector*, URL `https://scrapewhale.dev/mcp`, then sign in and approve. Other clients with OAuth support (e.g. ChatGPT) use the same sign-in flow. Connected apps can be disconnected at [/settings/apikeys](/settings/apikeys). **Claude Code** — sign in with your account (no key needed): ```bash claude mcp add --transport http scrapewhale https://scrapewhale.dev/mcp ``` Then run `/mcp` inside Claude Code, pick **scrapewhale** → *Authenticate*, and approve in the browser. Or use an API key instead (handy for CI and headless runs): ```bash claude mcp add --transport http scrapewhale https://scrapewhale.dev/mcp \ --header 'Authorization: Bearer ' ``` Add `--scope user` to make it available in every project. **Codex** (`~/.codex/config.toml`, with the key in `SCRAPEWHALE_API_KEY`) ```toml [mcp_servers.scrapewhale] url = "https://scrapewhale.dev/mcp" bearer_token_env_var = "SCRAPEWHALE_API_KEY" ``` **Cursor** and other clients that take JSON config (`.cursor/mcp.json`) ```json { "mcpServers": { "scrapewhale": { "url": "https://scrapewhale.dev/mcp", "headers": { "Authorization": "Bearer " } } } } ``` Tools return markdown by default; pass `format: "json"` for the structured data. Local file conversion is not available over the hosted server — upload files with `POST /api/v1/markdown/file`. ## Using ScrapeWhale in Claude & ChatGPT Once connected, ask in plain language — the assistant picks the tool. Example prompts: - "Build a brand dossier for allbirds.com: traffic, authority, the ads they're running, and what their homepage says." - "What ads is Liquid Death running on Facebook right now? Summarize the main angles." - "Compare estimated monthly traffic and traffic sources for notion.so, coda.io, and clickup.com." - "Pull the transcript of this YouTube video and list the product claims: https://www.youtube.com/watch?v=…" - "Find coffee roasters in Austin on Google Maps with a rating above 4.5." - "What's trending on Google in the US today, and how has interest in 'protein coffee' changed over the past year?" - "Read https://example.com/pricing and turn it into a comparison table." **What it can't do:** it reads public data only — no logging in to other sites, no posting or changing anything. Every tool is read-only. **Limits:** each tool call spends the same credits as its API endpoint (most cost 1, ads and AI Overview 2, the brand dossier up to 6); failed calls are refunded and recently fetched data is free. Results over 50,000 characters are truncated, with an `id` to fetch the rest via `get_result`. API keys are limited to 60 requests per minute. **Disconnect:** remove the connector in your AI app, or revoke it under Connected apps at [/settings/apikeys](/settings/apikeys). **Support:** email support@scrapewhale.dev or use [/contact](/contact). Machine-readable spec: https://scrapewhale.dev/openapi.json Interactive reference: https://scrapewhale.dev/docs ## Endpoints ### Web #### GET /api/v1/markdown/url — URL to markdown Any public web page as clean, LLM-ready markdown — headings, lists, tables, links, and images preserved; nav, ads, and boilerplate removed. JavaScript-rendered pages included. Plain page reading only: YouTube, X, Facebook, and other structured sources have their own endpoints and are not auto-detected here. Recently read pages are served from cache for free. Credits: 1. Parameters: - `url` (required) — The web page to convert. Example: ```bash curl 'https://scrapewhale.dev/api/v1/markdown/url?url=https://example.com/article' -H 'Authorization: Bearer ' ``` #### POST /api/v1/markdown/file — File to markdown Upload a document and get it back as markdown: PDF, Word, Excel, CSV, OpenDocument, Apple Numbers, and images (scanned PDFs and images are OCR'd). `multipart/form-data` with a `file` field, 10 MB cap; HTML files are rejected — use the URL endpoint for web pages. The upload is validated before any credit is charged. Credits: 1. Example: ```bash curl -X POST 'https://scrapewhale.dev/api/v1/markdown/file' -H 'Authorization: Bearer ' -F 'file=@report.pdf' ``` ### Search #### GET /api/v1/search — Google web search Google web search results for any query: organic results (title, link, snippet, position), answer box, knowledge graph, videos, People Also Ask, and related searches. Raw payload in `data`, a clean readable digest in `markdown`. Field fullness varies by upstream backend (People Also Ask and videos can come back empty). Results are cached for 1 hour — repeat queries are free. Credits: 1. Parameters: - `q` (required) — The search query. - `gl` — Country code (Google gl), e.g. "us", "gb", "de". - `hl` — Language code (Google hl), e.g. "en", "es". - `num` — Results per page (1–20, default 10). - `page` — Result page (1–10, default 1). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/search?q=best+running+shoes+2026' -H 'Authorization: Bearer ' ``` #### GET /api/v1/search/ai-overview — Google AI Overview Google’s full AI Overview for a query — the rich multi-source answer shown above the results: ordered text blocks (paragraphs, headings, lists) with `[n]` citation markers linking into a references list of cited sources. The key input for answer-engine (GEO) monitoring: see what Google’s AI says about a topic and which domains it cites. Overviews exist for roughly half of queries and Google generates them probabilistically — when none is produced the request fails with `empty_result` and the credits are refunded (retrying later can succeed). Cached for 24 hours. Credits: 2. Parameters: - `q` (required) — The search query. - `gl` — Country code (Google gl), e.g. "us", "gb", "de". - `hl` — Language code (Google hl), e.g. "en", "es". - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/search/ai-overview?q=best+crm+for+small+business' -H 'Authorization: Bearer ' ``` ### Maps #### GET /api/v1/maps — Google Maps place search Google Maps place search for any query: name, address, coordinates, rating and review count, type, website, phone, price level, opening hours, booking links, and whether the listing is claimed by its owner (`claimed: false` is a lead-gen signal). Each place carries a `cid` usable with `/api/v1/maps/place`. Without `ll` results are biased by the scraper exit IP’s location — pass `ll` for deterministic results, especially when paginating. Credits: 1. Parameters: - `q` (required) — The place search query. - `ll` — Location bias as `@lat,lng,zoom`, e.g. `@40.7504,-73.9824,14z`. - `gl` — Country code, e.g. "us", "gb", "de". - `hl` — Language code, e.g. "en", "es". - `page` — Page of 20 places (1–5, default 1). Deeper pages widen the search radius. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/maps?q=coffee+shops+near+5th+avenue&ll=@40.7504,-73.9824,14z' -H 'Authorization: Bearer ' ``` #### GET /api/v1/maps/place — Google Maps place details Full details for one place by `cid` (from `/api/v1/maps` or a `maps.google.com/?cid=` link) or `fid`: everything the search returns plus the owner-written long description and the canonical Maps URL. Credits: 1. Parameters: - `cid` — Place CID (decimal). Provide `cid` or `fid`. - `fid` — Place FID (`0x…:0x…`), alternative to `cid`. - `gl` — Country code, e.g. "us". - `hl` — Language code, e.g. "en". - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/maps/place?cid=6182067377655413673' -H 'Authorization: Bearer ' ``` ### Trends #### GET /api/v1/trends/interest — Google Trends — interest over time Search-interest timeline for up to 5 comma-separated keywords, compared against each other like the Trends UI. Values are Google’s normalized 0–100 indexes, not absolute volumes. Credits: 1. Parameters: - `q` (required) — Keyword(s) — up to 5, comma-separated. - `geo` — ISO country ("US"), subregion ("US-CA"), or empty for worldwide. - `time` — Timeframe in Trends syntax: "now 1-H", "now 7-d", "today 1-m", "today 3-m", "today 12-m", "today 5-y", "all", or "YYYY-MM-DD YYYY-MM-DD". - `cat` — Trends category id (0 = all). - `gprop` — Property: omit for web search, or "images", "news", "youtube", "froogle". - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/trends/interest?q=nextjs,nuxt&geo=US' -H 'Authorization: Bearer ' ``` #### GET /api/v1/trends/related — Google Trends — related queries Related queries for a keyword: top queries (0–100 index) and rising queries (growth percentages; "Breakout" means above 5000% growth). Useful for keyword expansion and spotting emerging demand. Credits: 1. Parameters: - `q` (required) — The keyword. - `geo` — ISO country ("US"), subregion ("US-CA"), or empty for worldwide. - `time` — Timeframe in Trends syntax (see /trends/interest). - `cat` — Trends category id (0 = all). - `gprop` — Property: omit for web search, or "images", "news", "youtube", "froogle". - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/trends/related?q=protein+powder&geo=US' -H 'Authorization: Bearer ' ``` #### GET /api/v1/trends/geo — Google Trends — interest by region Where a keyword is searched most: regions ranked by normalized interest (0–100). `resolution` picks the granularity — COUNTRY, REGION, DMA, or CITY (CITY only returns data for subregion geos like `geo=US-CA`, and sparsely). Credits: 1. Parameters: - `q` (required) — The keyword. - `geo` — ISO country ("US"), subregion ("US-CA"), or empty for worldwide. - `resolution` — Region granularity. - `time` — Timeframe in Trends syntax (see /trends/interest). - `cat` — Trends category id (0 = all). - `gprop` — Property: omit for web search, or "images", "news", "youtube", "froogle". - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/trends/geo?q=vercel&geo=US&resolution=DMA' -H 'Authorization: Bearer ' ``` #### GET /api/v1/trends/trending — Google Trends — trending now Currently trending searches for a country, each with an approximate traffic figure and related news links. Cached for 1 hour — trending data moves fast. Credits: 1. Parameters: - `geo` — 2-letter country code. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/trends/trending?geo=US' -H 'Authorization: Bearer ' ``` ### Traffic & SEO #### GET /api/v1/traffic — Estimated website traffic Traffic estimate for a domain: global/country/category rank, monthly visits, engagement (bounce rate, pages per visit, visit duration), traffic sources, top countries, top keywords, and AI-assistant-referred traffic with sample prompts. Credits: 1. Parameters: - `domain` (required) — The website to analyze. Accepts a bare domain or a full URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/traffic?domain=nike.com' -H 'Authorization: Bearer ' ``` #### GET /api/v1/seo/authority — Domain authority (Ahrefs Domain Rating) The domain's Ahrefs Domain Rating (0–100). The response includes Ahrefs' license link, which must be preserved when the metric is displayed. Credits: 1. Parameters: - `domain` (required) — The domain to rate. Accepts a bare domain or a full URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/seo/authority?domain=nike.com' -H 'Authorization: Bearer ' ``` #### GET /api/v1/seo/organic — Organic search footprint The domain's organic-search footprint from a major SEO index (US database): number of organic keywords it ranks for, estimated monthly organic visits, traffic cost (what the same traffic would cost in Google Ads, USD/month), and organic rank. A domain the index has never seen fails with `empty_result` (refunded). Credits: 1. Parameters: - `domain` (required) — The domain to analyze. Accepts a bare domain or a full URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/seo/organic?domain=nike.com' -H 'Authorization: Bearer ' ``` #### GET /api/v1/seo/backlinks — Backlink counts Backlink counts from a major SEO index by scope: root domain (the headline number), hostname, and exact URL. A domain the index has never seen reports zeros. Credits: 1. Parameters: - `domain` (required) — The domain to analyze. Accepts a bare domain or a full URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/seo/backlinks?domain=nike.com' -H 'Authorization: Bearer ' ``` ### Domain #### GET /api/v1/domain/registration — Domain registration facts (RDAP) Registration facts via RDAP: registered / expiry / last-updated dates, computed domain age, registrar, nameservers, DNSSEC, and status codes. Domain age is a standard trust signal for a brand dossier. TLDs without an RDAP service (some ccTLDs) fail with `empty_result` (refunded). Cached for 7 days. Credits: 1. Parameters: - `domain` (required) — The domain to look up. Accepts a bare domain or a full URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/domain/registration?domain=nike.com' -H 'Authorization: Bearer ' ``` #### GET /api/v1/agent-ready — Agent-readiness probe Whether a site publishes the agent-facing files AI agents look for: `/llms.txt`, `/agents.md`, and the `/.well-known/ucp` Universal Commerce Protocol merchant profile (with its versions, capabilities, payment handlers, and MCP endpoint when advertised). Follows redirects; HTML soft-404 pages count as absent. Shopify stores publish all three by default. Credits: 1. Parameters: - `domain` (required) — The site to probe. Accepts a bare domain or a full URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/agent-ready?domain=stashtea.com' -H 'Authorization: Bearer ' ``` ### Chrome Web Store #### GET /api/v1/chrome-store/extension — Chrome Web Store extension listing Listing snapshot for a Chrome Web Store extension: name, rating and rating count, user count, category, current version, publish date, homepage, and icon. Identify the extension by its 32-character `id` or by its store `url`. Credits: 1. Parameters: - `id` — 32-character extension id. - `url` — Store URL (`chromewebstore.google.com/detail/…`), alternative to `id`. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/chrome-store/extension?id=hjbfbllnfhppnhjdhhbmjabikmkfekgf' -H 'Authorization: Bearer ' ``` #### GET /api/v1/chrome-store/reviews — Chrome Web Store extension reviews Reviews for a Chrome Web Store extension, newest first, 100 per page: author, rating, text, date, extension version, language, and the developer reply when present. Voice-of-customer and competitor research in one call. `has_more: true` means more pages exist than `max_pages` fetched. Credits: 1. Parameters: - `id` — 32-character extension id. - `url` — Store URL (`chromewebstore.google.com/detail/…`), alternative to `id`. - `max_pages` — Pages of 100 reviews to fetch (1–10, default 1). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/chrome-store/reviews?id=hjbfbllnfhppnhjdhhbmjabikmkfekgf' -H 'Authorization: Bearer ' ``` ### Ads #### GET /api/v1/ads/meta — Ads a Facebook page is running (Meta Ad Library) Active and recent ads for a Facebook page from the Meta Ad Library, including creative text, CTA, format, dates, and the raw ad nodes in `data`. Identify the page by numeric `page_id`, by page `slug`, or by page `url` — slugs/URLs are resolved to the page id internally at no extra credit cost. `total_count` reports the full Ad Library count even when fewer ads are fetched. Credits: 2. Parameters: - `page_id` — Numeric Facebook page id (fastest — skips resolution). - `slug` — Facebook page slug, e.g. `nike` for facebook.com/nike. - `url` — Full Facebook page URL. - `max_pages` — Pages of 30 ads to fetch (1–10, default 3). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/ads/meta?slug=nike&max_pages=3' -H 'Authorization: Bearer ' ``` #### GET /api/v1/ads/google — Ads a domain is running (Google Ads Transparency Center) Ad creatives a domain is running, scraped from Google's Ads Transparency Center (which has no official API). Returns creative ids with Transparency Center links in markdown and the raw creative items in `data`. 40 creatives per page. Credits: 2. Parameters: - `domain` (required) — The advertiser domain. Accepts a bare domain or a full URL. - `max_pages` — Pages of 40 creatives to fetch (1–20, default 3). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/ads/google?domain=nike.com' -H 'Authorization: Bearer ' ``` #### GET /api/v1/ads/google/creative — Google ad creative detail One creative from the Google Ads Transparency Center: format (text / image / video), every preview variant, first and last shown dates, the regions it ran in, and the advertiser block. Use the `AR…` / `CR…` ids from `/api/v1/ads/google` items. Google does not expose landing URLs for any format. Credits: 1. Parameters: - `advertiser_id` (required) — Advertiser id (`AR…`), key `1` of an `/ads/google` item. - `creative_id` (required) — Creative id (`CR…`), key `2` of an `/ads/google` item. - `include_advertiser` — Set `false` to skip the embedded advertiser lookup. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/ads/google/creative?advertiser_id=AR01371945870127267841&creative_id=CR07636846834660409345' -H 'Authorization: Bearer ' ``` #### GET /api/v1/ads/google/advertiser — Google Ads advertiser identity Who is behind an `AR…` advertiser id in the Google Ads Transparency Center: display name, legal name, country, and verification status. Credits: 1. Parameters: - `advertiser_id` (required) — Advertiser id (`AR…`). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/ads/google/advertiser?advertiser_id=AR01371945870127267841' -H 'Authorization: Bearer ' ``` ### Brand #### GET /api/v1/brand — One-call brand dossier Runs several domain-keyed extractions in parallel and aggregates them into one response. Default parts: logo, estimated traffic, domain authority, Google ads, and the homepage as clean markdown (max 6 credits). Opt-in parts, requested explicitly via `parts=`: `organic` (organic keywords/traffic/rank), `backlinks` (backlink counts), `registration` (RDAP domain facts), `agent_ready` (llms.txt / agents.md / UCP probe) — 1 credit each. Each part succeeds or fails independently (`parts..ok`), so one blocked source never sinks the report. Credits = sum of successful, non-cached parts (cached parts are free). `markdown` is the combined report. Credits: 6. Parameters: - `domain` (required) — The brand domain to profile. Accepts a bare domain or a full URL. - `parts` — Comma-separated parts. Default: logo, traffic, authority, ads, homepage. Opt-in extras: organic, backlinks, registration, agent_ready. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/brand?domain=nike.com' -H 'Authorization: Bearer ' ``` #### GET /api/v1/brand/logo — Brand logo / favicon The domain's logo/favicon: a stable, hotlinkable image URL plus metadata (source, size, content type). The markdown output embeds the image. Logos are cached for 7 days — repeat lookups are free. Credits: 1. Parameters: - `domain` (required) — The domain to fetch the logo for. Accepts a bare domain or a full URL. - `size` — Requested image size in pixels (16–256, default 128). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/brand/logo?domain=nike.com' -H 'Authorization: Bearer ' ``` ### Social #### GET /api/v1/social/facebook/page — Facebook page details Page name, category, followers/likes, website, phone/email/address when public, page id (usable with `/api/v1/ads/meta`), and whether the page is currently running ads. Credits: 1. Parameters: - `slug` — Facebook page slug, e.g. `nike` for facebook.com/nike. - `url` — Full Facebook page URL (alternative to `slug`). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/facebook/page?slug=nike' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/instagram/profile — Instagram profile Profile name, bio, follower/following counts, verification, category, website and bio links, read from the public profile with no login. Typical latency 3–8 s; retry on `provider_error` (credits are refunded automatically). Credits: 1. Parameters: - `username` (required) — Instagram handle, with or without the leading `@`. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/instagram/profile?username=nike' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/tiktok/profile — TikTok profile Profile display name, bio, verification, follower/following/like/video counts, and bio link. Credits: 1. Parameters: - `username` (required) — TikTok handle, with or without the leading `@`. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/tiktok/profile?username=gymshark' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/linkedin/company — LinkedIn company page Company name, tagline, description, website, industry, size bucket, headquarters, founding year, type, specialties, follower and employees-on-LinkedIn counts, address, logo, affiliated and similar pages, and up to 10 recent posts. Read from the public page LinkedIn shows search engines — no login or LinkedIn API access. Credits: 1. Parameters: - `slug` — The `/company/` part of the page URL, e.g. `stripe`. Provide this or `url`. Numeric company ids are not supported. - `url` — A `linkedin.com/company/` URL, in any locale subdomain. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/linkedin/company?slug=stripe' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/linkedin/person — LinkedIn public profile Public member profile: name, headline, location, about, follower and connection counts, badges, avatar, websites, current companies, experience, and education. `public_view` is `full` for members LinkedIn features (Creator / Top Voice) and `limited` for everyone else — then the headline and most experience/education entries are withheld. Profiles LinkedIn hides from logged-out visitors (nonexistent or public visibility off) return `not_found`. Credits: 1. Parameters: - `handle` — The `/in/` part of the profile URL, e.g. `williamhgates`. Provide this or `url`. - `url` — A `linkedin.com/in/` profile URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/linkedin/person?handle=williamhgates' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/youtube/video — YouTube video details Title, description, tags, category, duration, publish date, **exact** view and like counts, thumbnail, and the channel (name, handle, subscriber count). Pass a video id or any watch / youtu.be / Shorts URL. Credits: 1. Parameters: - `video_id` — 11-character YouTube video id. Provide this or `url`. - `url` — A `youtube.com/watch?v=…`, `youtu.be/…`, or `youtube.com/shorts/…` URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/youtube/video?video_id=dQw4w9WgXcQ' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/youtube/channel — YouTube channel profile Channel title, description, country, join date, subscriber count, lifetime view count, video count, external links (socials, website), avatar, and banner. Credits: 1. Parameters: - `handle` — Channel handle, with or without the leading `@`. Provide this, `channel_id`, or `url`. - `channel_id` — Channel id (`UC…`, 24 characters). - `url` — A `youtube.com/@handle`, `/channel/UC…`, `/c/…`, or `/user/…` URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/youtube/channel?handle=mkbhd' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/youtube/transcript — YouTube video transcript Full transcript of a video as clean prose paragraphs **and** timestamped blocks (markdown), with timed snippets and the joined text in `data`, plus the video metadata (title, channel, views, likes, publish date). Works for manual and auto-generated captions; falls back to English, then the first available track, and machine-translates into `lang` when no native track exists. Credits: 1. Parameters: - `video_id` — 11-character YouTube video id. Provide this or `url`. - `url` — A `youtube.com/watch?v=…`, `youtu.be/…`, or `youtube.com/shorts/…` URL. - `lang` — Preferred caption language code (`en`, `ja`, `pt-BR`, …). Default: English, then the first available track. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/youtube/transcript?url=https://youtu.be/dQw4w9WgXcQ' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/youtube/channel/videos — YouTube channel videos A channel's uploads, 30 per page — title, view count, published-ago, and length — sorted latest, most popular, or oldest (YouTube's own tab sorts). See what a competitor posts and what performs; pair with `/social/youtube/video` for exact figures on any id. Credits: 1. Parameters: - `handle` — Channel handle, with or without the leading `@`. Provide this, `channel_id`, or `url`. - `channel_id` — Channel id (`UC…`, 24 characters). - `url` — A `youtube.com/@handle` (or `/channel/UC…`) URL. - `sort` — `latest` (default), `popular`, or `oldest`. - `max_pages` — Pages of 30 videos to fetch (1–5). Default 1. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/youtube/channel/videos?handle=mkbhd&sort=popular' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/youtube/search — YouTube search YouTube search results — videos, Shorts, channels, and playlists in page order, with estimated total, per-result views / age / length / channel, and YouTube's own filters (type, upload date, duration, relevance vs popularity). Content and keyword research for video. Accepts the same query you would type into YouTube. Credits: 1. Parameters: - `q` (required) — Search query. - `sort` — `relevance` (default) or `popularity`. YouTube web search no longer offers upload-date or rating sorts — use `upload_date` to restrict recency. - `type` — Restrict to `video`, `shorts`, `channel`, `playlist`, or `movie`. - `upload_date` — Uploaded within: `today`, `week`, `month`, or `year`. - `duration` — `short` (< 3 min), `medium` (3–20 min), or `long` (> 20 min). - `max_pages` — Result pages to fetch (1–5), ~20 items each. Default 1. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/youtube/search?q=best+running+shoes+2026&upload_date=month' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/x/profile — X (Twitter) profile X (formerly Twitter) profile: followers, following, post and media counts, bio, website, location, join date, verification type, and the "based in" country from X's About-this-account panel. No X API key or login. "X" and "Twitter" refer to the same platform throughout. Credits: 1. Parameters: - `handle` — X handle, with or without the leading `@`. Provide this or `url`. - `url` — An `x.com/` or `twitter.com/` profile URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/x/profile?handle=huel' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/x/tweets — X (Twitter) posts timeline One page (about 20) of an X (Twitter) account's posts, newest first, pinned post first — text, media, likes, reposts, replies, quotes, bookmarks, views, quoted posts, and reposts. Pass the returned `next_cursor` as `cursor` for the next page (each page is one credit). Credits: 1. Parameters: - `handle` — X handle, with or without the leading `@`. Provide this or `url`. - `url` — An `x.com/` profile URL. - `cursor` — `next_cursor` from the previous page. - `with_replies` — `true` to include the account's replies (the "Posts & replies" tab). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/x/tweets?handle=huel' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/x/tweet — X (Twitter) single post + thread One X (Twitter) post with full engagement counts, media, the quoted post if any, and the author's own unrolled thread (1/n, 2/n, …). X Articles (long-form posts) come with their full body as `article` (markdown, text, images) and are rendered in the markdown output. Accepts a post id or an `x.com//status/` URL. Credits: 1. Parameters: - `id` — Numeric post id. Provide this or `url`. - `url` — An `x.com//status/` URL. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/x/tweet?url=https://x.com/huel/status/2080284119838204327' -H 'Authorization: Bearer ' ``` #### GET /api/v1/social/x/replies — X (Twitter) replies to a post Replies from other users to an X (Twitter) post, as structured posts with author, engagement, and `replying_to` so you can rebuild the tree. The original post is included on the first page. Paginate with `next_cursor`. X hides some low-quality replies from logged-out viewers, so fewer replies than the post's reply count is normal. Credits: 1. Parameters: - `id` — Numeric post id. Provide this or `url`. - `url` — An `x.com//status/` URL. - `cursor` — `next_cursor` from the previous page. - `ranking` — `relevance` (X's default ordering), `recency`, or `likes`. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/social/x/replies?id=2080284119838204327&ranking=recency' -H 'Authorization: Bearer ' ``` ### TikTok Shop #### GET /api/v1/tiktok/shop/product — TikTok Shop product detail Product detail with **exact** figures: units sold, per-SKU stock and pricing, rating with the full 1–5 star histogram, categories, images, and a small review sample. Snapshot stock over time and diff it to estimate sales velocity. Region (prices/currency) follows the scraper exit IP. Credits: 1. Parameters: - `product_id` (required) — Numeric product id (~19 digits), from a `/view/product/` or `/shop/pdp//` URL, or from the store/category endpoints. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/tiktok/shop/product?product_id=1729587769570529799' -H 'Authorization: Bearer ' ``` #### GET /api/v1/tiktok/shop/product/videos — Videos promoting a TikTok Shop product TikTok videos promoting a product (up to ~10 — TikTok caps this feed): creator, play and like counts, upload date, and video links. Pair with the product endpoint (units sold) to attribute a product’s sales to the creators and videos driving them. Credits: 1. Parameters: - `product_id` (required) — Numeric product id (~19 digits). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/tiktok/shop/product/videos?product_id=1729587769570529799' -H 'Authorization: Bearer ' ``` #### GET /api/v1/tiktok/shop/seller — TikTok Shop seller summary Shop summary by seller id: name, region, rating, items sold, followers, product and review counts. A lightweight lookup — use `/api/v1/tiktok/shop/store` when you also want the product catalog. Get the `seller_id` from a product’s detail response. Credits: 1. Parameters: - `seller_id` (required) — Numeric seller/store id (~19 digits). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/tiktok/shop/seller?seller_id=7495794203056835079' -H 'Authorization: Bearer ' ``` #### GET /api/v1/tiktok/shop/store — TikTok Shop store catalog Store info plus the product catalog (~30 products per page, with sold counts, ratings, and prices). The bulk-discovery path: walk a seller’s full catalog here, then feed `product_id`s into `/api/v1/tiktok/shop/product` for per-product detail. `has_more: true` means more pages exist than `max_pages` fetched. Credits: 2. Parameters: - `seller_id` (required) — Numeric seller/store id (~19 digits). - `max_pages` — Catalog pages of ~30 products to fetch (1–10, default 1). - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/tiktok/shop/store?seller_id=7495794203056835079&max_pages=5' -H 'Authorization: Bearer ' ``` #### GET /api/v1/tiktok/shop/category — TikTok Shop category listing Category info, child categories (walk `children[]` recursively to enumerate the tree), and the top ~15 products. Deeper category pagination is not exposed upstream — use the store endpoint for exhaustive per-seller catalogs; use categories for top-of-category sampling and tree discovery. Credits: 1. Parameters: - `category_id` (required) — Numeric category id, e.g. `700645` (Health). Get child ids from this endpoint’s `children[]` or a product’s `categories[]`. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/tiktok/shop/category?category_id=700645' -H 'Authorization: Bearer ' ``` ### Shopify #### GET /api/v1/shopify/categories — Shopify store catalogue categories What a Shopify store sells, in Shopify's standard product taxonomy: the primary category, vertical, and top categories with product counts, from the store's public storefront catalogue (250 products per page). Works for any store on Shopify — no store credentials. Headless or non-Shopify hosts return an empty result. Credits: 1. Parameters: - `domain` (required) — Store domain, e.g. `allbirds.com`. Accepts a bare domain or a full URL. - `max_pages` — Pages of 250 products to scan (1–8). Default 4. - `per_product` — `true` to also return every product with its category. - `format` — "json" (structured data only), "markdown" (markdown only), or "both" (default). Example: ```bash curl 'https://scrapewhale.dev/api/v1/shopify/categories?domain=allbirds.com' -H 'Authorization: Bearer ' ``` ### Conversions #### GET /api/v1/conversions/{id} — Fetch a past conversion or extraction Returns a conversion owned by the API key’s user, including its stored markdown and — for extractions — the structured `data` payload. Free (no credits charged). Credits: free. Example: ```bash curl 'https://scrapewhale.dev/api/v1/conversions/' -H 'Authorization: Bearer ' ``` ## Notes for agents - Prefer `format=markdown` when feeding results straight into a context window; prefer `format=json` when post-processing. - Responses include an `id` — re-fetch any past result free of charge via `GET /api/v1/conversions/{id}`. - 422 responses mean the extraction failed and credits were refunded; the `code` field explains why (e.g. `provider_error`, `empty_result`, `rate_limit`). Retrying `provider_error` after a short pause often succeeds.