Direct answer: scraping 100,000 pages a month costs roughly USD 49–499 at public list prices, but this is not a "who is cheaper" question — the four vendors bill in different units. Firecrawl bills credits (about 1 credit per page), ScrapingBee bills requests (5 credits with JS rendering, 10 or 25 with premium proxies), Bright Data bills successful records (USD 1.5 per 1,000), and Apify bills events or compute units (CU) plus residential-proxy gigabytes. The same 100,000-page job can differ 5–10× depending on whether you need JS rendering, structured JSON, and residential proxies. When you self-host, proxy bandwidth — not servers — is usually the biggest line.
This guide is for technology, data and procurement teams using public web data for market intelligence, competitor monitoring, RAG corpora or agent tooling. Data cut-off: 31 October 2026; prices come from four official pricing pages, in USD, excluding tax, with no FX conversion. The worked examples are editorial models, not quotes; compliance and licensing must be assessed per target site and applicable law.
Convert to cost per successful record before comparing; JS rendering, structured extraction and residential proxies are the three main surcharges; calculate proxy traffic before self-hosting — 150 GB × USD 8/GB is USD 1,200.
What does each vendor actually charge for?
Firecrawl bills credits (1 per page), ScrapingBee bills requests (1 credit, 5 with JS, 10/25 with proxies), Bright Data bills successful records (USD 1.5/1,000), and Apify bills events or CU plus proxy GB. Different units cannot be compared directly.
| Vendor | Official billing unit | Price anchor (official) | What it includes |
|---|---|---|---|
| Firecrawl | Credits; 1 credit per scrape page | Standard USD 99/mo for 100,000 credits | Scrape, crawl, search, PDF parsing, JSON extraction (extra credits) |
| ScrapingBee | Credits; from 1 credit per request | Freelance USD 49/mo for 250,000 credits | Headless browser, rotating proxies, screenshots, CSS/XPath extraction |
| Bright Data | Successful records | PAYG USD 1.5/1,000 records; Scale USD 499/mo incl. 384,000 | Proxies, unblocking, CAPTCHA, parsing; billed on successful delivery |
| Apify | Events (results) or CU; proxy per GB | CU USD 0.2 (Starter); residential proxy USD 8/GB | Platform, Actor runs, proxy and storage billed separately |
Sources: official Firecrawl, ScrapingBee, Bright Data Web Scraper API and Apify pricing pages (31 October 2026). Anchors are entry or common tiers; enterprise agreements and promotions are excluded.
What does 100,000 pages a month cost?
Plain scraping runs about USD 49–150; with JS rendering or structured extraction it rises to 99–399; successful-record tiers reach USD 499/month. The gap comes from credits consumed per page, not arbitrary pricing.
| Option | Cost per page | Monthly (example) | Note |
|---|---|---|---|
| ScrapingBee Freelance (plain requests) | 1 credit | USD 49 | 250,000 credits; 100k requests fit |
| Firecrawl Standard (plain scrape) | 1 credit | USD 99 | 100,000 credits exactly cover 100k pages |
| ScrapingBee Startup (JS rendering) | 5 credits | USD 99 | 500k credits within the 1,000,000 allowance |
| Bright Data PAYG | 1 record | USD 150 | USD 1.5/1,000 × 100 |
| Apify (event example) | 1 result | About USD 230 | Instagram Scraper at USD 2.30/1,000 as an example, plus plan usage |
| ScrapingBee Business (premium proxy ×25) | 25 credits | USD 249 | 2.5M credits within the 3,000,000 allowance |
| Firecrawl Growth (JSON extraction) | 5 credits | USD 399 | 500k credits; staying on Standard with PAYG would be about USD 1,099 |
| Bright Data Scale | 1 record | USD 499 | Includes 384,000 records, suited to steady volume |
Editorial conversion from official per-page/request/record credits and plan allowances; excludes tax, labour and retries. The Apify figure is an event-pricing example; actual cost also depends on the Actor, compute units and proxy traffic. Firecrawl JSON extraction adds 4 credits per page.
What is the cost per record?
At official rates: Firecrawl about USD 0.00099/page, ScrapingBee about USD 0.000196/credit (about 0.00098/page with JS), Bright Data USD 0.0015/record, Apify's example USD 0.0023/result. The spread mostly reflects whether proxies and parsing are included.
| Option | Effective unit price | Data shape it fits |
|---|---|---|
| ScrapingBee (Freelance) | About USD 0.000196/credit | Static pages with clear rules, parsed in-house |
| Firecrawl (Standard) | About USD 0.00099/page | Pages needing main-content extraction/Markdown |
| ScrapingBee (JS rendering) | About USD 0.00098/page | Client-rendered pages needing clicks/scrolls |
| Bright Data (PAYG) | USD 0.0015/record | Structured fields accepted per successful record |
| Apify (event example) | USD 0.0023/result | Ready-made Actors with results you can load directly |
Conversions use the lowest or common official tier; credits do not roll over (Firecrawl only on Scale and above for one month; ScrapingBee credits are monthly). Unused allowance does not make the unit price lower. Excludes tax and retries.
Why does the invoice double?
Five surcharges: JS rendering, structured JSON extraction, premium/residential proxies, failed pages still billed, and per-check monitoring. Miss any one and the budget is understated by half or more.
| Surcharge | Official rule | Budget impact |
|---|---|---|
| JS rendering | ScrapingBee: 5 credits per successful request | Unit price ×5 |
| Premium proxies | ScrapingBee: 10 or 25 credits depending on JS | Unit price ×10–×25 |
| Structured extraction | Firecrawl: JSON format +4 credits per page | One page becomes 5 credits |
| Failed pages | Firecrawl: no result is not charged, but 403/404 still return and cost 1 credit | Anti-bot sites cost more |
| Residential proxy traffic | Apify: USD 8/GB (Starter) | Heavy pages accumulate fast |
| Per-check monitoring | Firecrawl: 1 credit per page per check | Daily checks × 30 days |
| Success-only billing | Bright Data: charges only successful deliveries | No data fee on failures, but time is spent |
Official rules from each pricing page; definitions of "failure" differ, so confirm how 403/404, CAPTCHA, empty results and timeouts are billed before signing.
Is self-hosting always cheaper?
No. If 100,000 pages average 1.5 MB and all go through residential proxies, 150 GB × USD 8/GB = USD 1,200 — already above most managed options. You save the subscription but add proxies, anti-bot work and parsing maintenance.
| Cost line | How to meter | Example parameter (assumption) |
|---|---|---|
| Compute/browser | CU (1 CU = 1 GB RAM × 1 hour) or your own server | Measure with Apify's USD 0.2/CU basis |
| Proxy traffic | GB × price | 150 GB × USD 8 = USD 1,200 |
| Parsing and cleaning | Person-days/month | 2 person-days |
| Anti-bot and rule maintenance | Person-days/month | 2 person-days |
| Storage and egress | GB storage + egress | Per cloud bill |
| Compliance review | One-off + quarterly | Per legal effort |
Formula: self-hosted monthly cost = compute + (proxy GB × price) + (person-days × day rate) + storage/egress + compliance. The 1.5 MB/page, 150 GB and USD 8/GB figures are example assumptions to show magnitude, not a quote; replace person-days and day rate with your own.
What should procurement and acceptance check?
Cost per successful record, field completeness, duplication, failure-billing rules, whether proxies and JS are included, and compliance boundaries — not just the plan fee.
| Acceptance item | Suggested definition | Evidence to keep |
|---|---|---|
| Cost per record | Monthly fee ÷ successfully stored records | Invoice, ingestion stats |
| Field completeness | Share of target fields that are non-empty | Sampling report |
| Duplication | Duplicates after URL/ID dedupe | Dedupe log |
| Failure billing | Whether 403/404, CAPTCHA and empty results are charged | Contract terms, sample invoice |
| Proxy and rendering | Whether JS and residential proxies are in the unit price | Request parameters, plan details |
| Refresh cadence | Scrape cycle and data freshness | Scheduler log, timestamps |
| Compliance | robots, site terms, personal data, rate limits | Review records, authorisations |
| Exit and export | Export format, migration and deletion | Export sample, contract clause |
Editorial guidance, not legal advice; scraping public pages still requires respecting target-site terms, robots conventions and applicable law, with separate assessment for personal data.
Who should read this — and who should not?
For teams building competitor monitoring, price collection, RAG corpora or agent tooling. Not for projects that need to bypass logins/paywalls, or that lack a clear data purpose.
Read if: you are in technology, data or procurement; you are comparing Firecrawl, ScrapingBee, Bright Data and Apify; you need a budget or acceptance basis for 100,000 pages a month. Skip if: the target site explicitly forbids scraping; you need to bypass CAPTCHA/login; the purpose and authorisation are unclear. Supports: like-for-like comparison, plan selection, self-host comparison, failure-billing and compliance acceptance.
Lingyi does not rank scraping tools or promise coverage. To connect collection, cleaning and retrieval to an enterprise knowledge base or agent, see LingHarness; to fold data-collection cost into project budgets and compliance acceptance, talk to us.
Related reading: AI crawlers and robots policy, RAG knowledge-base cost, and knowledge-base content governance; this article only covers the collection layer's units, worked examples and self-host comparison.
FAQ
What does 100,000 pages a month cost?
At official public prices: plain scraping about USD 49–150 (ScrapingBee Freelance 49, Firecrawl Standard 99, Bright Data PAYG 150); with JS rendering or JSON extraction about 99–399; successful-record tiers reach USD 499/month. Actual cost depends on credits per page and whether residential proxies are needed.
Why do quotes differ by 5–10×?
Because the units differ (page, request, record, result) and so does what is included: JS rendering ×5, premium proxies ×10–×25, JSON extraction +4 credits per page. Converting to cost per successful record usually narrows the gap.
Per page or per result — which should we choose?
If you need the original text/Markdown for RAG, per page is intuitive. If you need structured fields and acceptance by delivered volume, successful records fit better (failures are not charged). Either way, write into the contract how failures, empty results and duplicates are billed.
Is a self-hosted crawler cheaper?
Not necessarily. At 1.5 MB per page and residential proxies, 100,000 pages is 150 GB × USD 8/GB = USD 1,200, already above most managed options; self-hosting also adds parsing rules, anti-bot maintenance and compliance review.
Are failed requests charged?
It varies. Firecrawl states that a scrape returning no result is not charged, but a 403/404 page is still returned and costs 1 credit; Bright Data bills only successful deliveries. Ask about CAPTCHA, timeouts and empty results before signing.
What should we watch when scraping public pages?
Respect site terms and robots conventions, control request rates, do not bypass logins or paywalls, and avoid collecting personal data unrelated to the purpose; assess personal data and cross-border transfer separately. This is not legal advice.
How should collection quality be accepted?
Check cost per successful record, field completeness, duplication, freshness, failure rate and failure billing; run the same 500–1,000-record sample across vendors before choosing a plan, rather than comparing monthly fees alone.
Method and limitations
This article uses only four official pricing pages accessible on 31 October 2026: Firecrawl, ScrapingBee, Bright Data Web Scraper API and Apify. All amounts are converted from official list prices and plan allowances; no enterprise agreements, promo codes or reseller quotes are used. USD is not converted to RMB and no vendor ranking is implied. Credit rules, allowances, failure billing and proxy prices can change; Apify's event example uses its published example (Instagram Scraper at USD 2.30/1,000), and actual cost also depends on the Actor and compute units. The 100,000 pages, 1.5 MB/page, 150 GB and USD 8/GB combination is an editorial assumption to show magnitude, not a quote. The article does not assess data quality or coverage and is not legal advice.
Sources and basis
Data cut-off: 31 October 2026. The following are official pricing pages; prices, allowances and terms follow the original text and invoice.
- Firecrawl, Pricing: firecrawl.dev/pricing. Plans and credits; 1 credit per scrape page, JSON +4, PDF 1 credit/page, search 2 credits/10 results, monitor 1 credit/page/check, interact 2 credits/browser minute; failure-billing notes; PAYG increments.
- ScrapingBee, Pricing: scrapingbee.com/pricing. Plans and credits; JS rendering 5 credits/request; premium proxies 10 or 25 credits; monthly credits do not roll over.
- Bright Data, Web Scraper API Pricing: brightdata.com/pricing/web-scraper. PAYG USD 1.5/1,000 records; Scale USD 499/mo incl. 384,000 records and USD 1.3/1,000 extra; success-only billing.
- Apify, Pricing: apify.com/pricing. Plans and prepaid usage; CU at USD 0.2/0.16/0.13; residential proxy USD 8/7.5/7 per GB; SERP USD 2.5/1,000; event example USD 2.30/1,000 results.