Best practices
The API is simple, but running it in production is where the small details matter. These are the habits that keep a scraping client reliable.
Set a generous timeout
ScraperAPI may retry a fetch internally before returning a response. Cutting the request off too early throws away that work. A client-side timeout of 60–70 seconds is a sensible default.
Retry on transient failures
Treat 500 responses and network errors as retryable. Two or three retries with a short backoff (1s, 3s, 7s) cover almost all transient issues. Do not retry 401 — your API key is wrong, retrying will not fix it.
Match concurrency to your plan
Each ScraperAPI plan comes with a concurrent-request limit. Send too many parallel requests and you'll get 429 responses. Use a bounded worker pool (a semaphore, a queue, or a library like p-limit) that sits just under your limit.
Check status codes, not just the body
Always inspect the HTTP status before parsing the body. A truncated HTML page or a JSON error object is easy to mistake for success if you skip the status check.