Troubleshooting blocked and incomplete scrapes

The same symptom can have several causes. scrape-diagnose uses bounded evidence before recommending a paid request.

Run a free diagnosis

No account or API key is needed for a direct check. This pinned command tests a harmless public example:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- https://example.com/ --expect-text "Example Domain"

Read the diagnosis decision guide · See all CLI options

Scraper gets 403 but the page works in a browser

Run the direct diagnostic without a provider key:

scrape-diagnose "$TARGET_URL" --expect-text "Required marker"

The first request identifies itself as scrape-diagnose. If it returns 403, the tool makes one free comparison with ordinary browser-like User-Agent, Accept, and Accept-Language headers.

• If the comparison passes, use equivalent headers in your own client. The report returns HEADER001 and recommends no proxy.
• If both profiles return 403, an IP/reputation or managed anti-bot boundary is plausible. A budgeted provider test may be useful.
• If either response is an authentication wall, paywall, or interactive CAPTCHA, stop. Those flows are deliberately unsupported.

Do not rotate headers or proxies at scale against a site that has denied access. Confirm that your use is authorized and respect the site’s terms and rate limits.

Scraper returns 200 but HTML is empty

An HTTP 200 is not enough. Add a marker that proves the needed content exists:

scrape-diagnose "$TARGET_URL" --expect-text "Product title"

CONTENT003 means the response has too little visible HTML and may be a JavaScript shell. CAPTCHA001 means the apparent success looks like a challenge page. MARKER001 alone on otherwise complete static HTML does not trigger a provider recommendation; first verify the URL, selector/marker, locale, and whether the content is actually public.

If shell evidence remains and you already have a provider key, an explicitly budgeted live run can test rendering:

scrape-diagnose "$TARGET_URL" --expect-text "Product title" --live --max-total-credits 25

ScraperAPI render vs premium

These settings address different failure evidence:

• render=true targets client-side JavaScript shells and content absent from raw HTML.
• premium=true targets blocks more consistent with IP reputation or anti-bot routing.
• Combining them is more expensive and is tested only if earlier modes fail and the exact preflight fits the remaining total budget.

Inspect cost without sending a paid target request:

scrape-diagnose cost "$TARGET_URL" --render --json
scrape-diagnose cost "$TARGET_URL" --premium --json

Force a single configuration instead of the automatic ladder:

scrape-diagnose probe "$TARGET_URL" --render --max-total-credits 10

The tool reads SCRAPERAPI_KEY only from the process environment. It rejects command-line keys, never loads .env, and generates snippets that keep both key and target in environment variables.

Python Requests reproduction

If the failing client uses Python, start with the bounded Requests example and finding-code guide. It separates transport failures from JavaScript shells and parser mistakes without logging the response body, URL path, query string, or expected marker.