Your scraper gets 403, but the page works in a browser

A browser loading a page does not prove that a simple HTTP client will receive the same response. Sites can vary responses by request headers, IP reputation, JavaScript execution, cookies, rate, or an authentication boundary. The reverse is also true: a 200 OK does not prove that the requested content arrived. It may be an empty application shell or a challenge page.

scrape-diagnose is a small, open-source CLI and GitHub Action that separates those cases. It tests one public URL directly, checks the response content rather than trusting the status alone, and reports the cheapest technically plausible next step. Direct success explicitly means “do not buy a proxy.”

Run a direct diagnosis

Node.js 20 or newer is required. No account or API key is needed:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- \
  https://example.com/ \
  --expect-text "Example Domain"

The immutable commit in that command is the source audited for the first release. Add --json for a deterministic machine-readable report.

The diagnosis makes one read-only direct request. Only a direct 403 adds a second, free comparison using ordinary browser-like request headers. It follows at most five redirects locally, validates every destination before requesting it, and reads at most 1 MiB of response bytes.

Read the result as a decision tree

Observed result What it usually means Next action
200, expected text present, real visible content The direct fetch works Keep the direct request; no proxy is indicated
First request 403, browser-header comparison passes The request profile was the difference Copy the working header approach; no proxy is indicated
Both request profiles return 403 IP reputation or a stronger anti-bot control may be involved Test one bounded premium-proxy request if authorized
200 with an empty root element and scripts The server returned an application shell Test rendering if the content is public and JavaScript-produced
200 challenge or CAPTCHA shell The status is a false success Stop and review the site’s access rules; do not automate interaction
429 The target is rate-limiting the client Reduce request rate and respect Retry-After; a proxy is not a license to evade limits
401 or 402 Authentication or payment is required Stop; authenticated targets and paywalls are outside the tool’s scope
Expected marker missing from otherwise complete HTML The page is not the document you expected Check selectors, locale, redirects, and source assumptions before buying anything

Why “200 but empty HTML” is a real failure

Many modern pages return a short document containing only a mount point such as <div id="root"></div> and one or more script tags. A browser later executes those scripts and fills the page. A basic scraper sees only the shell.

scrape-diagnose estimates visible text after removing markup and scripts. It also looks for common shell structure and, when supplied, verifies your expected marker. This catches three common false positives:

  1. the server returned an application shell instead of the data;
  2. a challenge page returned 200 OK;
  3. a different page returned valid HTML but omitted the content the job actually requires.

That distinction matters because each failure has a different remedy. Rendering can help with a genuine client-rendered public page, but it does not fix a login wall, an incorrect URL, or missing data that never appears in the document.

ScraperAPI render versus premium

If a direct diagnosis fails and you already have a ScraperAPI key, live mode can test a small fallback ladder:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- \
  https://example.com/ \
  --expect-text "Required marker" \
  --live \
  --max-total-credits 25 \
  --max-paid-probes 3

Every paid attempt is preceded by ScraperAPI’s live /account/urlcost estimate. The request is skipped when that estimate would exceed the remaining cumulative ceiling. Provider-side redirect following is disabled, and the ladder stops on the first passing response.

Use rendering for strong JavaScript-shell evidence. Use premium proxies for persistent 403 evidence after the free header comparison fails. Do not turn on both by reflex: the useful result is the minimum configuration that satisfies the content check.

Preflight estimates are a gate, not a billing guarantee. The report records the provider’s observed credit-cost header when available and warns when it differs from the estimate.

Security and privacy boundaries

The tool accepts one public HTTP or HTTPS URL. It rejects credentials in URLs and blocks private, loopback, link-local, metadata-service, documentation, transition, multicast, and other special-use address ranges. DNS is checked before every direct request and every redirect destination.

Reports contain the target scheme and hostname, but never the target path, query string, fragment, expected marker, response body, API key, raw error, or redirect destination. The key can only come from SCRAPERAPI_KEY or a GitHub Actions secret input; command-line credentials and automatic .env loading are deliberately rejected.

There is no login automation, POST request, form submission, interactive CAPTCHA solving, extraction pipeline, bulk URL input, telemetry, or response-body export. Only test pages you are authorized to access.

Use it in a repository check

The same source is a dependency-free Node 24 GitHub Action:

permissions: {}

steps:
  - uses: sanjayamaharjancodes/scrape-diagnose@abe305a38cee37a0287d5bb9d1097964759b8c72
    with:
      target-url: https://example.com/
      expect-text: Example Domain
      live: "false"

Use the Action only to test software associated with the repository containing the workflow, not as a generic monitor or crawling workload.

Continue