Python requests gets 403 or empty HTML while the browser works

When requests.get() fails but the same page opens in a browser, changing the User-Agent at random is not a diagnosis. The server may be reacting to the complete request profile, session state, redirect destination, client IP, TLS behavior, request rate, or JavaScript that only a browser executes. A 200 OK can also be the wrong page: an empty application shell, a challenge, or valid HTML that does not contain the data your job needs.

Start with one bounded-memory request and one content assertion. Record only fields that explain the failure; do not log the response body, secret query strings, cookies, or the marker itself.

Run a free diagnosis

No account or API key is needed for a direct check. This pinned command tests a harmless public example:

npx --yes github:sanjayamaharjancodes/scrape-diagnose#abe305a38cee37a0287d5bb9d1097964759b8c72 -- https://example.com/ --expect-text "Example Domain"

Read the diagnosis decision guide · See all CLI options

Reproduce the failure safely with Requests

The example below reads TARGET_URL and EXPECTED_TEXT from environment variables. Both have harmless defaults, so running the file unchanged checks example.com. It accepts only globally reachable addresses, disables environment-proxy inheritance, validates every redirect destination before following at most five hops, uses separate connect/read timeouts, streams at most 1 MiB into memory, and prints only the target scheme and hostname, a coarse content category, the byte count, redirect state, status, and whether the expected marker arrived.

"""Make one bounded, redacted Requests check against an authorized public page."""

import ipaddress
import json
import os
import socket
import sys
from urllib.parse import urljoin, urlsplit

import requests


MAX_RESPONSE_BYTES = 1024 * 1024
MAX_REDIRECTS = 5
REDIRECT_STATUSES = {301, 302, 303, 307, 308}
DEFAULT_TARGET = "https://example.com/"
DEFAULT_MARKER = "Example Domain"
DEFAULT_USER_AGENT = (
    "scrape-diagnose-python-example/0.1 "
    "(+https://github.com/sanjayamaharjancodes/scrape-diagnose)"
)
FORBIDDEN_HOSTNAMES = {
    "localhost",
    "localhost.localdomain",
    "metadata",
    "metadata.google.internal",
}
IPV4_SPECIAL_NETWORKS = tuple(
    ipaddress.ip_network(value)
    for value in (
        "0.0.0.0/8",
        "10.0.0.0/8",
        "100.64.0.0/10",
        "127.0.0.0/8",
        "169.254.0.0/16",
        "172.16.0.0/12",
        "192.0.0.0/24",
        "192.0.2.0/24",
        "192.31.196.0/24",
        "192.52.193.0/24",
        "192.88.99.0/24",
        "192.168.0.0/16",
        "192.175.48.0/24",
        "198.18.0.0/15",
        "198.51.100.0/24",
        "203.0.113.0/24",
        "224.0.0.0/4",
        "240.0.0.0/4",
    )
)
IPV6_SPECIAL_NETWORKS = tuple(
    ipaddress.ip_network(value)
    for value in (
        "::/96",
        "::ffff:0:0/96",
        "64:ff9b::/96",
        "64:ff9b:1::/48",
        "100::/64",
        "100:0:0:1::/64",
        "2001::/23",
        "2001:db8::/32",
        "2002::/16",
        "2620:4f:8000::/48",
        "3fff::/20",
        "5f00::/16",
        "fc00::/7",
        "fe80::/10",
        "fec0::/10",
        "ff00::/8",
    )
)


def parse_target(value):
    """Parse one credential-free HTTP(S) URL."""
    try:
        parsed = urlsplit(value)
        hostname = parsed.hostname
        parsed.port
    except ValueError as error:
        raise ValueError("invalid target") from error

    if (
        parsed.scheme not in {"http", "https"}
        or not hostname
        or parsed.username is not None
        or parsed.password is not None
    ):
        raise ValueError("invalid target")

    return parsed


def sanitized_target(value):
    """Return only non-secret URL fields or raise a generic configuration error."""
    parsed = parse_target(value)
    return {"scheme": parsed.scheme, "hostname": parsed.hostname.lower()}


def validate_public_target(value):
    """Reject a literal or current DNS answer that is not globally reachable."""
    parsed = parse_target(value)
    hostname = parsed.hostname
    if hostname.lower().rstrip(".") in FORBIDDEN_HOSTNAMES:
        raise ValueError("target is not public")
    port = parsed.port or (443 if parsed.scheme == "https" else 80)

    try:
        addresses = {ipaddress.ip_address(hostname)}
    except ValueError:
        answers = socket.getaddrinfo(hostname, port, type=socket.SOCK_STREAM)
        addresses = {
            ipaddress.ip_address(answer[4][0].split("%", 1)[0]) for answer in answers
        }

    if not addresses:
        raise ValueError("target is not public")

    for address in addresses:
        special_networks = (
            IPV4_SPECIAL_NETWORKS if address.version == 4 else IPV6_SPECIAL_NETWORKS
        )
        if (
            not address.is_global
            or address.is_multicast
            or any(address in network for network in special_networks)
        ):
            raise ValueError("target is not public")


def fetch_response(session, target_url, headers):
    """Fetch manually so every redirect destination is validated before use."""
    current_url = target_url
    for redirect_count in range(MAX_REDIRECTS + 1):
        validate_public_target(current_url)
        response = session.get(
            current_url,
            headers=headers,
            stream=True,
            allow_redirects=False,
            timeout=(5, 20),
        )

        location = response.headers.get("Location")
        if response.status_code not in REDIRECT_STATUSES or not location:
            return response, current_url, redirect_count

        if redirect_count == MAX_REDIRECTS:
            response.close()
            raise ValueError("too many redirects")

        next_url = urljoin(response.url, location)
        response.close()
        current_url = next_url

    raise ValueError("too many redirects")


def content_kind(value):
    """Reduce an untrusted Content-Type header to a small, printable category."""
    media_type = value.partition(";")[0].strip().lower()
    if media_type in {"text/html", "application/xhtml+xml"}:
        return "html"
    if media_type == "application/json" or media_type.endswith("+json"):
        return "json"
    if media_type.startswith("text/"):
        return "text"
    if media_type:
        return "other"
    return "missing"


def read_bounded(response):
    """Read at most one byte beyond the reporting ceiling to detect truncation."""
    collected = bytearray()
    ceiling = MAX_RESPONSE_BYTES + 1
    for chunk in response.iter_content(chunk_size=64 * 1024):
        if not chunk:
            continue
        remaining = ceiling - len(collected)
        collected.extend(chunk[:remaining])
        if len(collected) >= ceiling:
            break
    return bytes(collected[:MAX_RESPONSE_BYTES]), len(collected) > MAX_RESPONSE_BYTES


def main():
    target_url = os.environ.get("TARGET_URL", DEFAULT_TARGET)
    expected_text = os.environ.get("EXPECTED_TEXT", DEFAULT_MARKER)

    try:
        target = sanitized_target(target_url)
    except ValueError:
        print(json.dumps({"decision": "ERROR", "reason": "invalid_target"}))
        return 2

    headers = {
        "User-Agent": os.environ.get("REQUEST_USER_AGENT", DEFAULT_USER_AGENT),
        "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.1",
        "Accept-Language": os.environ.get(
            "REQUEST_ACCEPT_LANGUAGE", "en-US,en;q=0.5"
        ),
    }

    try:
        with requests.Session() as session:
            session.trust_env = False
            response, final_url, redirect_count = fetch_response(
                session, target_url, headers
            )
            with response:
                body, truncated = read_bounded(response)
                final_target = sanitized_target(final_url)
                kind = content_kind(response.headers.get("Content-Type", ""))
                encoding = response.encoding or "utf-8"
                try:
                    decoded = body.decode(encoding, errors="replace")
                except LookupError:
                    decoded = body.decode("utf-8", errors="replace")

                marker_matched = bool(expected_text) and expected_text in decoded
                report = {
                    "decision": "PASS",
                    "target": target,
                    "response": {
                        "status": response.status_code,
                        "content_kind": kind,
                        "bytes_read": len(body),
                        "truncated": truncated,
                        "redirected": redirect_count > 0,
                        "final_hostname": final_target["hostname"],
                        "marker_matched": marker_matched,
                    },
                }

                exit_code = 0
                if response.status_code in {401, 402}:
                    report.update(decision="STOP", reason="authentication_or_payment")
                    exit_code = 4
                elif response.status_code == 403:
                    report.update(decision="FAIL", reason="http_403")
                    exit_code = 5
                elif response.status_code == 429:
                    report.update(decision="STOP", reason="rate_limited")
                    exit_code = 6
                elif response.status_code >= 400:
                    report.update(decision="FAIL", reason="http_error")
                    exit_code = 7
                elif kind != "html":
                    report.update(decision="FAIL", reason="not_html")
                    exit_code = 8
                elif not marker_matched:
                    report.update(decision="FAIL", reason="marker_missing")
                    exit_code = 9

                print(json.dumps(report, sort_keys=True))
                return exit_code
    except (OSError, requests.RequestException, ValueError):
        print(
            json.dumps(
                {
                    "decision": "ERROR",
                    "reason": "request_failed",
                    "target": target,
                },
                sort_keys=True,
            )
        )
        return 3


if __name__ == "__main__":
    sys.exit(main())

Install Requests and run the default check:

python -m pip install requests
python docs/examples/python_requests_diagnostic.py

Set environment variables using the secure mechanism for your shell when checking your own authorized public URL. Avoid placing a URL with a private token directly in shell history. The example rejects credentials embedded in a URL and never prints its path or query, but the URL is still sent to its destination.

The default result should have "decision": "PASS", "status": 200, and "marker_matched": true. A nonzero exit distinguishes authentication/payment, 403, 429, other HTTP errors, non-HTML content, a missing marker, and a network/configuration error. Requests’ read timeout applies between socket reads; it is not a total wall-clock deadline. Put this diagnostic under a job-level deadline when that distinction matters.

Why a browser and Requests can receive different pages

Python Requests is an HTTP client, not a browser engine. Several differences can matter:

Headers: Accept, Accept-Language, and User-Agent can select different representations or trigger a server rule. One stable, truthful profile is easier to debug than rotating identities.
Session state: a browser may have consent, locale, or login cookies that a fresh requests.Session() does not. Do not copy private login cookies into an unattended scraper.
Redirects: the visible browser URL may lead to a different hostname, locale, or authentication boundary. Compare the final hostname without logging a sensitive destination path.
Network and TLS profile: a data-center IP or non-browser TLS handshake can be treated differently even when headers match.
JavaScript: Requests returns the server’s raw bytes. It does not execute scripts that later populate a <div id="root"></div> in a browser.
Rate: a browser visit is occasional; a job may send bursts and receive 429 Too Many Requests.

These causes require different remedies. Rendering does not repair an incorrect URL. A proxy does not supply login authorization. Browser-like headers do not execute JavaScript. Treating every failure as an IP block wastes credits and can conceal the real defect.

Decide from the evidence

Observation	Likely boundary	Safe next step
`403` from Requests, free header comparison passes	Request-profile difference	Use a consistent, authorized header profile and re-check the marker; no paid proxy is indicated
`403` from both direct profiles	Server policy, IP reputation, or stronger anti-bot routing	Confirm access is permitted; then make at most one budgeted provider test if justified
`200` with very little visible HTML and script tags	Client-rendered application shell	Test rendering on the one public page; headers alone will not execute JavaScript
`200`, complete HTML, expected marker missing	Wrong URL, locale, redirect, selector, or assumption	Inspect the source contract before buying a service
`200` challenge or CAPTCHA shell	False success	Stop and review the site’s rules; do not automate interaction
`401` or `402`	Authentication or payment boundary	Stop; use an official authenticated API or obtain access
`429`	Rate limit	Reduce frequency, cache, and respect `Retry-After`; do not rotate identities
Redirect to another hostname	Different origin or boundary	Verify that hostname is expected and authorized before continuing

The expected marker is important. Status and byte count alone cannot prove that your scraper received the document it was designed to parse.

Translate scrape-diagnose findings back to Python

The CLI’s finding codes make the next experiment explicit:

HEADER001: the free browser-header comparison succeeded. A paid proxy is not indicated. In your Python client, use a single explicit profile appropriate for the site, then confirm the expected content. Do not build a rotating User-Agent list.
CONTENT003: the response resembles a JavaScript shell. Requests cannot render it. Prefer an official data endpoint when one exists; otherwise test bounded rendering for an authorized public page.
MARKER001: the response is HTML but lacks your proof marker. Verify the exact URL, locale, redirects, content variant, and parser assumption. A marker-only failure on complete static HTML does not justify a commercial fallback.
HTTP403: the server refused the request. If the free header comparison also fails, confirm permission before evaluating a premium route.
HTTP429: the client is being rate-limited. Back off and respect the server’s signal.
CAPTCHA001: the response appears to be an interactive challenge. Stop; solving or bypassing it is outside this workflow.

If HEADER001 is the only finding, make the client profile explicit rather than relying on Requests defaults. Replace only the headers mapping in the complete bounded example above so its redirect validation and generic exception handling remain intact:

import os
headers = {
    "User-Agent": os.environ.get(
        "REQUEST_USER_AGENT",
        "my-public-fetcher/1.0 (+https://example.org/contact)",
    ),
    "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.1",
    "Accept-Language": os.environ.get("REQUEST_ACCEPT_LANGUAGE", "en-US,en;q=0.5"),
}

Use an identifier and contact URL that truthfully describe your client. If the target only accepts a value that falsely claims to be an interactive browser, pause and check whether the site offers an API or grants automation access.

Verify the remedy, not just the request

A fix is complete only when the content contract passes. Re-run the same expected-marker check after changing one variable. Keep the URL, rate, and marker stable while comparing:

the explicit direct Python profile;
the free direct/browser-header diagnosis;
if CONTENT003 is present, one rendering configuration;
if persistent HTTP403 is present and access is authorized, one premium configuration.

Stop on the first passing result. Do not combine rendering and premium routing by reflex. The CLI’s optional live mode asks the provider for a cost estimate before each paid probe, enforces a cumulative preflight ceiling, and stops on the first response containing the marker. Provider-reported cost can still differ from a preflight estimate, so inspect the final report.

Affiliate disclosure: If the bounded checks above justify a managed test and you do not already have an account, this ScraperAPI pricing link is an affiliate link. The maintainer may earn a commission if you sign up through it and later purchase service. The static sd-python-guide label identifies this guide placement; it contains no user or session identifier. Opening the link does not change a diagnostic result.

Empty response, empty DOM, or empty BeautifulSoup result?

These descriptions are often confused:

Empty HTTP response: the body has zero or very few bytes. Check status, content type, redirect destination, timeout behavior, and server logs.
Empty rendered DOM: the browser also has no required content after scripts finish. The source URL, application state, or upstream data may be wrong; adding rendering elsewhere will not create missing data.
Raw HTML shell: Requests receives a mount element and scripts, while the browser later shows content. This is the case where rendering may be technically relevant.
Empty BeautifulSoup selection: the HTML can be complete while select() returns no nodes. Verify the selector, namespace, locale, and marker against the exact raw document before changing transport.
Challenge page with 200: the body is not empty, but it belongs to the access-control system rather than the target page. Detect its content; do not trust the status.

Test transport first, then parsing. If the expected phrase is absent from response.text, changing a BeautifulSoup selector cannot recover it. If the phrase is present but the selector is empty, a proxy or browser is unlikely to help.

Safety boundaries

Use these checks only for public pages you are authorized to fetch. Keep the test to one URL and a bounded rate. Do not weaken the public-address or redirect checks, rotate identities, replay private browser cookies, automate CAPTCHA or login flows, bypass paywalls, or use proxies to evade 429 limits. Keep API keys and markers in environment variables; do not put them in source, command-line flags, reports, or issue screenshots.

For the provider-independent decision tree, see Troubleshooting blocked and incomplete scrapes. For report redaction and network boundaries, see Privacy and safety design.