Scraping is an engineering system, not a script

Separate acquisition, parsing, validation, normalization, and persistence. When these stages are independent, a parser can be tested against stored fixtures and failed records can be replayed without downloading the source again.

This separation also makes browser automation a transport choice rather than the architecture. Start with an HTTP client and introduce a browser only where JavaScript execution or interaction is truly required.

• Acquire the raw response with source metadata.
• Parse into an intermediate representation.
• Validate required fields and invariants.
• Normalize into the destination schema.
• Persist with idempotent keys.

Selectors based on one fragile class name turn a small redesign into a silent data incident. Prefer semantic attributes, stable URL patterns, multiple extraction strategies, and validation that detects implausible results.

Store representative HTML fixtures and run parsers against them in CI. A fixture does not predict every redesign, but it prevents accidental regressions while extraction logic evolves.

item = parse_product(document)

if not item.title:
    raise DataQualityError("missing title")
if item.price is not None and item.price < 0:
    raise DataQualityError("invalid price")
if not item.source_url.startswith(ALLOWED_ORIGIN):
    raise DataQualityError("unexpected source")

Retries help with transient network failures, but repeating every failure creates load and hides permanent problems. Classify timeouts, rate limits, authentication failures, blocked sessions, parsing failures, and invalid data separately.

Use bounded exponential backoff with jitter for retryable failures. Persist the attempt count and final reason so operators can distinguish a temporarily unhealthy source from a parser that needs maintenance.

Scheduled collectors will encounter the same records repeatedly. Choose a stable source identifier or construct a deterministic content key, then use upserts or versioned records instead of blind inserts.

Idempotency makes replay safe. A failed batch can be processed again without duplicating rows or sending the same downstream notification twice.

Request success rate is not enough. Track items discovered, items parsed, required-field completeness, duplicate ratio, extraction latency, and changes in field distributions.

Alert on meaningful deviations rather than every individual error. A sudden drop from 500 records to 12 is often more important than a handful of timeouts, even if every HTTP response returned 200.