Practical case studies, code-heavy tutorials, and production-grade Apify actors for data extraction at scale. Author: Aleksei Spinov.https://blog.spinov.online/https://blog.spinov.online/blog/3-telegram-channels-production-data-engineering/https://blog.spinov.online/blog/3-telegram-channels-production-data-engineering/Three Telegram channels (@dataeng, @apache_airflow, @bigdata_en) I keep reading for distributed-systems patterns, orchestration depth, and downstream feedback that improves real production scrapers.Mon, 11 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/5-apify-run-log-patterns-faster-debugging/https://blog.spinov.online/blog/5-apify-run-log-patterns-faster-debugging/Five Apify-actor logging patterns I use across 78 production scrapers — tag-prefix, structured retries, soft-block detection, dedup checkpointing, summary-line — turning 30-min log greps into 30-sec lookups.Fri, 01 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/5-apify-scheduler-mistakes-burning-compute-units/https://blog.spinov.online/blog/5-apify-scheduler-mistakes-burning-compute-units/Five scheduler misconfigurations I've made or watched customers make on Apify, with exact cron/actor.json fixes and cost-of-mistake math from running 31 published actors.Fri, 01 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/5-production-scraping-failures-1k-runs/https://blog.spinov.online/blog/5-production-scraping-failures-1k-runs/Real failure modes from 2190 lifetime Apify runs across 32 actors — schema drift caught silently, retry self-DDoS, concurrency WAF traps, memory creep on long runs, silent webhook failures. With code.Tue, 12 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/a-budget-brake-stops-your-scraper-before-200/https://blog.spinov.online/blog/a-budget-brake-stops-your-scraper-before-200/Spend alerts fire after the money is gone. A budget brake refuses the next run before it spends. Here's a 40-line preventive fuse, run locally, with the real output, plus where it stops working.Wed, 03 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/ai-for-production-scrapers-rule-of-thumb/https://blog.spinov.online/blog/ai-for-production-scrapers-rule-of-thumb/After 1,819 production runs across 32 Apify actors — a practical map of where AI helps with scraper code, where it's neutral, and the 30% where it quietly breaks production.Mon, 11 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/apify-dataset-deduplication-patterns/https://blog.spinov.online/blog/apify-dataset-deduplication-patterns/Five production patterns to prevent silent dataset duplication on Apify — uniqueKey, content hashing, KV-store guards, and SQL-backed dedup. Real numbers from 968 Trustpilot runs.Sun, 17 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/apify-run-log-patterns-debugging/https://blog.spinov.online/blog/apify-run-log-patterns-debugging/Five log patterns I added to a production Apify scraper after 951 runs: fatal markers, pagination cursors, proxy audit, retry telemetry, run summary.Fri, 01 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/apify-scheduler-mistakes-cost-cu/https://blog.spinov.online/blog/apify-scheduler-mistakes-cost-cu/Five real Apify scheduler misconfigurations from 32-actor portfolio + cron / actor.json fixes + cost-of-mistake math, ordered by cost-impact.Fri, 15 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/apify-vs-self-hosted-decision/https://blog.spinov.online/blog/apify-vs-self-hosted-decision/A decision framework for when to use Apify Store vs build a self-hosted scraper, based on run data from 31 public actors including one at 949 production runs.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/apify-webhook-integration-patterns/https://blog.spinov.online/blog/apify-webhook-integration-patterns/Five production-tested Apify webhook patterns from 1818 lifetime runs across 79 actors: signed payloads, idempotency, dead-letter queues, retry budgets, and schema-drift detection.Sun, 03 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/conditional-get-incremental-scraping/https://blog.spinov.online/blog/conditional-get-incremental-scraping/Real numbers from 2,190 lifetime runs: 304 Not Modified saved 32-71% of bandwidth across Trustpilot, exchange-rate and npm-package actors. Code, failure modes, and when to skip it.Tue, 19 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/cost-per-result-apify-worksheet/https://blog.spinov.online/blog/cost-per-result-apify-worksheet/What does one record actually cost end-to-end? A simple 4-line worksheet that surfaces hidden costs across every Apify actor in your portfolio.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/dead-features-in-your-own-code/https://blog.spinov.online/blog/dead-features-in-your-own-code/An honest postmortem on finding two README-documented features that didn't exist in my own production scraper, with the audit script I now run on every actor.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/description-drift-serverless-catalogs/https://blog.spinov.online/blog/description-drift-serverless-catalogs/Why function-catalog descriptions go stale within months, and a 30-second monthly refresh playbook with Python code, drawn from a 32-actor Apify portfolio.Tue, 12 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/does-this-page-need-a-browser/https://blog.spinov.online/blog/does-this-page-need-a-browser/Half the 'you don't need a browser' takes on my feed are right and none of them tell you how to check. Here's a stdlib probe that reads the raw HTTP response and votes NO_BROWSER, JS_REQUIRED, or MAYBE. I ran it on 10 named public URLs; 6 returned their data without Chrome.Fri, 05 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/duckdb-dbt-zero-cost-analytics/https://blog.spinov.online/blog/duckdb-dbt-zero-cost-analytics/Why I run dbt-duckdb on a VM instead of paying 90/month for Snowflake, with the full repo layout and CI workflow.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/ethical-scraping-is-a-rate-limit-question/https://blog.spinov.online/blog/ethical-scraping-is-a-rate-limit-question/Ethics-of-scraping posts argue about robots.txt and ToS. After 2,190 production runs across 32 scrapers, the line between 'ethical' and 'banned' turned out to be the same line — and it's drawn by conditional GET and a sane rate limit, not by a checkbox. Here's the working pattern.Mon, 25 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/fingerprint-spoofing-is-a-losing-arcade/https://blog.spinov.online/blog/fingerprint-spoofing-is-a-losing-arcade/Spoofing JA3, TLS and header order is a race you lose by design. After 2,190 production scraper runs, the thing that survives is how the run behaves — not how human its fingerprint looks.Tue, 02 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/five-input-schema-mistakes-and-fixes/https://blog.spinov.online/blog/five-input-schema-mistakes-and-fixes/Five real input-schema mistakes I shipped across 78 Apify actors, what each cost in support emails and re-runs, and the exact schema patterns I use now.Fri, 01 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/framework-isnt-what-breaks-your-scraper/https://blog.spinov.online/blog/framework-isnt-what-breaks-your-scraper/After 2,190 production scraper runs, the framework almost never decided whether a job lived or died. Three disciplines did: element-targeted waiting (not networkidle), browser memory recycling, and bounded retry with backoff and jitter — with reproducible stdlib code.Thu, 28 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/free-llm-apis-2026-no-credit-card/https://blog.spinov.online/blog/free-llm-apis-2026-no-credit-card/Nine LLM APIs with a genuinely free tier and no credit card in 2026 — limits, OpenAI-compatibility, and which ones survive an extraction workload. Verified May 2026.Sun, 31 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/http-200-is-a-lie-schema-canary/https://blog.spinov.online/blog/http-200-is-a-lie-schema-canary/Your scraper returns 200, the parser doesn't crash, and your corpus quietly rots. Across 962 runs on one source, the failure that bit me wasn't a block — it was the source reshaping its output. Here's a stdlib schema canary that asserts the shape of the data, not just the response.Sat, 30 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/idempotent-webhooks-in-50-lines/https://blog.spinov.online/blog/idempotent-webhooks-in-50-lines/Stop losing duplicate Stripe/GitHub/Slack webhooks. A 50-line Python + Postgres pattern that survives retries and crashed workers — code, schema, and a 5-minute reproducible test.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/memory-leaks-long-running-scrapers/https://blog.spinov.online/blog/memory-leaks-long-running-scrapers/Production scraping memory leaks: BeautifulSoup retention, growing URL queues, and connection-pool exhaustion. Real fixes with measured before/after from 968+ live Trustpilot scraper runs.Mon, 18 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/minio-backup-automation/https://blog.spinov.online/blog/minio-backup-automation/Replace paid S3 with self-hosted MinIO. Step-by-step backup automation in Python with 50 lines of code.Wed, 29 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/operational-rules-after-100-runs/https://blog.spinov.online/blog/operational-rules-after-100-runs/Schema-drift detection, IP-budget enforcement, and golden-diff snapshots — three rules I added to a production Apify actor after 100+ runs revealed where silent failures hide.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/proxy-health-monitor-247-scraper-uptime/https://blog.spinov.online/blog/proxy-health-monitor-247-scraper-uptime/A production-ready Python proxy health monitor: detects failures, rotates dead proxies, and alerts you before scrapers go down. Built from a 100K-page incident postmortem.Wed, 13 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/raw-html-is-a-token-tax-i-measured-it/https://blog.spinov.online/blog/raw-html-is-a-token-tax-i-measured-it/Everyone says 'markdown beats HTML for tokens.' Nobody shows a number. I tokenized raw HTML vs extracted text across 10 public pages with tiktoken. Here's the real multiplier, the 30-line meter, and what it costs when it repeats on a schedule.Fri, 29 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/retry-logic-broken-30-line-fix/https://blog.spinov.online/blog/retry-logic-broken-30-line-fix/Most Python services retry network calls wrong — fixed delay, wrong errors, no upper bound. Here is the 30-line jitter+deadline pattern I use in production after a 4-minute incident.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/schema-drift-3-contract-tests/https://blog.spinov.online/blog/schema-drift-3-contract-tests/When a vendor silently flips a JSON field type or drops a value, your scraper keeps running and your data lies. Three small contract tests that catch schema drift before it lands in your warehouse.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/scraping-text-is-the-easy-10-percent-dedup-and-decay/https://blog.spinov.online/blog/scraping-text-is-the-easy-10-percent-dedup-and-decay/Getting the text out is the easy 10%. After 962 production scraper runs, the hard 90% is deduplication, re-collection, and decay — with a reproducible stdlib-only corpus deduper.Wed, 27 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/token-bucket-vs-exponential-backoff/https://blog.spinov.online/blog/token-bucket-vs-exponential-backoff/After 966 production runs of the Trustpilot scraper, I replaced exponential backoff with a token bucket in five actors. Code, numbers, and the failure modes that disappeared.Fri, 15 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/token-economics-agent-driven-scraping/https://blog.spinov.online/blog/token-economics-agent-driven-scraping/Six months of production scrapers (970r on a single actor) showed LLM agent loops cost 30-80× more than deterministic crawlers above ~50 pages. Real token math, two narrow agent-win cases, and the fallback-only pattern.Mon, 18 May 2026 00:00:00 GMThttps://blog.spinov.online/blog/traefik-docker-reverse-proxy/https://blog.spinov.online/blog/traefik-docker-reverse-proxy/Traefik watches the Docker socket, auto-discovers new containers, and routes traffic to them based on labels. No config files to edit. No reloads. Just docker compose up and go.Wed, 29 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/trustpilot-scraper-949-runs-postmortem/https://blog.spinov.online/blog/trustpilot-scraper-949-runs-postmortem/A post-mortem of three production failures from the Trustpilot scraper that crossed 949 runs on Apify Store: silent selector death, IP-block math, and the residential-proxy temptation.Wed, 29 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/trustpilot-scraper-production/https://blog.spinov.online/blog/trustpilot-scraper-production/Real numbers from a public Apify actor: which anti-bot tricks actually mattered, what the error budget looked like, and what I would do differently.Sat, 25 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/welcome-and-roadmap/https://blog.spinov.online/blog/welcome-and-roadmap/A code-first blog about web scraping, data extraction, and AI research, written by someone shipping production scrapers on Apify.Mon, 27 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/when-not-to-scrape/https://blog.spinov.online/blog/when-not-to-scrape/After shipping 79 Apify actors, here's the 60-second decision rule I now apply before writing any selector code.Thu, 30 Apr 2026 00:00:00 GMThttps://blog.spinov.online/blog/you-pay-for-the-bandwidth-that-returns-nothing/https://blog.spinov.online/blog/you-pay-for-the-bandwidth-that-returns-nothing/A per-GB proxy bill charges you for failed requests and retries too. On one config 53% of the bytes returned zero rows. Here's a model you can run with your own numbers.Thu, 04 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/your-scraper-collected-50-rows-there-were-4000/https://blog.spinov.online/blog/your-scraper-collected-50-rows-there-were-4000/A scraper can finish green, return only valid rows, and still hand you a quarter of the dataset. Pagination cutoffs are silent. Here is a 40-line completeness probe that catches them.Sun, 07 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/your-scraper-died-at-row-12000/https://blog.spinov.online/blog/your-scraper-died-at-row-12000/A long scrape that dies three hours in didn't lose one request — it lost the whole run, and rerunning from zero means paying twice for data you already had. Here's the ~40-line stdlib pattern that resumes a crashed job, fetches only the missing delta, and writes zero duplicates. Real captured output of a crash and a clean resume.Sat, 06 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/your-scraper-got-clean-data-the-site-lied/https://blog.spinov.online/blog/your-scraper-got-clean-data-the-site-lied/A site can detect your scraper and serve a 200 with a perfect schema and plausible values that are deliberately false. Status codes and sanity checks are blind to it by design. Here's a 30-line probe that grounds each row to an independent invariant — and why naive cross-source consensus gets fooled too.Tue, 09 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/your-scraper-passes-every-run-its-still-rotting/https://blog.spinov.online/blog/your-scraper-passes-every-run-its-still-rotting/Your scraper exits 0 on every run. Schema valid, row count plausible. And the yield has been sliding for weeks. A 20-line lagged-baseline probe over your own run log catches the drift before it becomes a breakage.Mon, 08 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/your-scraper-re-downloads-everything-most-didnt-change/https://blog.spinov.online/blog/your-scraper-re-downloads-everything-most-didnt-change/A scheduled scraper re-downloads its whole corpus every run, even though almost nothing changed since last time. The fix isn't faster fetching — it's deciding FETCH/SKIP/CONDITIONAL from a manifest before the first request. A 30-line planner, its real output, and the production trap (weak rotating ETags) that fakes the savings.Wed, 10 Jun 2026 00:00:00 GMThttps://blog.spinov.online/blog/your-scraper-returned-a-clean-row-it-was-wrong/https://blog.spinov.online/blog/your-scraper-returned-a-clean-row-it-was-wrong/HTTP was 200, the selectors held, the JSON parsed — and the LLM still returned a plausible, syntactically valid, semantically false value. A 60-line field-level sanity check catches the lie that schema validation can't see.Mon, 01 Jun 2026 00:00:00 GMT