John Sabo
← All capabilities

Data engineering at scale

Collecting and cleaning data at sizes most teams never touch, then keeping it correct on a schedule nobody has to babysit.

What it does for a business

Data problems stop being interesting right around the point they start being expensive. A pipeline that works at ten thousand records and falls over at ten million is not a smaller version of one that scales. The difference shows up as a query that used to be instant.

The work is usually three things: make the query fast at the size you actually have, make the daily job survive its own overlaps and failures, remove the human who currently notices when either goes wrong.

None of it requires exotic technology. It requires someone who has run it in production and knows which of the obvious approaches will not hold.

Shipped proof

2.76M
rows searched sub-second
9.25M
price rows backfilled
721K
records validated
22
stage daily pipeline

Stack

PostgreSQL (materialized views, GIN trigram, partitioning) multi-source ETL scraping at scale enrichment deduplication record linkage Python asyncio Redis Docker systemd

What this solves

Datasets that have outgrown what a team can hold by hand. The pipelines move millions of records a day without anyone watching them. They report their own failures.

Working on something like this?

One sentence about the problem is enough.

john@johnsabo.org