40+
data sources unified
24h→15m
segment refresh time
-50%
pipeline failures
2 weeks
to staff the data pod
The Client
A marketing analytics company serving mid-market and enterprise brands, whose product promises a unified view of each brand's customers across advertising, e-commerce, CRM and support channels. The engineering organization was strong on product and visualization but had only two data engineers maintaining the pipelines everything else depended on.
The Challenge
Behind the polished dashboards, the customer data platform had grown organically into dozens of one-off pipelines — a mix of cron scripts, vendor connectors and hand-written SQL, each with its own conventions and failure modes. There was no identity resolution to speak of: the same consumer appeared as separate records across ad platforms, e-commerce orders and support tickets, which meant audience counts clients could see didn't add up.
Customer segments rebuilt in a nightly batch that frequently overran into the business day, so "yesterday's audience" was often the freshest thing available for campaign activation. Pipeline failures were a weekly ritual, and the two in-house data engineers spent most of their time firefighting rather than building. The company wanted a rebuilt platform — but also wanted the team doing it to stay embedded, because the CDP was their core product, not a side project to hand off.
What We Did
Dillo staffed a 4-person data pod — two senior data engineers, an analytics engineer and a streaming specialist — in two weeks. The pod joined the client's sprints and worked alongside the in-house data engineers, with the client's head of engineering owning priorities and Dillo owning employment, on transparent flat-margin pricing.
- Replaced the connector sprawl with a unified ingestion layer: standardized extraction into Snowflake with consistent schemas, schema-drift detection and per-source data contracts, covering 40+ ad, commerce, CRM and support sources.
- Rebuilt transformation logic in dbt — versioned, tested, documented models replacing the undocumented SQL scripts — with Airflow orchestrating dependencies instead of overlapping cron jobs.
- Built an identity graph that resolves consumers across sources using deterministic keys first and survivorship rules for conflicts, giving every downstream feature a single customer identifier.
- Introduced Kafka-based streaming ingestion for high-velocity event sources, so segments update from a stream rather than waiting for the nightly batch.
- Added observability and data-quality gates — freshness SLAs, volume anomaly checks and test coverage in CI — with Looker dashboards making pipeline health visible to the whole company.
The Results
- 40+ data sources unified under one ingestion framework with consistent schemas and contracts — adding a new source went from a custom project to a configuration task.
- Segment refresh dropped from 24 hours to about 15 minutes, letting clients activate audiences on near-real-time behavior instead of yesterday's batch.
- 50% fewer pipeline failures, and the failures that do occur are caught by quality gates before clients see bad numbers.
- Identity resolution ended the mismatched-audience-count support tickets that had dogged the sales and success teams.
- The client's two in-house data engineers moved from firefighting to feature work, and the pod remains embedded as the platform's core data team.
Figures reflect outcomes reported for this engagement; they are project results, not audited benchmarks.
Tech Stack
Services Used
IT Staff Augmentation — a dedicated data-engineering pod embedded in the client's team, on Dillo's transparent flat-margin pricing.