Security Analytics Pipeline for a Managed Security Services Provider

The Challenge

Every customer onboarded meant another flood of raw events: firewall denies, EDR detections, cloud audit trails, failed logins. The SOC's tooling forwarded vendor alerts nearly one-to-one into the analyst queue, so analysts were triaging tens of thousands of alerts a day, the overwhelming majority of them noise. Genuine incidents risked being buried, alert-fatigue was driving analyst turnover, and the only scaling lever was hiring more analysts — which killed the unit economics of every new contract.

The MSSP's engineering bench was small and fully occupied keeping the existing stack alive. Leadership wanted a dedicated senior data-and-platform team that could build a real analytics layer over their telemetry — working inside their SOC, under their CISO's direction, with their security engineers defining detection logic.

What We Did

Dillo assembled a five-engineer augmented pod — two data engineers, a backend engineer, an ML engineer, and a front-end engineer — screened for prior security-domain work and cleared through the client's vendor-security review. The pod embedded with the SOC, pairing with the client's detection engineers throughout.

• Built a Kafka-based ingestion layer normalizing logs from dozens of source types — firewalls, EDR, cloud audit logs, identity providers — into a common event schema.
• Added enrichment services in Python that stamp every event with asset criticality, user context, GeoIP, and threat-intelligence indicators before it reaches a rule.
• Implemented a correlation engine that groups related events across sources and time windows into single incidents — one credential-stuffing campaign becomes one case, not four hundred alerts.
• Trained ML-based alert scoring on two years of historical triage outcomes, ranking incidents by likelihood of being actionable so analysts work the queue top-down.
• Deployed OpenSearch as the searchable event store, giving analysts fast pivot-and-hunt queries across the full 3TB+ daily volume.
• Delivered a React triage UI presenting each incident with its correlated evidence, score rationale, affected assets, and one-click escalation to customer workflows.

The Results

• The pipeline processes more than 3TB of logs per day across the customer base, with capacity headroom for onboarding.
• Correlation and scoring cut alert noise by roughly 80% — analysts see ranked incidents, not raw vendor alerts.
• Average triage time per incident fell by 65%, and SLA breaches on escalations dropped to near zero.
• The SOC now onboards new customers without proportional analyst hiring, restoring margins on new contracts.
• The five-engineer pod remains embedded, evolving detection content and models alongside the client's security team.

Figures reflect outcomes reported for this engagement; they are project results, not audited benchmarks.

Tech Stack

Services Used

IT Staff Augmentation — a five-engineer pod embedded in the client's security operations center.