EventStream 2.0

The client’s deposits had migrated to a new upstream payment system, but the event pipeline still read only the legacy one. Teams had wired their own direct feeds to fill the gap, so different parts of the business reported different numbers for the same transaction. This was causing data integrity problems in the customer support chat bots which was powered by Dialogflow. The alerting engine was a PL/pgSQL stored procedure on a one minute cron loop, evaluating 261 rules sequentially against 189M rows. It took over 4 minutes from event to alert and structurally could not express the multi-event alert types the business was asking for.

Technical lead on the replacement. A single Go binary on Cloud Run built on an Apache NiFi-inspired model. Every integration is a configurable graph of processors rather than a bespoke microservice, so new destinations become configuration instead of deployments. I own the services that make the platform operable in production: DLQ management, event provenance, a versioned schema registry and source connectors, plus the Terraform behind the GCP footprint. I also set the resource bar for hot path code running millions of times a day. Measured provenance overhead at 4.5% of a core per 1,000 hops a second and designed a 1% sampling rollout that still captures every failure.

Alert latency dropped from over 4 minutes to seconds. Roughly $7.5k a month of BigQuery spend removed. A record shape fix cut logging volume, storage and scan cost by around 2.5 times. One governed pipeline now replaces the fragmented direct feeds across analytics, operations and customer facing chatbots. Multi event alert types that were structurally impossible in the old engine are now expressible.