EventStream 2.0

Alerting ran through a PL/pgSQL engine that evaluated 261 rules sequentially against 189M rows. From event to alert took over 4 minutes — long enough that an alert was a record of something that had already finished happening.

Every new integration was also written by hand, so the cost of connecting another source was another codebase to keep.

Built an event processing platform in Go on GCP that handles millions of events a day. The design is borrowed from NiFi: every integration is a configurable graph of processors, so a new source is assembled rather than written.

While tracing the data path, found and accounted for BigQuery spend that no team owned.

Alerts now fire in seconds rather than over four minutes.

Roughly $7.5k a month of unaccounted BigQuery spend traced and removed.