Challenge
Campaign data from Facebook, Instagram, TikTok and Snapchat was fragmented, three teams repeatedly reimplemented the same ingestion, and the existing Data team, focused on computer vision for feed labelling, operated in isolation.
What I led
- Made the case to leadership for a Data Guild: a four-person team that integrated the existing Data team.
- Built a data lake with Airflow and Snowflake that centralised campaign data from all four platforms. Snowflake was selected by other stakeholders; I designed and implemented the pipelines.
- Decided to migrate the UGC project from Vert.x to Spring Boot for its simplicity and better documentation.
- Delivered with a five-person team operating on a startup model.
Impact
- Three teams stopped duplicating ingestion and moved to a single source of truth.
- The foundation of Ads Composer, a product that provides clients with metrics and identifies the optimal segments for investment.
- Faster, more reliable delivery of data projects.