Reflecting on Architecture Summit 2024: Shaping the Future of Zeta’s Data Strategy

By Manoj Agarwal, Principal Architect and Bharat Goyal, Head of Engineering. The Zeta Architecture Summit 2024 marked a pivotal moment for our organization, bringing together over 30 engineering and product leaders to tackle one of our most pressing challenges: designing a scalable, efficient, and forward-looking data architecture. This internal conference served as a catalyst for innovation, collaboration, and alignment, all centered around two core themes: scaling efficiently and enhancing ZMP’s value proposition through a clear strategy for growth and innovation.

Achievements and Insights

The summit delivered significant outcomes, generating actionable recommendations that will guide our data strategy in 2025 and beyond.

1. Optimizing Data Infrastructure with a Lakehouse Approach

Our current data storage and processing approach is not optimized for semi-structured and unstructured data, leading to inefficiencies. To address this, we explored evolving our data architecture to include a Data Lakehouse, which offers scalable and cost-effective solutions for managing diverse data types. While certain tools and technologies will remain integral for specific needs, the focus will be on streamlining usage to enhance overall efficiency.

2. Unified Dataflow Framework

Currently, multiple data pipeline frameworks exist across different teams, supporting thousands of batch jobs but leading to fragmentation and inefficiency. A proposed Unified Dataflow Framework would consolidate batch and stream processing, integrating observability and data quality controls to create a cohesive and scalable solution.

3. Data Canonicalization

Standardizing domain-specific schemas will enable reusable, modular solutions that accelerate customer onboarding and improve the out-of-the-box value of our platform. This initiative aims to enhance consistency and reduce time-to-value for new users.

4. Advancing the Data Architecture

The primary focus is to enhance developer productivity by creating a comprehensive data catalog that improves data discoverability across teams, enabling faster insights and smoother collaboration. Additionally, establishing robust test environments with sufficient and representative data is crucial for ensuring reliable development, thorough testing, and accurate system validation.

Another key priority is advancing data-sharing capabilities with customers and partners. This includes leveraging advanced collaboration tools and techniques such as Federated SQL for real-time querying and data clean rooms, which enable secure collaboration on shared datasets while safeguarding data privacy. These initiatives aim to streamline operations and foster seamless, secure partnerships.

Notable Highlights

• Collaboration at scale: Participation exceeded expectations, with diverse but right sized teams contributing to cross-functional alignment.
• Rapid consensus: Despite the complexity of discussions, solutions were discussed and agreed upon in many sessions in record time, reflecting the expertise and maturity of our technical leadership.
• Energized culture: Attendees left the summit inspired and motivated to influence and tackle the ambitious roadmap for 2025.

Next Steps

To maintain momentum and incorporate summit recommendations into our 2025 roadmap, we have outlined the following actions:

1. Data strategy document: Publish a comprehensive data strategy document summarizing all recommendations and outlining the blueprint for the next three years.
2. Proposals for top initiatives: Create detailed project plans for the top three priorities to launch in Q1 2025, with dedicated owners and timelines to ensure sustained progress.
3. 2025 roadmap: Deliver a cohesive roadmap aligning business, product, and engineering goals to drive our data strategy forward.

Organizing Your Own Data Summit: A Guide for Leaders

If you’re inspired to host a similar summit, here’s how we structured ours to ensure success:

1. Define Clear Objectives

• Focus on a few critical themes to guide discussions. For us, scaling efficiently and driving innovation were key.
• Align the summit’s purpose with long-term organizational goals to ensure its relevance. We connected the desired data strategy goals with our business strategy at the onset of the summit.
• An ideal time to convene a summit is when you need to discuss high-stakes and complex issues or anticipate major changes ahead.

2. Assemble a Diverse Team

• Invite the right mix of technical and product leaders and engineers from cross-functional teams.
• Encourage broad participation to ensure multiple perspectives are considered.
• Limit the attendees to the right and required builders in the room so that meaningful discussions happen.

3. Plan for Depth and Focus

• Organize sessions around specific challenges or opportunities, allowing deep dives into topics.
• Encourage thinking big by fostering a conducive environment for innovative thinking.
• Use structured workshops and breakout groups to foster collaboration.

4. Prioritize Actionable Outcomes

• Each session should conclude with tangible takeaways, such as recommendations, proposed solutions, or next steps.
• Assign clear ownership for follow-up actions to maintain accountability.
• A good summit generates new ideas and many next steps. A successful one can produce a range of outcomes.
• Creating a shared vision and recommendations for the future direction.

5. Facilitate Collaboration

• Create an open environment where attendees feel empowered to share ideas, challenge assumptions, and build consensus.
• Use facilitators or moderators to keep discussions productive and on track.
• Embrace it as a chance to disrupt old ways of thinking.

6. Document and Share

• Capture all discussions and decisions during the summit for later review.
• Publish a post-summit report or strategy document to align everyone on the outcomes and roadmap.

7. Follow Up

• Schedule regular check-ins to review progress on key initiatives identified during the summit.
• Keep the momentum going by celebrating wins and addressing challenges promptly.

By focusing on these principles, your summit can become a powerful platform for innovation, collaboration, and alignment—just as it was for us.

We hope our experience inspires you to think big and organize a summit tailored to your organization’s needs. Together, we can build systems that scale, innovate, and deliver extraordinary value.