Data engineering guiding principles
These are practices I’ve observed that go hand in hand with top tier development. These are meant to be more as guidelines and less like dogma - there are other ways of doing things that work well too. I just think this is a good baseline.
- Discoverability: Canonical (production) metrics and the dashboards that support them must be easy to find if someone only has a link to the landing page. This principle establishes the need for an intuitive metrics taxonomy.
- Minimize cognitive load: Metrics created by the data team must be simple, and visualizations easy to understand. Metrics definitions should be as easy to find as a tool-tip away. Any opportunity for confusion must be minimized.
- Democratize data intuition: The goal is to ensure the underlying data model is designed to make it easy for non-data experts to start learning how to answer their own questions without the help of a data expert. This is a crucial step toward enabling self-service on the platform.
- Robust usability: Users of the analytics tools must be able to explore data with low latency, and the data team must have a strategy for maintenance and SLAs.
- Dev-friendly: While the first four principles address the needs of the product teams as customers, data practitioners who support those teams must also be considered customers of the platform.