2  Design Principles

We took some heavy influence from Mage AI Core Design Principles and kept what was valuable for our project.

2.1 Great developer experience

• Scripts and Notebooks (language agnostic) for building ETL pipelines
• Local set up is easy

2.2 Engineering best practices

• Reusable code
• Unit tests: Data validation for each pipeline
• Integration tests: Data validation for all project pipelines as a whole

2.3 Observability

• Infrastructure metrics accessible to executives and analysts
• Inventory of pipelines
• Status monitoring
• lineages

2.4 Reproducibility

• each pipeline and data product can be versioned
• back filling data is a core operation