Data Engineering

Tools

• Pandera - A light-weight, flexible, and expressive data validation library for dataframes.
• atlas - A modern tool for managing database schemas.
• dbc - The package manager for modern database drivers
• kafka-ui - Open-Source Web UI for managing Apache Kafka clusters
• polars - Lightning-fast DataFrame library for Rust and Python.

Resources

• Data Mesh Architecture
• Gently Down the Stream - A gentle introduction to Apache Kafka.
• Data Engineering Vault

Glossary

• Data Engineering Terms Explained
• Modern Data Stack Glossary

Management

• Enterprise Data Team