Pebellslearn by building
RoadmapsMatrixCoursesSandboxSign in

Course details

Data Engineer

Learn how data travels from the apps that create it to the dashboards and models that use it. You will follow one bike-share pipeline from source systems through storage, modelling and scheduled loads.

Ready to learn6 sections · 23 lessons
Start learning

Course curriculum

Introduction to Data Engineering3 lessons

Meet PedalCity's data problem, see how data engineering differs from data science and analysis, and learn the few skills and three questions behind every technology choice.

  1. What Is Data Engineering?12 min
  2. Data Engineering vs Data Science12 min
  3. Skills and Choosing Technologies12 min
The Data Engineering Lifecycle5 lessons

Trace a PedalCity ride through generation, storage, ingestion and serving, and learn the questions that keep each stage dependable.

  1. The Lifecycle at a Glance12 min
  2. Data Generation12 min
  3. Data Storage12 min
  4. Data Ingestion12 min
  5. Data Serving12 min
Sources of Data3 lessons

Learn how to copy from a production database without hurting it, how to extract from APIs that page and limit you, and how to tame late and duplicate events from logs, apps and sensors.

  1. Databases as Sources12 min
  2. APIs as Sources12 min
  3. Logs, Mobile Apps and IoT12 min
Database Fundamentals4 lessons

See why transactional databases normalize, how analytical systems differ, and how to model a warehouse with facts, dimensions and slowly changing dimensions.

  1. Data Normalization12 min
  2. OLTP vs OLAP12 min
  3. Star and Snowflake Schemas12 min
  4. Slowly Changing Dimensions12 min
Warehouses, Marts and Lakes3 lessons

Learn what makes a data warehouse, how data marts give each team a focused view without splitting the truth, and when a data lake or lakehouse is the better home.

  1. What Is a Data Warehouse?12 min
  2. Data Marts12 min
  3. Data Lakes and Lakehouses12 min
Ingestion and Pipelines5 lessons

Choose between batch, streaming and hybrid ingestion, decide between ETL and ELT, make every load safe to rerun, and tie it all together with an orchestrated graph of tasks.

  1. Batch Ingestion12 min
  2. Realtime and Hybrid Ingestion12 min
  3. ETL and ELT12 min
  4. Idempotent Loads12 min
  5. Orchestration and Scheduling12 min