Skills and Choosing Technologies

A new PedalCity data engineer reads a job advert listing twenty tools and panics. But the work rests on a small set of skills, and tools are chosen to fit the problem, not the other way round. This lesson covers both: what you need to know, and how to pick a technology without falling for hype.

The rule

Pick tools for the job's size, speed and team — not the hype.

First, the skills. A data engineer's everyday toolkit is short:

• SQL: the language of almost every storage system you will touch. It is the single most used skill.
• One programming language, usually Python, for calling APIs, parsing files and gluing steps together. Java and Scala appear around big processing engines that run on the JVM, and Go shows up in infrastructure tools. You can learn those later if a job needs them.
• Linux and the shell, because pipelines run on servers.
• Git, because pipeline code is code and needs version control.
• Data modelling, cloud basics and a feel for how distributed systems fail.

Pebells has separate courses on SQL, Python, Linux and Git, so this course focuses on the ideas that are specific to data engineering.

The responsibilities follow from the first lesson: build and run pipelines, model data so it is easy to query, guard its quality, control who can see what, keep costs sensible, and notice failures before users do.

Now the choosing. Three questions decide most technology choices:

1. Size: how much data, and how fast is it growing? PedalCity logs about 40,000 rides a day, which is roughly 15 million rows a year. One ordinary database or warehouse handles that easily.
2. Speed: how fresh must the answer be? A daily report can wait hours. A live map of free bikes cannot.
3. Team: who will run it at 3 a.m.? Two engineers should prefer managed services to clusters they must patch themselves.

The shortcut

Before accepting any tool, say out loud: "Our data is ___ big, must be ___ fresh, and ___ people will run it." If the tool is sized for a different sentence, it is the wrong tool.

The trap is choosing a tool because big companies use it. Their size, speed and team sentence is not yours.

Drills

Answer fast — trust the rule.

Remember

• Core skills: SQL, one programming language, Linux, Git and data modelling.
• Pick tools for size, speed and team, not hype.
• Say the sentence first: "___ big, ___ fresh, ___ people will run it."