Data Engineers vs Software Engineers
Why Do We Have Data Engineers?
The Question
Why do we have data engineers? I think software engineers can do it all. Is it right to say data engineers are special types of software engineers?
Background Context
First, read this — this is a transcription of a video posted by @TheEconomist on YouTube:
Why are this year's World Cup tickets so expensive? From England '66 to Mexico '86, to Germany '06, while the players provide the World Cup spectacle, it is the fans who supply its soul. But FIFA risk pricing them out. This year's World Cup will be the most expensive ever. Adjusted for inflation, tickets cost four times what they did when America last hosted in 1994, and more than double Qatar, just four years ago. The cheapest group stage tickets have averaged around 200 dollars. A ticket to the final will set you back at least two Super Bowl tickets, or ten for the Champions League final.
It's all because FIFA has changed its commercial strategy. This is the first World Cup where FIFA has taken direct control of ticketing. It's embraced dynamic pricing, where prices rise with demand to squeeze the most money possible out of every seat. Demand for World Cup tickets has always outstripped supply. But until now, FIFA didn't see charging less than the market to bear as a bad thing. Cheaper tickets fill stadiums with passionate fans from around the world, and noisy stadiums won't make FIFA's main revenue rights — broadcasting streams and sponsorship deals — so valuable. Think of it as an input cost for the real product: televised football.
The last World Cup cycle brought in match-day receipts of about one billion dollars, but that was only 13 percent of total revenue. The vast majority came from broadcasting, marketing rights and licensing. This time, FIFA is aiming to bring 3 billion dollars — 28 percent of the total. Its president has made no secret of the model he is copying.
This is not the whole of it.
Back to the Discussion
I think the above work is made possible because of a data scientist.
What I Know
We have three types of data sources: data lakes, databases, and streaming data.
We need pipelines to extract data, and tools to transform and clean, and other pipelines to move it further for data scientists to use it on Power BI, Tableau, etc., and to data warehouses.
The Question Remains
Now, is this whole process, even to data science, not so vulnerable to AI takeover?
Data engineers build and maintain the infrastructure that makes data available, reliable, and accessible. Software engineers build applications and systems. The roles overlap but are distinct — data engineering is a specialization within the broader software engineering field, with its own unique challenges, tools, and expertise required to handle the scale and complexity of modern data systems.