I came across a LinkedIn post recently that got me thinking for a few days. It was making a quiet but sharp argument: AI has created a speed problem that the data platform is quietly absorbing and starting to struggle with.
I want to unpack why I think this observation is correct, where I'd push back slightly, and what it means for data engineers working inside Snowflake or Databricks environments right now.
What is entropy? - Jeff Phillips
The core argument goes like this: AI tools have dramatically accelerated the speed at which data engineers build pipelines. SQL gets generated in seconds. dbt models appear from prompts. Transformations that used to take days now take minutes.
But distributed data platforms reward discipline and punish entropy. Snowflake, Databricks, Spark, they all behave according to physical and economic patterns that don't change because you used a language model to write the SQL.
Bad partition pruning is still bad partition pruning. A skewed join is still a skewed join.
What AI changes is the rate. Before, a team might add five new dbt models to a shared environment in a week. Now they might add fifty. The platform doesn't care how the code was written. It taxes entropy at the same rate it always did. You just hit the wall ten times faster.
The result is a pattern that, apparently, is showing up across many environments:
Sound familiar?
I think the root cause is something specific: velocity increased faster than architectural constraints. The guardrails that used to slow teams down (code review, careful testing, a senior engineer who knew the whole platform) are now optional in a world where a junior engineer can generate working SQL in thirty seconds.
The code works. It just works in a way that, at scale and in combination with fifty other pipelines someone else generated, quietly degrades the system.
This is not a new problem, to be clear. Every senior data engineer I've worked with has seen some version of it. What is new is how fast it accumulates, and how invisible it is until the Snowflake bill arrives.
The post suggests that a new specialization is emerging: something like a system stabilization engineer. Not a pipeline builder. Not a traditional architect. Someone who sits between the platform and the pipelines, watching the system as a whole.
The work this role does includes things like:
The post is directionally correct. But I'd push back slightly on the framing of it as something truly new.
This work has always existed. It just used to be done informally by the staff engineer who ran the quarterly cost review, or the architect who maintained golden-path templates, or the senior analytics engineer who could read a query plan and immediately know there was a problem.
What AI has done is make this work urgent rather than aspirational. The organizations that treated platform hygiene as a nice-to-have are now seeing the bill for that decision in very concrete terms.
Whether this crystallizes into a dedicated job title or simply becomes a formalized set of responsibilities for senior data engineers: I genuinely don't know yet. My guess is it will be different in every organization. Large data platforms at enterprise companies will probably hire for it explicitly. Smaller teams will probably assign it to whoever has the most platform context.
If this resonates, you don't need a new hire to start addressing it. A few practical starting points:
Assign ownership for platform-level monitoring. Someone on the team should have explicit responsibility for watching the platform as a system, not just their own pipelines. If the answer to "who watches for cost drift?" is "everyone in theory," it's nobody in practice.
Implement query tagging. Both Snowflake and Databricks support labeling queries with team and workload metadata. This makes cost attribution possible and makes debugging much easier.
Add a query plan review to your PR process. Especially relevant if your team is generating SQL with AI tools. The PR template can include a simple checklist: does this query use proper filters? Is there a full table scan on a large table?
Run a monthly cost review. It doesn't need to be elaborate. Look at what changed, find the biggest movers, and ask whether those changes were intentional.
I find this conversation interesting partly because it reflects a broader tension in data engineering right now. We moved away from centralized governance deliberately, because it was slow. Now we're discovering that the slowness had a function: it was absorbing entropy. AI didn't create this tension, but it has made it impossible to ignore.
The tools are genuinely excellent. The speed gains are real. The challenge is making sure the platform can absorb what you're building inside it.
Someone should probably be responsible for that. Who? 🦉
Photo by Marc Sendra Martorell on Unsplash