Customer project: data as a product with dbt and Snowflake

I am writing this blog shortly after the winter break. I am currently ready to join a new project, using this time to develop and to reflect back on last year.

The project, in short

In 2025, I joined one of the data teams at one of the world’s leading logistics and transportation companies. The client company, whose name I won’t mention in this blog post, is in the process of changing their data architecture to a data mesh. 

If I have to describe Data Mesh with two (long) sentences:

“Data mesh is a decentralized architectural framework that shifts data ownership from a central team to the specific business domains, such as marketing or sales, that actually generate and understand the data. 

By treating data as a product, it empowers these domain teams to manage their own data pipelines and share high-quality, discoverable datasets across the organization through a self-service infrastructure and federated governance.” 

Without entering in too much detail, the project’s scope was to develop the dbt project by tackling tickets from a Kanban board. 

My contribution as a Data Engineer

The tickets entailed creating and editing Fact and Dimension tables, changing the logic to meet business demands, and improving team productivity and data quality. I used dbt core and built tables in Snowflake, used Gitlab for collaboration and entered the team workflows to ensure a smooth process. 

Over the span of a year, I worked on multiple tickets and developments. If I had to select two achievements worth mentioning, I would choose:

• Created a dbt macro that selected the Snowflake warehouse according to the span of data to process. The macro worked based on values specified in variables defined during the dbt run. A bigger timeframe resulted in the query running using a size S warehouse instead of XS. The thresholds could be specified in each of the models.  
• Resolved a workflow issue by adding a simple y/n question to an existing Python script. This solution was the alternative to a larger automation that would’ve slowed down the execution of the script. It proved to me that in some cases, simple is better. 

Overall, the team was happy with my performance and sad to see me go. I was happy to hear that I would be missed both for my technical contribution and my personality.

Challenge = opportunity for learning

Just like in every project, challenges existed, and they were always a chance to learn and improve myself or aspects of the team. 

Overcoming technical challenges

As much as technical challenges may seem the scariest, they can be resolved using the help of online resources, teammates, and AI agents.

Speaking of AI agents, did you know we can set up a customized AI agent that allows you to ask questions to your CRM data in natural language? 👉🏼 InsightsAI for CRM.

I had strict company policies to follow when it came to the AI agents to use. The team used GitHub Copilot for coding matters. This is because the premium version of Copilot does not use customers’ code to train its models.

Fun fact: traffic on Stackoverflow, for years the place for answers to programming questions, has seen sharp declines after the releases of GitHub Copilot in October 2021 and ChatGPT in November 2023. 

Overcoming non-technical challenges

Non-technical challenges are related to stakeholder and work management:

• How do I keep the team updated on my work? 
• How do I handle customer expectations? 
• How do I ask for feedback so that I can adjust my behavior before conflict arises? 

I definitely learned to be more open and keep communication with stakeholders frequent, as opposed to entering my bubble where I develop in silence. This placement definitely helped me see asking for help as something that benefits the team and the project, nothing to be ashamed of. Transparency allows us to tackle challenges early on and resolve bottlenecks. One day, I am the one who asks for help. On another day, I am the one offering it.

How I experienced the project

Overall, I enjoyed working on this project. I was particularly happy with the tech stack, the project itself, and the team. It was rewarding to receive positive feedback, which I'll add to my career dossier. I learned a great deal and will take these lessons with me to my next placement.

The life of a consultant isn’t always easy. For instance, there's the struggle of not knowing when the next project will start or what it will entail. Being "on the bench" is a valuable time to develop hard and soft skills, build one's professional profile (hence this blog post), and structure long-term career plans.

This leads me to a recent conversation I had with the in-house Data Architect. Given that AI agents are automating much of the development process, I intend to move towards a more strategic role. He was happy to have a 30-minute call with me, where he outlined a potential path forward: gain more experience in development to master different architectures and solutions, which will ultimately lead to a more strategic position.

He also recommended some readings for both technical and soft skill development. I've already started reading “How to Win Friends and Influence People” by Dale Carnegie on my Kindle.

Photo by Ricardo Gomez Angel on Unsplash