I just picked up my Databricks Fundamentals certificate, one more practical badge to back up the tinkering. This builds on my existing Azure Databricks experience and tools work and gives me the platform confidence to stop guessing and start building.
Next up: the plan I laid out earlier — finish the Databricks Data Analyst course, then move into Data Engineering. Practically that means small, repeatable familiar projects on Databricks...
Sometimes consulting feels less like a career choice and more like being handed a mystery box with a sticky note that says, “figure it out.” As a data-migration and data-visualization practitioner, my current company has shoved a few of those boxes my way and I’ve opened most of them. The latest one involved hardware telemetry on a big-data platform called Databricks. I did hear the name before but never had the time to deep-dive.... until now. Fate, or management, doesn’t always ask politely.
My path was not a straight line. I started as a game developer (that backstory deserves its own post how I slowly shifted my gear), then with many changes then landed onto data analysis, and got comfortable getting data, facts, producing dashboards and operational reports. For a while I thought, “Data analyst, that’s the lane!” Then reality and technology started to argue. As data becomes more standardized and AI grows smarter at consuming metadata and domain models, many routine analyst tasks will get automated. That does not mean analysts vanish tomorrow, but the role is evolving, and I do rather shape the steering wheel than get shuffled out of the car.
So now I am stepping up: data engineering. That is the team that designs, builds, and keeps the pipelines and systems humming as tech evolves. It’s less about ad-hoc queries and more about architecting reliable, repeatable data flows. Which, to be honest, suits me: I have already led ETL development and built dashboards for government projects, translating business requirements into end-to-end frameworks and data lineage so stakeholders actually trust the numbers I produced. I have designed and implemented ETL workflows that move data from source systems (we are talking a few hundred tables) into a centralized data warehouse, and I even automated script generation to cut human error when adding new sources and yes, that saved time and dignity in equal measure.
On the visualization side, I have built data models and daily-operational dashboards that help agencies manage operations — not pretty charts for the sake of prettiness, but tools that answer real questions and reduce frantic “where’s the data?” emails. Part of my job has been to mediate between business users and data owners, offering technical guidance while keeping the conversation practical. I also helped the Data Reference Team with an analytical summary dashboard and advised on which data points needed calibration to improve outcomes.
This Databricks project feels like a nudge toward where I want to be. With my background in software, hardware, and governance, the jump to data engineering is not random, it’s a logical upgrade. The real hurdles are brushing up on Python and getting comfortable with Spark, but my programming foundations and discipline for clean code make that more of a sprint than a mountain. Plus, learning Spark feels a bit like learning a new chess opening frustrating at first, then oddly satisfying when the gambits work.
So now here’s the plan: I’ll treat this project as a concentrated course in data engineering — Databricks as the classroom, telemetry as the case study. I’ll keep doing the things that matter (pipeline reliability, data quality, clear lineage, and dashboards that actually inform decisions) while sharpening the new skills I need to own the next level. If data analyst is the reconnaissance, then data engineer is the logistics and infrastructure and I’m ready to build the roads.
Take a look at this snippet of the code
so when you call the functions, you will normally call this way
Another way to improve and more flexible ways and cleaner code without much repetitive in codes.
You may notice at first the codes are not easily read to understand but if further notice that there is no repetitive in the code.
You can call it 2 ways instead.
You just need to call func('doAll') it will also act upon do1 and do2.
This is useful if you need to call custom function if you want to code it to run selective case (eg. funcA to run func1 and func2 then another funcB to run func1 and func3) so you reduce the number of lines and no repetitions of codes.
Make Assignment Operator a cleaner way using this method
At least 8-12 lines of codes including braces
You can write this way below to achieve the same result!
4 lines of code only :D
This is useful if you want to switch your operators from minus to plus and vice versa!
