Hire Databricks Data Engineers

The recurring problem with Databricks is not Spark. It is that a notebook which produced the right answer once gets promoted to a scheduled job, and nothing about it was designed to be run twice. It appends rather than merges, it depends on a widget somebody set by hand, and the first time it is re-run after a failure it doubles a table.

The engineering question is what a rerun does

Delta’s merge exists for exactly this, and using it means having a business key and knowing what a late-arriving record should do to a row that is already there. That is a modelling decision, and skipping it is the single most common cause of a lakehouse whose numbers nobody trusts.

An idempotent pipeline is one where running the same input twice leaves the same table behind. In Delta that usually means a merge on a business key, a deterministic partition or watermark for what is in scope, and no reliance on the job having run exactly once.

Notebooks, jobs, and the gap between them

A notebook is an exploration tool. It keeps state between cells, it depends on the order somebody ran them in, and it is very hard to test. None of that matters while a person is sitting in front of it and all of it matters at two in the morning.

The transition that has to happen is not cosmetic. Logic moves into modules that can be imported and unit tested, parameters come from the job definition rather than from a widget, and the notebook, if it survives at all, becomes a thin caller. Teams that skip this step end up with production logic that can only be changed by the person who wrote it, in a browser.

Layering works only if the contract between layers is real

Bronze, silver and gold are useful only if there is a real contract between the layers — bronze is what arrived, silver is conformed and deduplicated, gold is what the business agreed. Where those boundaries are decorative, every consumer reads from whichever layer is convenient and the model stops meaning anything.

The contract is enforceable in practice: bronze is append-only and never edited, silver has declared keys and expectations that fail the run when they are violated, and gold is the only layer a report is allowed to read. A dashboard querying bronze is not a shortcut, it is the point at which the layering became decoration.

Governance, and who can see what

Unity Catalog exists so that access is granted on tables and columns rather than on storage paths, and so that lineage can answer where a number came from. Estates that predate it tend to have access controlled at the storage account, which means access is effectively all or nothing and lineage is a person’s memory.

Lineage matters most in the least interesting moment: somebody disputes a figure in a report and the question is which upstream change produced it. Without it, that is an afternoon of reading notebooks.

Cost is mostly operational discipline

Right-sized clusters, jobs rather than all-purpose compute for scheduled work, auto-termination that is actually set, and someone who looks at what the largest job is doing. We staff people who have had that conversation with a finance team.

The single largest saving in most estates is not tuning. It is moving scheduled work off interactive clusters that were left running because somebody might need them, and that is a policy decision rather than an engineering one.

When Databricks is more than the problem needs

If the data volumes fit comfortably in a warehouse and the work is SQL transformation, a warehouse is less to operate and easier to hand over. Spark earns its cost on volume, on unstructured data, or on workloads that genuinely need Python rather than SQL.

If there is one pipeline and one consumer, a scheduled script with tests will do the job and can be understood by whoever inherits it.

The case for a lakehouse is several sources, mixed structured and unstructured data, machine learning work that needs the same tables as the reporting, and enough consumers that a shared governed layer is worth the effort of maintaining one.