Hire Snowflake Data Engineers

Cost stops being an infrastructure problem and becomes a modelling one

Snowflake removes the excuse that the warehouse is slow, which is useful, and replaces it with a bill that grows quietly, which is not. Compute is easy to add and easy to leave running, so cost becomes a design property rather than an infrastructure one — a query pattern that scans a whole table every fifteen minutes is not a tuning problem, it is a modelling problem with an invoice attached.

The controls that work are structural rather than vigilant. Separate warehouses per workload so that a runaway transformation cannot slow a dashboard and so that the bill says who spent it, auto-suspend measured in seconds rather than minutes, and resource monitors set before anyone needs them.

The modelling is where the value actually is

Most warehouses fail not because they are slow but because two teams compute the same figure differently and both believe theirs. Agreeing the grain of a fact table, and where a metric is defined once, is duller than building another dashboard and it is the thing that decides whether anyone trusts the numbers in a year.

Grain is the sentence that says what one row of a table represents — one order line per day, one shipment event, one open position at month end. Where nobody can state it, joins silently duplicate rows and totals become larger than reality in a way that is very hard to see.

Access, and the role that works

Snowflake’s role model is genuinely good and is very often set up as one role that works, because the deadline was Friday. Row access policies and secure views exist precisely so that a shared warehouse does not mean shared visibility, and in a regulated environment the difference is the finding.

Masking policies are worth applying to the obvious columns early rather than as a remediation project. Once analysts have built work on unmasked data, the conversation about restricting it becomes a negotiation with people who have deadlines.

Where an ELT pipeline actually breaks

Streams and tasks make incremental loading straightforward and make it straightforward to get wrong in one specific way: a task that fails, is fixed, and is re-run against a stream that has already been consumed. The offset moved even though the work did not complete.

The pattern that survives that is a merge on a business key into the target, so that a re-run converges rather than appends, and a load audit table recording what each run claimed to have processed. Without the second, nobody can answer the only question that matters after an incident, which is whether anything was missed.

Late-arriving data is the other recurring case. A record that turns up two days after the period it belongs to either updates history and changes a number somebody has already reported, or does not and quietly makes the warehouse wrong. Both are acceptable. Neither is acceptable by accident.

What we interview for

• The grain of a fact table they modelled, stated in one sentence
• What a re-run of their pipeline does, and how they know
• A query whose cost they reduced, and whether the fix was the model or the warehouse size
• How row-level visibility was enforced, and who could see what before they changed it

When Snowflake is not the right answer

If the data fits in the operational database and the reporting load does not disturb it, a warehouse adds a copy, a pipeline and a lag for no benefit. Plenty of organisations are better served by a read replica and some discipline about queries.

If the requirement is an operational lookup — a screen that needs a record now — a warehouse is the wrong shape entirely, and putting it there produces latency the user will notice.

The case for a warehouse is data from several systems that has to be reconciled into one set of agreed figures, volumes that make the operational database unhappy, or history that the source system does not keep.

What owning one teaches that building one does not

We staff people who have owned a warehouse after it was built, not only people who have built one. The habits are different, and only one of them survives contact with a cost review.