Hire Apache Kafka Engineers

Most Kafka problems are decided on the day the topic is created and discovered a year later. Partition count and partition key together determine what is ordered relative to what — and ordering is a business question, not an infrastructure one. If events for one order can land on three partitions, they can be processed out of sequence, and a cancellation can be applied before the booking it cancels.

The second decision is delivery semantics. At-least-once is the sane default and it means consumers must be idempotent, because they will see the same event twice. Teams that assume exactly-once because the documentation mentions it, without enabling or understanding what it covers, end up with duplicates in a downstream system and no idea where they entered.

Schemas are the third, and the one that stops teams being able to deploy independently. Without a registry and an agreed compatibility mode, adding a field becomes a coordinated release across every consumer — which is precisely the coupling the event spine was introduced to remove.

Then there is operating it. Consumer lag that nobody watches, a rebalance storm caused by a slow consumer inside the poll interval, retention set to a week when the replay you eventually need is from a month ago. We staff people who have been on call for a stream, because that is where these lessons come from.