At-Least-Once Is Lying To You: Kafka Delivery Guarantees

Almost every distributed messaging conversation eventually arrives at the holy grail: exactly-once delivery. New Kafka users often assume at-least-once is good enough and exactly-once is marketing, or the reverse, that Kafka magically dedupes everything. Both views are wrong. Delivery semantics in Kafka are the product of how producers retry, how consumers commit offsets, and whether you use transactions to bind reads, processing, and writes into one atomic unit. The honest framing is that exactly-once is achievable within Kafka's boundaries, and at-least-once with idempotent processing is what most real systems should target. As a backend developer, you have probably written a payment handler and worried about double charging on retry. That instinct is exactly the muscle you need here. Let us pull apart the three classic guarantees, see where each breaks, and understand what Kafka actually provides and what remains your responsibility.

The Three Delivery Semantics

There are three possible guarantees. At-most-once means a message may be lost but never processed twice; you get this by committing offsets before processing, so a crash mid-work skips the record forever. At-least-once means a message is never lost but may be processed more than once; you get this by processing first and committing after, so a crash before commit replays the record. Exactly-once means each message affects your state precisely once, no loss and no duplication. At-least-once is the sensible default for most pipelines because losing data is usually worse than handling a rare duplicate. The crucial insight is that these are not abstract settings you toggle; they emerge from the ordering of three operations: producing, processing, and committing offsets. Where you place the offset commit relative to your side effects determines which guarantee you actually have.

Why Duplicates Are Inevitable

Duplicates are not a bug you can configure away in a distributed system; they are a consequence of network uncertainty. Imagine a producer sends a record, the broker writes it successfully, but the acknowledgement is lost on the way back. The producer cannot tell whether the write succeeded or failed, so it retries, and now the same record exists twice. The identical problem appears on the consumer side when processing succeeds but the offset commit fails before a crash. No amount of careful coding eliminates this, because the failure happens in the gap between an action and its confirmation. This is why mature systems lean on idempotency rather than trying to prevent retries. If processing the same event twice produces the same result as processing it once, duplicates stop mattering, and you can retry freely without fear of corrupting state.

The Idempotent Producer

Kafka's first line of defense is the idempotent producer, enabled with enable.idempotence set to true, which is the default in recent versions. Each producer is assigned a unique producer id, and every message in a partition carries a sequence number. The broker tracks the last sequence number it accepted per producer per partition and rejects any duplicate or out-of-order sequence. This means the lost-acknowledgement retry scenario no longer creates duplicates: the broker recognizes the retried record's sequence number and silently deduplicates it. Importantly this guarantee is scoped to a single producer session and a single partition; it does not span producer restarts or multiple partitions on its own. But it removes the most common source of duplication essentially for free, and because it pairs with acks=all, turning it on gives you exactly-once writes from one producer to one partition with no application changes.

Transactions Across Partitions

The idempotent producer handles one partition, but real workflows write to several topics and partitions atomically. Kafka transactions extend the guarantee so that a set of writes across multiple partitions either all become visible or none do. You assign a transactional.id, begin a transaction, produce your records, and commit. Until you commit, consumers configured with isolation.level set to read_committed will not see the records, and an aborted transaction makes them invisible permanently. This lets you publish to three topics and have downstream readers observe an all-or-nothing result, similar to a database transaction spanning multiple tables. The transactional.id also enables zombie fencing: if an old producer instance comes back after a new one took over, the broker fences out the stale one, preventing it from corrupting the stream. This is the building block for true exactly-once stream processing.

Read-Process-Write Atomicity

The killer application of transactions is the read-process-write loop, which is the shape of most stream processors: consume from an input topic, transform, and produce to an output topic. The danger is that producing the output and committing the input offset are two separate actions, so a crash between them either drops or duplicates work. Kafka solves this by letting you commit consumer offsets inside the same transaction as your output writes, using sendOffsetsToTransaction. Now the offset advance and the output records commit atomically: either both happen or neither does. This is what people mean by Kafka exactly-once. It only holds when both the source and sink are Kafka, because the atomicity lives inside the Kafka protocol. The moment your sink is an external database or an email send, the transaction can no longer wrap it, and you are back to needing idempotency on that external effect.

Exactly-Once Stops At Kafka's Edge

This boundary is the single most important thing to understand. Kafka transactions give exactly-once semantics for data flowing Kafka-to-Kafka, including offset commits. They cannot extend that guarantee to external systems Kafka does not control. If your consumer reads an event and charges a credit card, Kafka cannot make that charge transactional with the offset commit. The card gets charged through an external API, and if your offset commit fails afterward, you will reprocess and charge again. The realistic engineering answer is to make external side effects idempotent: pass an idempotency key to the payment provider, or use a database unique constraint keyed on the event id so a duplicate insert harmlessly fails. Treat Kafka's exactly-once as covering the internal pipeline, and treat every external write as something you must independently protect against duplication.

The Practical Default To Choose

For most teams, the right target is at-least-once delivery combined with idempotent processing, not full transactional exactly-once. Configure acks=all, enable the idempotent producer, process each record, and commit offsets only after processing succeeds. Then make your handlers idempotent by deduplicating on a stable event id, whether through a unique key in your database, a Redis set of seen ids, or an upsert that is naturally safe to repeat. This gives you the practical correctness of exactly-once without the operational weight and throughput cost of Kafka transactions, which add latency and complexity. Reserve full transactions for genuine Kafka-to-Kafka stream processing where atomic read-process-write actually pays for itself. The mindset shift from MERN is to stop trying to guarantee a message arrives once and instead guarantee that processing it many times is indistinguishable from processing it once.

Testing For Duplicate Safety

Because duplicates are inevitable, you should deliberately test that your consumers tolerate them rather than hoping they never occur. A useful practice is to inject duplicates in a staging environment: replay the same batch of events twice and assert that your database, counters, and downstream effects end in the same state as a single run. Chaos-style tests that kill a consumer mid-processing, before it commits offsets, surface exactly the replay path that bites you in production. Verify that your idempotency keys are actually unique and stable across retries, since a key derived from the current timestamp instead of the event id will silently fail to dedupe. Treat duplicate handling as a first-class correctness property with its own tests, the same way you would test authentication or money movement. The systems that survive are the ones that assumed retries from day one rather than discovering them during an incident.