FlameRangers | Firestore Triggers Fire Twice: Idempotency, Retry Storms, and the Dead-Letter Path

Background Cloud Functions are at-least-once. Not once. The documentation says so in one line, and most teams read past it, because in development the function runs exactly one time per write and the problem stays invisible until a customer is billed twice.

There are two separate failure modes hiding behind that one line, and they need different fixes.

Duplicate delivery. The same event is handed to your function more than once, even though your code succeeded. Infrastructure retries, redeliveries after a timeout, an ack that got lost.
Retry loops. Your function throws, retries are enabled, and the platform redelivers the same event for up to seven days. If the error is deterministic — a null field, a schema mismatch — every one of those attempts fails too.

The first corrupts data. The second corrupts the invoice.

Why the second one is expensive

A retry loop is quiet. Nothing pages. Here is what one deterministic bug in a write trigger costs on a collection taking 20,000 writes a day.

Each failing invocation: one function invocation, ~2s of GB-seconds, and whatever reads the handler did before it threw. Say the handler does 3 Firestore reads before hitting the bad field. Retries back off, but on a Gen 2 trigger with default settings a single event can be redelivered on the order of a few hundred times over its retry window before it expires.

20,000 poison events × ~200 attempts = 4 million invocations and 12 million reads, for work that never succeeds. At $0.06 per 100k reads, the reads alone are roughly $7 per day of the bug, plus compute, plus the log ingestion — and Cloud Logging at that volume is frequently the largest line of the three. The bug was shipped on a Friday. Nobody looks at the bill until the fifth.

The point is not the exact dollar figure. It is that a failing function is not free, and it fails on a schedule you did not choose.

Fix 1: make the handler idempotent

Every event has a stable event ID. Use it as a Firestore document ID in a dedupe collection, and claim it inside a transaction before doing the side effect.

import { onDocumentCreated } from 'firebase-functions/v2/firestore';
import { getFirestore, FieldValue } from 'firebase-admin/firestore';

const db = getFirestore();

export const chargeOnOrder = onDocumentCreated(
  { document: 'orders/{orderId}', retry: true },
  async (event) => {
    const eventId = event.id;
    const claim = db.doc(`_processed/${eventId}`);

    const fresh = await db.runTransaction(async (tx) => {
      const snap = await tx.get(claim);
      if (snap.exists) return false;          // already handled
      tx.set(claim, { at: FieldValue.serverTimestamp() });
      return true;
    });

    if (!fresh) return;                        // duplicate delivery, exit clean

    await chargeCard(event.data!.data());
  },
);

Two things to be honest about with this pattern:

The claim and the side effect are not atomic. If the process dies after the claim commits and before chargeCard returns, the retry sees the claim and skips the work. For money, write the claim and the result together, and have the retry finish an incomplete claim rather than skip it.
The dedupe collection grows forever. Put a TTL policy on the at field set to a few days past your retry window. Firestore deletes expired documents for you, and TTL deletes are billed as deletes — budget one delete per event.

If the side effect is in a third-party API, prefer their idempotency key over your own bookkeeping. Stripe, for one, takes an Idempotency-Key header; pass the event ID and let them dedupe.

The cheaper case: writes that are already idempotent

Not every handler needs a claim document. A handler whose only side effect is set() with fixed data, or an increment that you can derive rather than accumulate, is naturally safe to run twice. FieldValue.increment(1) is not — it is the classic duplicate-delivery bug. Recomputing a count from a query and writing the absolute value is.

Cheapest correct handler is the one that needs no dedupe state. Look for that shape first.

Fix 2: decide deliberately about `retry`

On Gen 2, event-driven functions have retries off by default. That is usually the right default, and teams turn it on without deciding what "on" means: at-least-once delivery for up to seven days, including for bugs.

The rule we use:

Retry on when the work must not be lost and the failures are transient — a downstream API timing out, a quota blip.
Retry off when a dropped event is recoverable by other means — a nightly reconciliation job, a backfill script you actually keep working.

If retries are on, make your handler distinguish the two error classes itself, and never throw on a permanent one:

try {
  await doWork(event);
} catch (err) {
  if (isPermanent(err)) {
    logger.error('poison event, parking', { eventId: event.id, err });
    await db.collection('_deadletter').doc(event.id).set({
      payload: event.data?.data() ?? null,
      error: String(err),
      at: FieldValue.serverTimestamp(),
    });
    return;                // swallow: do NOT trigger a redelivery
  }
  throw err;               // transient: let the platform retry
}

A malformed payload will still be malformed on attempt 200. Park it, alert on the size of _deadletter, and move on.

Also bound the window. Firestore triggers honor an event age limit; anything older than the cutoff is dropped rather than retried. Check event.time at the top of the handler and return early:

const ageMs = Date.now() - Date.parse(event.time);
if (ageMs > 10 * 60 * 1000) {
  logger.warn('event too old, dropping', { eventId: event.id, ageMs });
  return;
}

Ten minutes of retries is a real backoff budget. Seven days is a billing incident.

Fix 3: for Pub/Sub, use a real dead-letter topic

If the trigger is Pub/Sub rather than Firestore, you do not have to hand-roll the parking lot. Configure a dead-letter topic on the subscription with maxDeliveryAttempts (5 is a sane start). After five failures the message goes to the DLQ topic, the subscription moves on, and you get a queryable backlog of exactly what broke.

gcloud pubsub subscriptions update SUB_NAME \
  --dead-letter-topic=projects/PROJECT/topics/SUB_NAME-dlq \
  --max-delivery-attempts=5

Grant the Pub/Sub service account roles/pubsub.publisher on the DLQ topic and roles/pubsub.subscriber on the source subscription, or the DLQ silently does nothing and you are back to infinite redelivery.

Fix 4: cap the blast radius

Two settings on the function, independent of correctness:

maxInstances. A retry storm scales out like real traffic. An uncapped function will happily run 1,000 instances against a broken downstream and take your Firestore write quota with it. Set a ceiling you can afford.
concurrency (Gen 2, HTTP-style workloads). Higher concurrency means fewer instances for the same throughput, which lowers both cold starts and GB-seconds — but a handler with per-request global state will break. Verify before raising it.

And set a budget alert scoped to the function's cost, not just the project total. A 30% spike in one service is detectable long before it moves the project number. We covered the alerting plumbing in Hard Caps on the Firebase Bill.

Testing it

The emulator will not deliver an event twice for you. Test duplicate handling directly: call the exported handler with the same synthetic event object twice and assert one side effect.

const evt = makeEvent({ id: 'evt_1', data: { amount: 500 } });
await chargeOnOrder(evt);
await chargeOnOrder(evt);
expect(chargeCard).toHaveBeenCalledTimes(1);

Three more cases worth having: a permanent error lands in _deadletter and does not throw; a transient error does throw; an event older than the age cutoff returns without doing work. Four tests, and the seven-day loop stops being possible.

What to check this week

List every event-driven function with retry: true. For each, ask what happens on a deterministic bug.
Find the handlers using FieldValue.increment inside a trigger. Those are your duplicate-delivery bugs.
Confirm maxInstances is set on every trigger that writes to Firestore or calls a paid API.
Check whether anything is already looping — Cloud Logging, filter on severity=ERROR grouped by function, look for one event ID appearing dozens of times.

Item 4 finds something surprisingly often. If it does on your project, that is the audit finding; the other three are the fix.

If you want a second pair of eyes on a trigger architecture — or on the invoice it produced — tell us what you are running.

Firestore Triggers Fire Twice: Idempotency, Retry Storms, and the Dead-Letter Path