Retries & failures

Your endpoint will fail. A deploy will restart it mid-request, a database will lock, a certificate will expire on a Sunday. What happens next depends on the status code you return and on how many events in a row go wrong.

This page is the delivery contract from the failure side: the retry ladder, what stops it, the delivery log you can read in the console, Replay, the counter that switches your endpoint off, and what you can recover once an outage is over. Owners and admins manage all of it in Developers → Webhooks; The Developers page shows every screen.

The retry ladder #

Each event is written to the delivery log as queued before it is sent, and then delivered from a durable queue: a restart or a new release on our side does not lose it. A failed attempt that is worth retrying is tried again, up to four attempts in total.

So the waits add up to about 14 seconds. Each attempt can itself take up to 10 seconds (5 to connect), so on an endpoint that hangs, the whole ladder is over in under a minute. Treat the spacing as approximate; build nothing that depends on the exact seconds.

If the fourth attempt fails, or an attempt fails in a way that is not retried, the event is dead-lettered: it stays in the delivery log with its full payload, and an admin can send it again with Replay once the receiver is fixed.

Log Amomic-Delivery-Attempt. Anything above 1 means an earlier attempt of this event failed — useful context when a duplicate turns up in your database.

What retries, and what stops #

Retried: a 429, any 5xx, a timeout, and anything that kept the request from getting an HTTP answer. Not retried: everything else that is not a 2xx, including redirects.

A 404 means the route does not exist; sending it again changes nothing. A 401 means your endpoint rejected the request, which will not change in eight seconds. Stopping at once keeps a misconfigured endpoint from being hammered.

Redirects fail quietly: a platform that redirects /hooks/amomic to /hooks/amomic/ turns every event into a dead letter. Register the exact final URL.

Fifteen failed events disables the endpoint #

Every endpoint keeps a count of events in a row that ended in failure.

• An event that is delivered — on any attempt — sets it back to 0.
• An event that ends in failure — four failed attempts, or one that is not retried — adds 1. Test pings and replays count too.

At 15 the endpoint is switched off. In the console it shows Disabled, with the line Disabled automatically after 15 consecutive failed deliveries. Fix the endpoint, then re-enable it.

The count is consecutive, so one success anywhere resets it. An endpoint that fails fourteen events, succeeds once, and fails fourteen more is never switched off.

The console does not show the running count. What it shows is Last and the status of the most recent finished event (Last 200, or Last 503 in amber), and the Disabled badge once it trips.

A disabled endpoint receives nothing #

While an endpoint is Disabled — switched off automatically or with Disable — no new events are queued for it, and nothing is buffered for later. Retries already waiting for it are dropped too: those deliveries stay queued in the log and are not sent. Other endpoints in the workspace are unaffected.

So events from the disabled period have no delivery to replay. Catch them up through the REST API — see After an outage.

Re-enabling #

The delivery log #

Click Deliveries on an endpoint to open Recent deliveries: the newest 50 delivery rows for that endpoint. Rows are kept for 7 days.

Every attempt is its own row. The ticket.created above failed with a timeout, then a 502, then succeeded on attempt 3: three rows with the same event id, newest first.

The list does not refresh by itself; close and reopen it to see new rows. It does not show which row was the final one; a ✗ row with no newer row for the same event id, or a 4xx or redirect status, is a dead letter.

Replaying a delivery #

Every row that has been sent — failed or delivered — has a Replay button. It queues the stored event again for the same endpoint and says Webhook redelivery queued.

• The replay is the same event: same id, same type, same data. It gets a fresh signature and timestamp, and Amomic-Delivery-Attempt starts again at 1.
• Because the id is the same, a receiver that deduplicates will acknowledge and skip an event it already processed. That is the point: replaying everything after an outage is safe.
• A replay has its own retry ladder and its own new rows. The original rows are not changed.
• Replay only reaches an enabled endpoint. On a disabled one the replay stays queued.
• Rows older than 7 days are gone and can't be replayed. A very old row that predates stored payloads answers This older delivery did not retain a replayable payload.

There is no bulk replay and no "resend everything since". Replay row by row, or reconcile through the API.

Build a receiver that survives #

1. Deduplicate on the event id #

Delivery is at-least-once. An attempt that times out at 10 seconds and succeeds on your side at 11 seconds is retried, and both copies are real. A Replay sends the same event again.

Make the check the first thing your handler does, and make it atomic:

-- The whole mechanism. Zero rows affected means you have seen this event.
INSERT INTO amomic_events (event_id, event_type, created_at, received_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (event_id) DO NOTHING;

• Key on the envelope's top-level id, the same value as the Amomic-Event-Id header. Not on data.event_id, which is a different, internal id.
• Insert before you do the work, not after. If you process first and record second, a crash in between gives you a duplicate on the retry.
• Keep ids for at least 7 days. Retries finish within a minute, but an admin can replay a delivery for as long as the log keeps it.

Deduplication is not a substitute for order-safe business logic. Events arrive unordered, so compare the envelope's created_at against what you have stored and let the newer one win.

2. Acknowledge fast, work later #

The 10-second timeout is an acknowledgement budget, not a work budget. Inside the request handler, do only this:

1. Read the raw bytes.
2. Verify the signature. Fail → 400.
3. Insert the event id. Already present → 200, stop.
4. Write the event to a durable queue. Fail → 500.
5. Return 200.

Everything else — the CRM write, the email, the API call back to Amomic-AI — happens in a worker. A handler that does real work inline will, on its slowest day, cross 10 seconds while succeeding: the attempt is cut off, counted as failed, retried, and processed twice.

3. Return the right status code #

app.post("/hooks/amomic", express.raw({ type: "application/json" }), async (req, res) => {
  // Unfixable by retry — and the ONLY case that deserves a 4xx.
  if (!verify(req.body, req.get("Amomic-Signature"), SECRET)) {
    return res.status(400).send("bad signature");
  }

  let event;
  try {
    event = JSON.parse(req.body.toString("utf8"));
  } catch (err) {
    // Tempting to return 400 here. Do not — that skips every retry.
    return res.status(500).send("parse failed");
  }

  try {
    const isNew = await recordEventId(event.id); // INSERT … ON CONFLICT DO NOTHING
    if (isNew) await queue.enqueue(event.id, event);
  } catch (err) {
    return res.status(500).send("could not enqueue"); // buys more attempts
  }

  return res.status(200).send("ok");
});

4. Monitor your own endpoint #

The automatic switch-off is a circuit breaker, not an alert. Watch:

• Events received per hour. Alert on absence: zero events in a window where you normally see traffic. This catches a disabled endpoint, a DNS change and an expired certificate — none of which produce an error in your logs, because nothing reaches them.
• Amomic-Delivery-Attempt above 1. A rising rate means you are failing first attempts.
• Your handler's p99 latency against the 10-second budget. Alert at 3 seconds.
• The endpoint in the console. A Last status in amber means the most recent event ended in failure.

After an outage: replay and reconcile #

What you can recover through the API #

The pattern: anything with a read endpoint is recoverable, anything without one is not.

Designing so the gaps do not hurt #

Store the id the moment you create something. The call_id from POST /calls and the ticket_id from POST /tickets are the only handles you will get. Write them to your own database in the same transaction that decided to make the call.

Treat message.status as enrichment, not truth. The durable fact is the 201 from POST /messages. Build reporting so a missing delivered reads as "unknown", not "failed".

Run the conversation sync anyway. Webhooks for reacting, a scheduled walk of GET /conversations for completeness. If that job keeps a watermark, most of an outage repairs itself on the next run. Sync conversations has the job.

The runbook #

Related #