# Retries & failures

> Four attempts over about a quarter of a minute, what stops a retry, the delivery log in the console, Replay, the automatic switch-off after 15 failed events, and catching up after an outage.

Source: https://docs.amomic.in/webhooks/reliability

---

Your endpoint will fail. A deploy will restart it mid-request, a database will lock, a certificate will expire on a Sunday. What happens next depends on the status code you return and on how many events in a row go wrong.

This page is the delivery contract from the failure side: the retry ladder, what stops it, the delivery log you can read in the console, **Replay**, the counter that switches your endpoint off, and what you can recover once an outage is over. Owners and admins manage all of it in **Developers → Webhooks**; [The Developers page](/console/developers#webhooks) shows every screen.

## The retry ladder

Each event is written to the delivery log as `queued` before it is sent, and then delivered from a durable queue: a restart or a new release on our side does not lose it. A failed attempt that is worth retrying is tried again, up to **four attempts** in total.

```mock
{"component": "RetryDiagram", "props": {"caption": "Four attempts, the waits doubling from about 2 seconds. A final failure stays in the delivery log, where Replay can send it again."}}
```

| Attempt | Sent | `Amomic-Delivery-Attempt` |
|---|---|---|
| 1 | Straight away | `1` |
| 2 | About 2 seconds after attempt 1 failed | `2` |
| 3 | About 4 seconds after attempt 2 failed | `3` |
| 4 | About 8 seconds after attempt 3 failed | `4` |

So the waits add up to about **14 seconds**. Each attempt can itself take up to **10 seconds** (5 to connect), so on an endpoint that hangs, the whole ladder is over in under a minute. Treat the spacing as approximate; build nothing that depends on the exact seconds.

If the fourth attempt fails, or an attempt fails in a way that is not retried, the event is **dead-lettered**: it stays in the delivery log with its full payload, and an admin can send it again with **Replay** once the receiver is fixed.

Log `Amomic-Delivery-Attempt`. Anything above `1` means an earlier attempt of this event failed — useful context when a duplicate turns up in your database.

## What retries, and what stops

**Retried:** a `429`, any `5xx`, a timeout, and anything that kept the request from getting an HTTP answer. **Not retried:** everything else that is not a `2xx`, including redirects.

| Your response | Retried | Why |
|---|---|---|
| `200`–`299` | **No** — delivered | The endpoint's failure count goes back to zero. |
| `429 Too Many Requests` | **Yes** | You are asking for a slower pace, not saying the request is wrong. |
| `500`, `502`, `503`, `504` | **Yes** | Your server broke. The same request may work seconds later. |
| Connection refused, DNS failure, TLS error | **Yes** | Nothing reached your application. |
| Timeout — no answer within 10 seconds | **Yes** | It is unknown whether you processed it, so it is sent again. |
| `301`, `302`, `307` and other redirects | **No** | Redirects are not followed, and a redirect is not retried. The event is dead-lettered on the first attempt. |
| `400`, `401`, `403`, `404`, `410`, `422` and other `4xx` | **No** | A 4xx says the request itself is wrong. The event is dead-lettered on the first attempt. |

A `404` means the route does not exist; sending it again changes nothing. A `401` means your endpoint rejected the request, which will not change in eight seconds. Stopping at once keeps a misconfigured endpoint from being hammered.

::: danger Never return a 4xx for a problem you could recover from
A `400` returned because your queue was full or a dependency was briefly down skips every automatic retry. The event is kept, but it waits for someone to press **Replay**.

- Return **`400`** for exactly one thing: the signature did not verify.
- Return **`500`** for everything else that went wrong — a parse failure, a full queue, a dead database, an unexpected exception. A `500` buys three more attempts.

If you are unsure, return `500`. An extra retry costs a duplicate you already deduplicate; a wrong `400` costs a manual replay.
:::

Redirects fail quietly: a platform that redirects `/hooks/amomic` to `/hooks/amomic/` turns every event into a dead letter. Register the exact final URL.

## Fifteen failed events disables the endpoint

Every endpoint keeps a count of **events in a row that ended in failure**.

- An event that is delivered — on any attempt — sets it back to **0**.
- An event that ends in failure — four failed attempts, or one that is not retried — adds **1**. Test pings and replays count too.

At **15** the endpoint is switched off. In the console it shows **Disabled**, with the line *Disabled automatically after 15 consecutive failed deliveries. Fix the endpoint, then re-enable it.*

![The Webhooks tab with two enabled endpoints showing Last 200, and a disabled endpoint with Last 503 and a note that it was disabled automatically after 15 consecutive failed deliveries](https://docs.amomic.in/screens/developers/webhooks.light.webp)

::: warning The count is per event, not per attempt
Fifteen failed **events**, each of which may have used four attempts — up to sixty requests before anything is switched off. And the count measures events, not time: a busy workspace trips it within minutes, a quiet one can take days. Don't rely on the switch-off as your monitoring.
:::

The count is consecutive, so one success anywhere resets it. An endpoint that fails fourteen events, succeeds once, and fails fourteen more is never switched off.

The console does not show the running count. What it shows is **Last** and the status of the most recent finished event (`Last 200`, or `Last 503` in amber), and the **Disabled** badge once it trips.

### A disabled endpoint receives nothing

While an endpoint is **Disabled** — switched off automatically or with **Disable** — no new events are queued for it, and nothing is buffered for later. Retries already waiting for it are dropped too: those deliveries stay `queued` in the log and are not sent. Other endpoints in the workspace are unaffected.

So events from the disabled period have no delivery to replay. Catch them up through the REST API — see [After an outage](#after-an-outage-replay-and-reconcile).

### Re-enabling

::: steps

::: step Fix the endpoint first

Re-enabling a still-broken endpoint spends another fifteen events. Check your server's logs, certificate, routing and the URL itself.
:::

::: step Click Enable

On the endpoint in **Developers → Webhooks**, click **Enable**. The failure count goes back to zero and the disabled message clears. URL, signing secret and events are unchanged.
:::

::: step Prove it with Test

Click **Test**, then open **Deliveries** and look for the `ping` with a `2xx`. A test on a disabled endpoint is never sent, which is why **Enable** comes first.
:::

::: step Replay and reconcile

**Replay** the dead-lettered deliveries from before the switch-off, and catch up the disabled period through the REST API — see [below](#after-an-outage-replay-and-reconcile).
:::

:::

## The delivery log

Click **Deliveries** on an endpoint to open **Recent deliveries**: the newest 50 delivery rows for that endpoint. Rows are kept for 7 days.

![Recent deliveries: a queued ping, a message.received delivered with 200 and its payload expanded, one ticket.created event tried three times — no response, then 502, then 200 on attempt 3 — a lead.created refused with 404, and a call.outcome delivered with 200](https://docs.amomic.in/screens/webhooks/deliveries.light.webp)

**Every attempt is its own row.** The `ticket.created` above failed with a timeout, then a `502`, then succeeded on attempt 3: three rows with the same event id, newest first.

| What you see | Meaning |
|---|---|
| ↻ and `queued` | Waiting to be sent. A row that stays `queued` belongs to an endpoint that is disabled or was deleted. |
| ✓ and a status such as `200` | Your server accepted that attempt. |
| ✗ and a status, or `no response` | That attempt failed. `no response` means there was no HTTP answer at all — a timeout, refused connection or TLS failure. |
| Red text | The first 200 characters of your server's response body, or the error for a transport failure. |
| `attempt 2`, `attempt 3`, `attempt 4` | A retry of the same event. Shown only from attempt 2. |
| Milliseconds on the right | How long that attempt took. |
| The `evt_…` line | The event's `id` — the same value as `Amomic-Event-Id` and your dedupe key. Match it against your own logs. |
| **Payload** | Expands to the full event exactly as it was sent, whatever its size. |
| **Replay** | Sends the event again. See below. |

The list does not refresh by itself; close and reopen it to see new rows. It does not show which row was the final one; a ✗ row with no newer row for the same event id, or a `4xx` or redirect status, is a dead letter.

## Replaying a delivery

Every row that has been sent — failed or delivered — has a **Replay** button. It queues the stored event again for the same endpoint and says `Webhook redelivery queued`.

- The replay is the **same event**: same `id`, same `type`, same `data`. It gets a fresh signature and timestamp, and `Amomic-Delivery-Attempt` starts again at `1`.
- Because the `id` is the same, a receiver that deduplicates will **acknowledge and skip** an event it already processed. That is the point: replaying everything after an outage is safe.
- A replay has its own retry ladder and its own new rows. The original rows are not changed.
- Replay only reaches an **enabled** endpoint. On a disabled one the replay stays `queued`.
- Rows older than 7 days are gone and can't be replayed. A very old row that predates stored payloads answers `This older delivery did not retain a replayable payload.`

There is no bulk replay and no "resend everything since". Replay row by row, or reconcile through the API.

## Build a receiver that survives

### 1. Deduplicate on the event id

Delivery is at-least-once. An attempt that times out at 10 seconds and succeeds on your side at 11 seconds is retried, and both copies are real. A Replay sends the same event again.

Make the check the first thing your handler does, and make it atomic:

```sql
-- The whole mechanism. Zero rows affected means you have seen this event.
INSERT INTO amomic_events (event_id, event_type, created_at, received_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (event_id) DO NOTHING;
```

- **Key on the envelope's top-level `id`**, the same value as the `Amomic-Event-Id` header. Not on `data.event_id`, which is a different, internal id.
- **Insert before you do the work, not after.** If you process first and record second, a crash in between gives you a duplicate on the retry.
- **Keep ids for at least 7 days.** Retries finish within a minute, but an admin can replay a delivery for as long as the log keeps it.

Deduplication is not a substitute for order-safe business logic. Events arrive unordered, so compare the envelope's `created_at` against what you have stored and let the newer one win.

### 2. Acknowledge fast, work later

The 10-second timeout is an **acknowledgement budget, not a work budget**. Inside the request handler, do only this:

1. Read the raw bytes.
2. Verify the signature. Fail → `400`.
3. Insert the event id. Already present → `200`, stop.
4. Write the event to a durable queue. Fail → `500`.
5. Return `200`.

Everything else — the CRM write, the email, the API call back to Amomic-AI — happens in a worker. A handler that does real work inline will, on its slowest day, cross 10 seconds while succeeding: the attempt is cut off, counted as failed, retried, and processed twice.

### 3. Return the right status code

```javascript
app.post("/hooks/amomic", express.raw({ type: "application/json" }), async (req, res) => {
  // Unfixable by retry — and the ONLY case that deserves a 4xx.
  if (!verify(req.body, req.get("Amomic-Signature"), SECRET)) {
    return res.status(400).send("bad signature");
  }

  let event;
  try {
    event = JSON.parse(req.body.toString("utf8"));
  } catch (err) {
    // Tempting to return 400 here. Do not — that skips every retry.
    return res.status(500).send("parse failed");
  }

  try {
    const isNew = await recordEventId(event.id); // INSERT … ON CONFLICT DO NOTHING
    if (isNew) await queue.enqueue(event.id, event);
  } catch (err) {
    return res.status(500).send("could not enqueue"); // buys more attempts
  }

  return res.status(200).send("ok");
});
```

### 4. Monitor your own endpoint

The automatic switch-off is a circuit breaker, not an alert. Watch:

- **Events received per hour.** Alert on *absence*: zero events in a window where you normally see traffic. This catches a disabled endpoint, a DNS change and an expired certificate — none of which produce an error in your logs, because nothing reaches them.
- **`Amomic-Delivery-Attempt` above 1.** A rising rate means you are failing first attempts.
- **Your handler's p99 latency** against the 10-second budget. Alert at 3 seconds.
- **The endpoint in the console.** A `Last` status in amber means the most recent event ended in failure.

## After an outage: replay and reconcile

::: warning Replay covers recorded deliveries, not time ranges
Dead-lettered events from before the switch-off are in **Recent deliveries** — replay them. Events from while the endpoint was **Disabled** were never queued for it, so there is nothing to replay; catch those up through the REST API. The log shows the newest 50 rows only, so an endpoint that failed hundreds of events can't be fully replayed from the console either.
:::

### What you can recover through the API

| Missed event | Recoverable? | How |
|---|---|---|
| `conversation.started` | **Yes** | Walk [`GET /conversations`](/api/conversations), newest activity first, and stop at your watermark. |
| `message.received`, `message.echo` | **Yes**, as transcript turns | For every conversation that moved, re-read [`GET /conversations/{id}/messages`](/api/conversations) and upsert on `message_id`. The WhatsApp `wamid` and any ad `referral` are not recoverable. |
| `call.ended`, `call.outcome` for calls **you placed** | **Yes** | You stored the `call_id` from [`POST /calls`](/api/calls). Read [`GET /calls/{call_id}`](/api/calls) — `outcome`, `duration_seconds` and `result`. |
| `call.started`, `call.ended`, `call.outcome` for **inbound** calls | **No** | There is no call list. The call may appear in `GET /conversations` with `channel: "voice"` — your only trace through the API. |
| `ticket.created`, `ticket.updated`, `ticket.replied` for tickets **you opened** | **Partly** | Read [`GET /tickets/{ticket_id}`](/api/tickets) for the current state. The reply's `delivery_state` is not on that record. |
| Ticket events for tickets opened by agents or staff | **No** | There is no ticket list endpoint. |
| `message.status` | **No** | Delivery receipts have no read side. |
| `lead.created` | **Yes**, as contacts | Walk [`GET /contacts`](/api/contacts). The capture time is not reconstructed. |
| `campaign.launched`, `campaign.completed` | **Yes** | [`GET /campaigns`](/api/campaigns) and `GET /campaigns/{id}`. |
| `payment.*` | **Yes**, as current state | [`GET /apps/duesly/dues`](/api/apps-duesly). |
| `appointment.*` | **No** | There is no appointments API. Check the console calendar. |

The pattern: **anything with a read endpoint is recoverable, anything without one is not.**

### Designing so the gaps do not hurt

**Store the id the moment you create something.** The `call_id` from `POST /calls` and the `ticket_id` from `POST /tickets` are the only handles you will get. Write them to your own database in the same transaction that decided to make the call.

**Treat `message.status` as enrichment, not truth.** The durable fact is the `201` from `POST /messages`. Build reporting so a missing `delivered` reads as "unknown", not "failed".

**Run the conversation sync anyway.** Webhooks for reacting, a scheduled walk of `GET /conversations` for completeness. If that job keeps a watermark, most of an outage repairs itself on the next run. [Sync conversations](/recipes/sync-conversations) has the job.

### The runbook

::: steps

::: step Confirm what broke

In **Developers → Webhooks**, check the badge (**Enabled** or **Disabled**), the `Last` status, and the disabled message. Open **Deliveries**: `no response` means no HTTP answer, a `5xx` means your app answered and failed, a `3xx` means a redirect you didn't know about.
:::

::: step Fix it, enable, test

Fix the receiver. Click **Enable** if the endpoint is disabled, then **Test**, and reopen **Deliveries** until the `ping` shows a `2xx`.
:::

::: step Establish the window

The time of the oldest dead-lettered row, and the last event id in your own database, bound the gap.
:::

::: step Replay, then reconcile forward

**Replay** the dead-lettered rows still in the log. Then walk `GET /conversations` from page one until you pass the start of the gap, re-read transcripts for anything that moved, and read `GET /calls/{id}` for calls you placed. Accept the rows above that can't be recovered, and log the gap.
:::

:::

## Related

::: cards

::: card Getting events
[The delivery contract](/webhooks/overview) — envelope, headers, and the rules your endpoint has to follow.
:::

::: card Event reference
[All twenty event types](/webhooks/events) — exact fields, and the `ping` your endpoint must tolerate.
:::

::: card Verifying signatures
[The one 4xx worth returning](/webhooks/signatures) — HMAC format, tolerance window, raw bytes.
:::

::: card Sync conversations
[The reconciliation job](/recipes/sync-conversations) — the scheduled walk that repairs most of an outage.
:::

:::