How to write an incident postmortem (with a template)

An incident postmortem is a written record of one incident. It says what happened, what it cost, why it happened and what you will change so the same failure does not come back. The people who handled the incident write it after service is restored.

It is not a status update. A status update tells customers what is happening now. A postmortem comes later, with the full facts, and its main readers are your own team. Some teams call it a post-incident review.

When an incident deserves one

Set fixed triggers so the decision is not argued each time. A reasonable starting list:

• Customers could not use a core feature for more than 15 minutes.
• Any data was lost, corrupted or shown to the wrong person.
• You missed a promised uptime figure. See how uptime SLAs work.
• Someone had to step in by hand: a rollback, a failover, a restore.
• A near miss, where nothing broke for customers but only by luck.

Skip it for a five-minute blip with a known cause and a fix already shipped. A postmortem costs about half a working day to write, so spend that where there is something to learn.

Blameless means the document explains every mistake by looking at the system around the person, not at the person. "The system" means the tools, checks, documents, alerts and time pressure that were in place when the decision was made.

The reason is practical. People who expect to be punished leave details out. Then the timeline is wrong, the cause you find is wrong and the fix does not work.

What it does mean

• You describe actions and what was known at the time.
• You ask why the action looked reasonable, and what would have stopped it.
• You assume anyone on the team, with the same tools and information, could have done the same.

What it does not mean

• It does not mean nobody owns the fix. Every action still has one named owner and a date.
• It does not mean hiding who did what. The timeline can name people. It does not judge them.
• It does not mean faults go unnamed. "The rollback steps were out of date" names a fault in a document, and a document can be fixed.
• It does not cover deliberate harm. Management handles that outside the postmortem.

A quick test for each sentence: if a different engineer had been on shift, could this still have happened? If yes, say what in the system allowed it.

One person writes it: the engineer who led the response. The first draft should exist within two working days, and the review meeting should happen within five.

Support adds the customer side: ticket counts and when the first report arrived. The author of the triggering change explains what it was meant to do. One person who was not involved reads the draft and marks anything unclear.

Why the deadline is short

Chat logs survive. Reasons do not. A week later nobody remembers why a step looked right at the time, and that reasoning is what you most need. Logging and metrics tools also keep detailed data for a limited time, so export graphs and logs on the day.

In this guide's example the incident was on Tuesday 6 October 2026. The draft was done on Thursday 8 October, the review was on Friday 9 October, and the action check was booked for Tuesday 20 October.

Start with a summary of three to five sentences, then state the impact in numbers. A reader who stops after these two sections should know what broke, for how long, who was affected and whether it is fixed.

Every example in this guide comes from one made-up incident. Fernhill is an imaginary online shop. On Tuesday 6 October 2026 its checkout returned errors for 47 minutes, after a deploy used up every available database connection.

Summary

Write it last, for a reader outside engineering. Say what customers saw, then the cause, then the fix.

Summary
On Tuesday 6 October 2026, from 14:11 to 14:58 UTC (47 minutes), most
attempts to pay on fernhill.example failed with a 502 error. A deploy of
checkout-api 2.41.0 at 14:02 UTC raised the number of worker processes,
and the extra workers opened more database connections than the database
allows. We rolled back to 2.40.3 and checkout recovered at 14:58 UTC.
Browsing, search and login were not affected.

A 502 (Bad Gateway) is what a load balancer returns when the server behind it gives no valid answer. See RFC 9110 or the guide to HTTP status codes.

Impact

Impact is what the incident cost customers and the business. Use counts and durations, and say where each number came from. "Some users had problems" is not an impact statement.

Impact
- Duration: 47 minutes (14:11 to 14:58 UTC).
- What failed: POST /checkout returned 502.
- Orders: 26 completed. The same 47 minutes on the previous four
  Tuesdays averaged 376. About 350 orders (93%) were not placed.
- Revenue: about 15,400 GBP not taken (350 orders x 44 GBP average).
- Support: 41 tickets about failed payments.
- Data: no orders lost. Nobody charged for a failed order.
- SLA: October has 44,640 minutes. A 99.9% target allows 44.6 minutes
  of downtime. Checkout has used 47, so the target is missed.

If you cannot measure something, give your best estimate and the method, as the order figure above does. The downtime cost calculator does this arithmetic for your own service.

The timeline is a list of facts, each with a timestamp in UTC, in the order they happened. The detection section then answers two questions: how did you find out, and how long did that take?

Timeline

Use UTC because server logs are usually in UTC, team members may be in different time zones, and local time shifts twice a year.

Include what the system did, which alerts fired, what people did and what they decided. Include the things that were tried and did not work. Leave opinions for later sections.

Build it from sources, not from memory: the chat export, the deploy log, the alert history, the load balancer logs.

Detection

Detection
- Impact started: 14:11 UTC.
- We found out: 14:26 UTC, from a customer support ticket.
- Time to detect: 15 minutes.
- Why it took that long: our only uptime monitor checks the home page,
  which does not use the database path that checkout uses. We had no
  alert on database connections or on the 502 rate.

If a customer told you before your own monitoring did, write that down plainly. The fix is cheap: a check on the thing that broke.

The root cause is the condition that, if you removed it, would have prevented the incident. Contributing factors are the things that made it more likely, longer or harder to spot.

State the root cause as a fact about the system. "Human error" is never a root cause.

Root cause
checkout-api 2.41.0 raised workers per instance from 4 to 12. Each worker
keeps its own pool of up to 10 database connections.

  Before: 6 instances x  4 workers x 10 = 240 connections at most
  After:  6 instances x 12 workers x 10 = 720 connections at most

The database allows 500 connections (max_connections = 500). Workers
open connections only when they need them, so the count rose with
afternoon traffic and reached 500 at 14:11 UTC, nine minutes after the
deploy. New connections were then refused, workers failed, and the load
balancer returned 502.

Contributing factors
- Staging runs 1 instance (120 connections at most), so the change
  passed there.
- No alert on database connection count.
- The uptime monitor covered the home page only.
- The rollback steps in the runbook named a script removed in March.

The five whys

State the problem and ask why it happened. Then ask why of the answer, and keep going. Five is a guide, not a rule. Stop when you reach something you can change. If an answer is a person's name, ask why the system let that happen.

1. Why did checkout return 502? The checkout-api workers could not open a database connection, so the load balancer had no valid response to pass on.
2. Why could they not open a connection? The database was at its limit of 500 connections.
3. Why was it at the limit? Version 2.41.0 tripled the workers per instance, which raised the most connections checkout-api could open from 240 to 720.
4. Why did nobody see that 720 is more than 500? The worker count, the pool size and the database limit are set in three different places. Nothing in review or CI multiplies them together.
5. Why did testing not catch it? Staging has 1 instance, not 6, so it can never reach the limit.

The useful actions come from whys 4 and 5. Stopping at why 3 would have produced "be careful when changing worker counts", which changes nothing.

The method follows one chain. Run it again from another question, such as "why did it take 15 minutes to notice?".

List what worked, list what did not, then turn each thing that did not work into an action with one owner and one due date. An action with no owner or no date is a wish.

What went well

This section records what to keep, so you do not lose it by accident.

• Sam acknowledged 3 minutes after the incident was raised.
• The log line "too many clients already" pointed straight at the cause.
• The rollback worked first time and took 6 minutes.

What went badly

• A customer found the problem before we did. Detection took 15 minutes.
• Restarting one instance at 14:37 used 4 minutes and could not have helped.
• Finding the rollback command took 3 minutes because the runbook was out of date.

Action items

A good action is specific enough that a stranger could tell whether it is done. Prefer changes to the system (a check, an alert, a limit) over changes to behaviour (a reminder, a training session).

Five or six actions is about the limit. Put the rest in a "considered, not doing" list with a reason for each.

Yes. Copy the block below and fill in every heading. If a section does not apply, write "Not applicable" and why, so readers know it was considered.

POSTMORTEM: [short title that says what customers saw]

Date of incident:  [YYYY-MM-DD]
Author:            [name]
Review meeting:    [YYYY-MM-DD]
Status:            [draft | reviewed | actions complete]

1. SUMMARY
[3 to 5 sentences: what customers saw, when, for how long, the cause,
the fix. Readable by someone outside engineering.]

2. IMPACT
- Duration: [N] minutes, from [HH:MM] to [HH:MM] UTC
- Who was affected: [which customers, how many, what share]
- What they could not do: [feature]
- Business cost: [orders, revenue, credits owed, support tickets]
- Data: [lost | exposed | none]
- How these numbers were worked out: [method and source]

3. TIMELINE (all times UTC)
[HH:MM]  [system event, alert, action or decision]
Mark: impact starts, detected, cause identified, impact ends.

4. DETECTION
- We found out at [HH:MM], from [alert | customer | staff]
- Time to detect: [N] min. To acknowledge: [N] min. To resolve: [N] min.
- Why detection was not faster: [...]

5. ROOT CAUSE
[A fact about the system, not about a person.]

6. CONTRIBUTING FACTORS
- [What made it more likely, longer or harder to spot]

7. FIVE WHYS
1. Why [problem]? Because [...]
2. Why [...]? Because [...]

8. WHAT WENT WELL
- [Things to keep]

9. WHAT WENT BADLY
- [Things that slowed detection, diagnosis or recovery]

10. ACTION ITEMS
#  | Action                 | Owner  | Due    | Status
1  | [specific, checkable]  | [name] | [date] | [open]

11. CONSIDERED, NOT DOING
- [Idea]: [reason]

12. FOLLOW-UP
- Action review date: [YYYY-MM-DD]
- Public version: [link, or "not needed" and why]

Keep every postmortem in one place the whole company can search. If you would rather not start from a blank page, incidents in PULSX include a postmortem write-up with a timeline and starter templates. It is one of the free features.

Weak postmortems fail in the same few ways: they blame a person, they list vague actions, and nobody follows up.

Track three times for every incident: time to detect, time to acknowledge and time to resolve. Each measures a different part of the response, and each has a different fix.

Agree the definitions once and write them down. Some teams count time to resolve from detection, which for Fernhill gives 32 minutes, not 47. Neither is wrong, but mixing the two makes the numbers useless.

Averaged over many incidents these become MTTD, MTTA and MTTR. Take care with MTTR: the R is read as resolve, repair, recover or respond, and those are different measurements.

How to use the numbers

Split the incident into stages and see where the minutes went. For Fernhill: 15 minutes when nobody knew, 3 waiting for a responder, 12 finding the cause, 8 deciding on and preparing the rollback, and 9 rolling back. That adds up to 47.

The largest stage is the first. A check on the checkout URL every 60 seconds would have removed most of it, which is why action 1 comes first.

Do not turn these into targets for individual people. That brings blame back in.

Publish a public version when customers noticed the incident. Write the internal version first, in full, then cut the public one from it. The public version is shorter, uses no internal names and promises only what you are certain to do.

Post it where you posted the incident updates, which for most teams is the status page. The status page guide covers how to write those updates.

Checkout errors on 6 October 2026

What happened
On Tuesday 6 October, between 14:11 and 14:58 UTC, most attempts to pay
on our site failed with an error. Browsing and logging in still worked.

Why it happened
A software update made our checkout service open more connections to
our database than the database accepts. Our testing did not catch this
because our test system is smaller than the live one.

What we did
We reversed the update and checkout recovered at 14:58 UTC. No order was
lost and nobody was charged for an order that failed.

What we are changing
- Checkout now has its own monitoring, checked every minute.
- By 16 October our release process will check this limit automatically.

We are sorry. If you need help with an order, write to
[email protected].

Two rules keep it honest. The public version must never disagree with the internal one: it says less, not something different. And use the present tense only for changes that are already live. Everything else gets a date.

Hold one meeting of about 50 minutes, within five working days, with everyone who took part in the response. Its job is to check the document is right and to agree the actions, not to write the document.

Send the draft 24 hours ahead. Pick a facilitator who was not a responder, so the author is free to answer questions.

1. Summary and impact (5 minutes). Confirm the numbers.
2. Timeline (15 minutes). At each decision, ask: what did you see, and what did you expect to happen?
3. Root cause and five whys (10 minutes). Challenge each "because".
4. Actions (15 minutes). Cut the list, then give each action an owner and a date.
5. Wrap up (5 minutes). Is a public version needed? Book the action check.

Ask "what" and "how" questions. "What made the restart look like the right step?" gets a useful answer. "Why didn't you check the logs first?" gets a defensive one. The facilitator stops blame when it appears and restates the point as a fact about the system.

Making sure actions get done

Most postmortems fail here: three months later the same incident happens with the actions still open. Five habits prevent that.

1. Put each action in your normal work tracker. An action that lives only in the document will be forgotten.
2. Agree owners in the meeting. Do not assign an action to someone who is not in the room.
3. Book the action check before the meeting ends. Ten minutes, about two weeks out. For Fernhill that was Tuesday 20 October.
4. At the check, each action gets one of three results: done, moved to a new date with a reason, or dropped with a reason.
5. Prove each action works. The alert at 400 connections is done when someone has triggered it in staging and seen it arrive.