SRE knowledge workflow
How to Build an SRE Incident Knowledge Base
Turn runbooks, postmortems, and approved incident notes into current, source-backed operational knowledge for on-call engineers.
Article packet
Workflows
SRE, platform engineering, and on-call teams maintaining runbooks and post-incident knowledge
9 min read
01
Keep one service, environment, and incident class inside a clear source boundary.
02
Trace each diagnostic step, mitigation, and verification signal to a current runbook or postmortem.
03
Review stale steps after every incident instead of letting old operational advice stay silently active.
Inspect the source and review state before trusting a runbook
This genuine Wenlan desktop capture comes from the app's deterministic test fixture, not an incident or customer workspace. It shows maintained Pages with source counts and a review queue, the same product surfaces an SRE can use to keep source changes and unresolved conflicts visible.
- 01
Bound the incident knowledge
Select one service, environment, incident class, and approved source set instead of importing an unrestricted operational archive.
- 02
Distill a reviewable runbook Page
Keep symptoms, evidence, read-only checks, mitigations, risks, abort conditions, and verification signals attached to current sources.
- 03
Review after change or incident
Recheck the cited revision, stale state, and unresolved conflicts, then test the procedure in an approved non-production exercise.
Worked incident knowledge packet
This example defines the fields a reviewer should inspect. Replace every row with approved evidence for one service and incident class.
- Evidence input
- A current runbook, dated postmortem, architecture note, or sanitized incident summary with owner and revision.
- Operational claim
- One symptom, diagnostic step, mitigation boundary, or verification signal linked to the exact supporting passage.
- Review result
- Current, contradicted, stale, unsafe, or unresolved, with the service version, environment, reviewer, and next test recorded.
01
Quick answer
Build one incident knowledge base per service or operational boundary. Register approved runbooks, postmortems, architecture notes, and incident summaries; then maintain one reviewable Page for the incident class with symptoms, preconditions, diagnostic steps, mitigation limits, abort conditions, verification signals, owner, and review date.
Wenlan can keep supported documents connected to source-backed Pages, citations, revisions, stale state, lint, and human review. It does not monitor production, receive alerts, ingest live telemetry, execute runbooks, approve changes, or replace an incident-management system.
02
When this problem appears
The failure appears when an engineer is paged for a problem the team has seen before, but the useful context is split across an old runbook, a postmortem, chat history, architecture notes, and somebody's memory. A plausible AI summary can make this worse if it hides the service version, environment, source revision, risk, or evidence behind a command.
03
Build one incident-to-runbook review loop
Start after the incident is stable. Use sanitized, approved documents and keep live credentials, customer data, raw secrets, and unreviewed production commands outside the knowledge base.
- Choose one service, environment, and incident class. Record what is included, what stays in the monitoring or incident system, and who owns the operational knowledge.
- Register the current runbook, relevant postmortem, architecture note, and sanitized incident summary with their dates, versions, owners, and source locations.
- Create one maintained Page with symptoms, scope, preconditions, read-only checks, mitigation boundaries, escalation triggers, abort conditions, verification signals, and the next review date.
- Keep observed facts, hypotheses, mitigations, and confirmed causes separate. Do not rewrite a postmortem into a single neat cause when the evidence shows several contributing factors.
- After an incident or system change, resync the approved sources and mark affected steps stale until an owner verifies them against the current service and environment.
- Run a tabletop or staging exercise. Record which steps were reproducible, which links or commands were outdated, and what evidence was missing before the page is trusted for on-call use.
- Open the exact source before acting. A retrieved runbook is context for a qualified engineer, not permission to execute a production change.
After Wenlan and the AI client are configured
wenlan status
wenlan sources add ~/Ops/approved-incident-knowledge
# In a Wenlan plugin client:
/distill <service and incident class>
/pages <service incident runbook>
/lint
/curate04
What to check next
Keep monitoring, alert routing, telemetry, incident command, credentials, change approvals, rollback execution, and emergency access in their purpose-built systems. Wenlan does not validate a command against production or guarantee that a runbook is safe. A named owner must review service version, environment, permissions, risk, expected output, abort conditions, and rollback before operational use.
Make one incident lesson reproducible
Choose one service and incident class, connect the current runbook and postmortem, then review every operational claim before the next on-call shift.
FAQ