SRE knowledge workflow
How to Build an SRE Incident Knowledge Base
Turn runbooks, postmortems, and approved incident notes into current, source-backed operational knowledge for on-call engineers.
About this guide
Workflows
SRE, platform engineering, and on-call teams maintaining runbooks and post-incident knowledge
9 min read
01
Keep one service, environment, and incident class inside a clear source boundary.
02
Trace each diagnostic step, mitigation, and verification signal to a current runbook or postmortem.
03
Review stale steps after every incident instead of letting old operational advice stay silently active.
Worked example
Build an SRE incident knowledge base
A fictional tabletop exercise in three files: triage the backlog with read-only observations, respect the abort limits, and keep causation open.
Try this task
Using only the three source files, triage the observed backlog, propose a read-only observation plan with its time and sampling limits, and separate hypotheses from decisions that require a human incident owner. Cite each conclusion with [[filename]].
Expected reasoning
Record the 8.6% 5xx, 1.8-second p95, and queue depth 420 as observations; stay within the 15-minute, three-view, 20-row guardrails; abort on mutation or exposed sensitive data; and treat the 09:05 deployment as a correlation hypothesis rather than a proven cause.
Why keep this connected
A single tabletop packet can be reviewed by hand. A source-linked incident wiki becomes useful when observations, hypotheses, abort rules, and later timeline corrections must remain connected without turning notes into operational authority.
Review boundary
Check the observed values, 15-minute/three-view/20-row limits, abort triggers, and event timing; leave escalation, rollback, and ownership to the incident owner.
Inspect the example in Wenlan
Wenlan v0.18.3 interface displaying data read back from an isolated test run. The sources are fictional and the reference answer was written for this exercise. No automatic AI generation or approval is shown.
After the cited source changed, Wenlan marked the page as out of date and kept its original text. The ‘updating…’ label indicates a pending rebuild here; it does not show a completed correction. Review the changed source before rebuilding and accepting a new answer.
Source files
Open a file to read its complete authored Markdown.
The source files are the same authored English dataset in all three locale views.
01incident-observations.mdNorthstar API — tabletop observations
# Northstar API — tabletop observations Status: fictional teaching observation sheet; no live service or customer data is involved. At 09:10 UTC, the alert window showed 8.6% HTTP 5xx responses and 1.8 seconds p95 latency for the Northstar API. The normal reference window in this exercise is 0.4% 5xx and 350 milliseconds p95. Queue depth was 420 items; the reference value is 40. These are observations, not a root-cause finding. This packet contains no customer identifiers, raw logs, or evidence of a particular deployment causing the change.
02read-only-triage.mdRead-only tabletop triage guardrails
# Read-only tabletop triage guardrails The responder may inspect the incident dashboard, a redacted log sample, and the deployment timeline. The observation window is at most 15 minutes, with at most three views and 20 redacted log rows. Record timestamps and observed values; do not turn correlation into causation. Do not restart a service, change configuration, invalidate a cache, roll back a release, acknowledge a customer action, or write to an incident system. Abort immediately if a view requests a mutation, exposes an unredacted identifier or secret, or cannot show whether its data is fictional and redacted. A human incident owner decides whether to escalate, page another team, or authorize a change.
03deploy-timeline.mdNorthstar API — fictional deployment timeline
# Northstar API — fictional deployment timeline A cache-configuration deployment completed at 09:05 UTC. The incident alert began at 09:10 UTC. The timeline records those two times but does not contain a causal test, rollback approval, or owner decision. The five-minute ordering makes the deployment a hypothesis to compare against read-only observations. It does not prove that the deployment caused the 5xx responses or latency. Rollback eligibility and the responsible owner are not specified in this packet.
Reference answer
Reference answer: observe within limits, preserve uncertainty
The exercise supports a bounded read-only backlog observation: compare the alert values with the reference window, inspect only the named redacted views, and abort on mutation, secrets, or unredacted identifiers. The 09:05 deployment is a temporal hypothesis, not a proven cause; escalation, rollback, and ownership remain human decisions.
Northstar API — triage reference
Observed state
The fictional alert window records 8.6% 5xx, 1.8 seconds p95 latency, and queue depth 420, compared with reference values of 0.4%, 350 milliseconds, and 40. These are observations and do not establish a root cause. incident-observations.md
Safe observation plan
A responder may inspect the incident dashboard, one redacted log sample, and the deployment timeline for at most 15 minutes, three views, and 20 redacted rows. The responder records timestamps and values only. Any requested mutation, unredacted identifier or secret, or unclear fictional/redaction status is an abort condition. read-only-triage.md
Hypothesis and human decisions
The cache-configuration deployment completed at 09:05 UTC, five minutes before the alert. That makes it a candidate correlation to compare, not proof of causation. The packet does not authorize a rollback, identify an owner, or decide whether to escalate; a human incident owner must make those decisions. deploy-timeline.md read-only-triage.md
Changed source: deploy-timeline.md
Expected update after this change
The reference's statement that the 09:05 deployment was a candidate cause becomes stale. The 09:05 event was only a review and the actual deployment followed the alert, so it cannot explain the initial onset. The read-only limits, observed values, and human decisions about escalation or rollback remain unchanged; the incident cause is still unknown.
# Northstar API — fictional deployment timeline, revision 2 A cache-configuration review was recorded at 09:05 UTC, but it made no production change. The cache-configuration deployment completed at 09:15 UTC, after the incident alert began at 09:10 UTC. The timeline still contains no causal test, rollback approval, or owner decision. The corrected ordering removes this deployment as an explanation for the alert onset. Other causes remain unknown. Rollback eligibility and the responsible owner are not specified in this packet.
01
Quick answer
Build one incident knowledge base per service or operational boundary. Register approved runbooks, postmortems, architecture notes, and incident summaries; then maintain one reviewable Page for the incident class with symptoms, preconditions, diagnostic steps, mitigation limits, abort conditions, verification signals, owner, and review date.
Wenlan can keep supported documents connected to source-backed Pages, citations, revisions, stale state, lint, and human review. It does not monitor production, receive alerts, ingest live telemetry, execute runbooks, approve changes, or replace an incident-management system.
02
When this problem appears
The failure appears when an engineer is paged for a problem the team has seen before, but the useful context is split across an old runbook, a postmortem, chat history, architecture notes, and somebody's memory. A plausible AI summary can make this worse if it hides the service version, environment, source revision, risk, or evidence behind a command.
03
Build one incident-to-runbook review loop
Start after the incident is stable. Use sanitized, approved documents and keep live credentials, customer data, raw secrets, and unreviewed production commands outside the knowledge base.
- Choose one service, environment, and incident class. Record what is included, what stays in the monitoring or incident system, and who owns the operational knowledge.
- Register the current runbook, relevant postmortem, architecture note, and sanitized incident summary with their dates, versions, owners, and source locations.
- Create one maintained Page with symptoms, scope, preconditions, read-only checks, mitigation boundaries, escalation triggers, abort conditions, verification signals, and the next review date.
- Keep observed facts, hypotheses, mitigations, and confirmed causes separate. Do not rewrite a postmortem into a single neat cause when the evidence shows several contributing factors.
- After an incident or system change, resync the approved sources and mark affected steps stale until an owner verifies them against the current service and environment.
- Run a tabletop or staging exercise. Record which steps were reproducible, which links or commands were outdated, and what evidence was missing before the page is trusted for on-call use.
- Open the exact source before acting. A retrieved runbook is context for a qualified engineer, not permission to execute a production change.
After Wenlan and the AI client are configured
wenlan status
wenlan sources add ~/Ops/approved-incident-knowledge
# In a Wenlan plugin client:
/distill <service and incident class>
/pages <service incident runbook>
/lint
/curate04
What to check next
Keep monitoring, alert routing, telemetry, incident command, credentials, change approvals, rollback execution, and emergency access in their purpose-built systems. Wenlan does not validate a command against production or guarantee that a runbook is safe. A named owner must review service version, environment, permissions, risk, expected output, abort conditions, and rollback before operational use.
Make one incident lesson reproducible
Choose one service and incident class, connect the current runbook and postmortem, then review every operational claim before the next on-call shift.
FAQ