Career roadmap
Site Reliability Engineer
Treat reliability as a product with a budget: measure it, spend it deliberately, and engineer away the toil.
Before you start SRE
- Production experience in some form
- One programming language you can write services in
- Linux and networking fundamentals
The reliability model
SRE is a set of ideas before it is a set of tools. This vocabulary is the interview.
The defining concept of the discipline, and the first thing any SRE interview probes.
- Choosing SLIs that reflect user experience
- Setting an SLO that is achievable and meaningful
- Error budgets and burn rate
- Policy: what happens when the budget is spent
The formal definition of toil, and the argument for capping it at fifty percent.
- Identifying and measuring toil
- Automation return on investment
- Self-service over ticket queues
- Deciding what not to automate
Nines, compounding dependencies, and why 100% is the wrong target.
- Availability arithmetic and compounding
- Cost of an additional nine
- Failure domains and blast radius
- Risk acceptance as an explicit decision
Forecasting demand and provisioning ahead of it, with headroom you can justify.
- Load modelling and growth forecasting
- Headroom and saturation targets
- Load testing to find the knee
- Regional and failover capacity
Sustainable rotations are an engineering problem, and interviews treat them as one.
- Rotation size and follow-the-sun
- Paging policy and alert quality
- Handover and shadowing
- Measuring on-call load
BuildDefine SLIs and SLOs for a real service, with an error budget policy agreed in writing.
Observability in depth
Not dashboards for their own sake: the ability to answer new questions about production.
PromQL fluency plus an understanding of what high cardinality costs you.
- Counters, gauges, histograms and quantiles
- PromQL: rate, histogram_quantile, aggregation
- Cardinality explosions and cost
- Federation and long-term storage
The only practical way to find latency in a service graph you did not design.
- OpenTelemetry instrumentation
- Span context propagation
- Tail-based sampling
- Finding the critical path in a trace
Logs are the most expensive telemetry. Deciding what not to log is the skill.
- Structured logging standards
- Retention tiers and cost control
- Log-based metrics and their pitfalls
- PII and compliance in logs
Increasingly standard at scale, and a differentiator in interviews.
- CPU and heap profiles in production
- Flame graphs and reading them fast
- Regression detection between releases
- Overhead and sampling rates
A dashboard nobody opens during an incident is a dashboard that failed.
- Service overview dashboard patterns
- Golden signals per service
- Drill-down structure
- Dashboards as incident entry points
BuildInstrument a service end to end and answer three questions you could not answer before.
Designing for failure
Systems fail. The engineering question is how they fail and how much they take with them.
Timeouts, retries and circuit breakers, and how naive retries cause outages.
- Timeout budgets across a call chain
- Retry with backoff and jitter
- Circuit breakers and bulkheads
- Load shedding and graceful degradation
Testing the failure paths that never run in normal operation.
- Hypothesis-driven chaos experiments
- Latency and error injection
- Dependency failure simulation
- Blast radius control
RTO, RPO and a restore you have actually performed rather than assumed.
- RTO and RPO definitions per service
- Backup verification and restore drills
- Multi-region failover design
- Data replication lag and consistency
Most incidents are caused by a change. Rollout design is the biggest reliability lever.
- Progressive delivery and canary analysis
- Automated rollback on SLO burn
- Change freezes and risk windows
- Config changes as a deploy
Your reliability is bounded by the things you call. Map them before the incident.
- Critical dependency mapping
- Hard vs soft dependencies
- Third-party SLA reality
- Fallback and cached responses
BuildRun a game day: inject a real failure into a staging system and publish the findings.
Incidents and learning
How an organisation handles incidents determines whether it gets more reliable or just busier.
Structure under pressure. Being able to run an incident is a hiring criterion.
- Incident commander, comms and ops roles
- Severity levels and escalation
- Status communication to stakeholders
- Mitigation before diagnosis
The cultural core of SRE, and a common interview discussion topic.
- Timeline reconstruction
- Contributing factors, not root cause
- Counterfactual reasoning traps
- Action items with owners and dates
Why 'human error' is the start of an investigation rather than its conclusion.
- Safety science basics
- Cognitive load during incidents
- Alert fatigue and burnout
- Psychological safety in reviews
The gate that stops unreliable services reaching production in the first place.
- Production readiness review checklists
- Launch and handover criteria
- Service ownership models
- Deprecation and decommissioning
DORA metrics and MTTR as evidence that the reliability work is working.
- DORA: lead time, frequency, MTTR, change failure
- Incident trend analysis
- Toil measurement over time
- Reporting reliability to leadership
BuildWrite two postmortems to publishable quality, including contributing factors and honest action items.
Interview preparation
SRE loops combine coding, systems design, troubleshooting and reliability theory.
The signature SRE interview: a broken system and a shared terminal.
- Systematic narrowing under pressure
- Forming and testing hypotheses out loud
- Knowing when to mitigate vs investigate
- Common Linux and network failure signatures
SRE coding is usually practical: parse this, automate that. Still real programming.
- Log parsing and aggregation scripts
- Rate limiter or retry implementation
- Concurrency and timeouts in code
- Writing testable operational tooling
Design a system to a stated availability target, and justify the cost.
Ch — Scalable APIs- Multi-region active/active design
- Failure domain isolation
- Capacity and headroom decisions
- Trade-off between cost and nines
Expect direct questions on SLOs, error budgets and toil definitions.
- Explain an error budget to a product manager
- Choosing SLIs for a given service
- When to halt releases
- Availability maths on the spot
Incident stories with specifics, and evidence you improved a system afterwards.
- An outage you led the response to
- Toil you eliminated with measured savings
- Pushing back on an unsafe launch
- Improving on-call for a team
BuildA written portfolio: an SLO document, a postmortem, and a chaos experiment report.
SRE tools on your CV
- Prometheus
- Grafana
- OpenTelemetry
- Kubernetes
- Terraform
- PagerDuty
- Go / Python
- Chaos tooling
What SRE employers ask to see
- An SLO document with an agreed error budget policy
- Two published blameless postmortems
- A chaos experiment report with findings and fixes
- An automation project with measured toil reduction
Concentrated at companies with real scale, and paid accordingly. The role expects software engineering ability alongside operations, which keeps the supply low.
Content last reviewed 2026-08-31. Guidance only — no institute or paid placement is endorsed anywhere in this book.