← Kenrick Tan / Back to work

Case study

Protocol Reliability Flight Deck

Release readiness, incident visibility, and rollback confidence for blockchain infrastructure.

Built a control plane that centralizes protocol health, pre-release validation, and emergency rollback workflows.

Impact

38% faster release cadence

Impact

70% faster incident triage

Impact

Zero critical release rollbacks in 2 quarters

Impact

99.95% validator uptime maintained

Impact

MTTD reduced from 18m to 5.2m

Problem

Release safety relied on manual checklists, and incident telemetry was fragmented across multiple dashboards, slowing decisions under pressure.

Architecture

Flight Deck Architecture

Select a component to explore its role and connections.

123456

Validators

Cluster health + consensus telemetry

Connects to: Telemetry Bus

Approach

  • Designed a single release readiness scorecard with automated canary gates.
  • Unified validator, node, and infra telemetry into one live mission-control view.
  • Added deterministic rollback playbooks with pre-approved remediation steps.

Outcomes

  • Engineers trust launch decisions with real-time guardrails.
  • On-call response is calmer and faster thanks to shared visibility.
  • Release managers get a clear go/no-go signal with audit trails.
  • SLA breaches reduced by 62% in 6 months.
  • On-call fatigue decreased with clearer escalation paths.

Metrics

Before → After comparison

Before

release Time
4.2 hours
triage Time
18 minutes
sli Breach Rate
4.2%
uptime
99.89%

After

release Time
2.6 hours
triage Time
5.2 minutes
sli Breach Rate
1.6%
uptime
99.95%

Stack

GoKubernetesPrometheusGrafanaVault