0% completed

All LessonsFree Lessons (9)

Living in Production

Introduction Scope and Overall Development Lifecycle Pragmatic Architecture Quiz

The Exception That Grounded an Airline

Case Study: The Exception That Grounded an Airline The Change Window The Outage Consequences Postmortem Hunting For Clues The Smoking Gun An Ounce of Prevention Quiz

Stabilize Your System

Real World Problems Defining Stability Expanding System Lifespan Failure Modes Stopping Crack Propagation Chain Of Failure Quiz

Stability Antipatterns

Antipatterns Integration points Socket Based Protocols Network Connection The 5 AM Problem TCP With Firewall Socket Connection Failure HTTP Protocols Vendor API Libraries Countering Integration Point Problems Chain Reactions Avoid Failures Quiz

Failures And Blockages

Cascading Failures Users Heap Memory Sockets Expensive To Serve Non-standard Users Deadlock And Bad Users Malicious Users Blocked Threads Spot The Blocking Libraries Self-denial attacks Avoiding Self-Denial Scaling Effects Shared Resources Unbalanced Capacities Drive Out Through Testing Dogpile Quiz

Force Multiplier

Outage Amplification Controls and Safeguards Slow Responses Unbounded Result Sets Reasons for unbound result set Quiz

Stability Patterns

Timeouts Timeouts Side Effects And Latency problems Circuit Breaker Bulkheads Steady State Data Purging Log Files Fail Fast Let It Crash Handshaking Test Harnesses More About Test Harness and Failure Modes Decoupling Middleware Shed Load Create Back Pressure Governor Quiz

Launching An Online Store

My First Holiday Load Test

Taking the Pulse

Thanksgiving Day and Black Friday

Diagnostic Tests

Comparing options

Foundations

Physical Fundamentals NICs and Names Programming for Multiple Networks Virtual Machines in the Data Center Containers in the Data Center The 12-Factor App Virtual Machines in the Cloud Containers in the Cloud Quiz

Processes on Machines

Deployment and Runtime View Code Configuration Transparency Designing for Transparency Enabling Technologies and Logging Human Readability of Log Files and Voodoo Operations Instance Metrics and Health Checks Quiz

Interconnect

Interconnect Layer DNS Global Server Load Balancing with DNS Availability Of DNS And Load Balancing Software and Hardware Load Balancing Health Checks Demand Control Preventing Disaster Network Routing Discovering Services Migratory Virtual IP Addresses Quiz

Control Plane

Mechanical Advantage System Failure, Not Human Error Automation Goes Really Fast Platform and Ecosystem Development Is Production System-Wide Transparency Economic Value The Risk of Fragmentation and Logs/Stats What to Expose Configuration Services Provisioning and Deployment Services Command and Control The Platform Players Quiz

Security

Security Breaches Broken Authentication and Session Management Cross-Site Scripting Broken Access Control Security Misconfiguration Sensitive Data Exposure Cross-Site Request Forgery The Principle of Least Privilege Configured Passwords Waiting for Godot Quiz

Design for Deployment

Planned Downtime Automated Deployments Continuous Deployment Phases of Deployment Relational Database Schemata Schemaless Databases Web Assets Rollout Cleanup and Deploy Like the Pros Quiz

Handling Versions

Non-breaking API changes Gap Between Documentation and Actual Implementation Breaking API Changes Handle Others’ Versions Quiz

Case Study: Trampled by Your Own Customers

Countdown and Launch Load Testing Murder by the Masses The Testing Gap Aftermath Quiz

Adaptation

Convex Returns Process and Organization Platform Team Painless Releases Service Extinction Team-Scale Autonomy Beware Efficiency Quiz

System Architecture

Evolutionary Architecture Loose Clustering Explicit Context Create Options Splitting Incorporating Change Quiz

Information Architecture

Databases Messages, Events, and Commands Services Control Their Identifiers URL Dualism Embrace Plurality Quiz

Chaos Engineering

Production Environment Problems Antecedents of Chaos Engineering The Simian Army Adopting Your Own Monkey Injecting Chaos Targeting Chaos Disaster Simulations Quiz

Bibliography

Distributed Systems: Building Software for the Real World/

...

/

Vital Signs

Vital Signs

We'll cover the following...

Shortly after the session

Response time
Looking into the thread dumps

Shortly after the session

The incident had started about 20 minutes before Daniel called me. The operations center had escalated to the on-site team. David, the operations manager, had decided to bring me in as well. Too much was on the line for our client to worry about interrupting a vacation day. Besides, I had told them not to hesitate to call me if I was needed. We knew a few things at this point, twenty minutes into the incident:

...