Appendix E - Launch Coordination Checklist

  1. Table of Contents
  2. Foreword
  3. Preface
  4. Part I - Introduction
  5. 1. Introduction
  6. 2. The Production Environment at Google, from the Viewpoint of an SRE
  7. Part II - Principles
  8. 3. Embracing Risk
  9. 4. Service Level Objectives
  10. 5. Eliminating Toil
  11. 6. Monitoring Distributed Systems
  12. 7. The Evolution of Automation at Google
  13. 8. Release Engineering
  14. 9. Simplicity
  15. Part III - Practices
  16. 10. Practical Alerting
  17. 11. Being On-Call
  18. 12. Effective Troubleshooting
  19. 13. Emergency Response
  20. 14. Managing Incidents
  21. 15. Postmortem Culture: Learning from Failure
  22. 16. Tracking Outages
  23. 17. Testing for Reliability
  24. 18. Software Engineering in SRE
  25. 19. Load Balancing at the Frontend
  26. 20. Load Balancing in the Datacenter
  27. 21. Handling Overload
  28. 22. Addressing Cascading Failures
  29. 23. Managing Critical State: Distributed Consensus for Reliability
  30. 24. Distributed Periodic Scheduling with Cron
  31. 25. Data Processing Pipelines
  32. 26. Data Integrity: What You Read Is What You Wrote
  33. 27. Reliable Product Launches at Scale
  34. Part IV - Management
  35. 28. Accelerating SREs to On-Call and Beyond
  36. 29. Dealing with Interrupts
  37. 30. Embedding an SRE to Recover from Operational Overload
  38. 31. Communication and Collaboration in SRE
  39. 32. The Evolving SRE Engagement Model
  40. Part V - Conclusions
  41. 33. Lessons Learned from Other Industries
  42. 34. Conclusion
  43. Appendix A. Availability Table
  44. Appendix B. A Collection of Best Practices for Production Services
  45. Appendix C. Example Incident State Document
  46. Appendix D. Example Postmortem
  47. Appendix E. Launch Coordination Checklist
  48. Appendix F. Example Production Meeting Minutes
  49. Bibliography

Launch Coordination Checklist

This is Google’s original Launch Coordination Checklist, circa 2005, slightly abridged for brevity:

Architecture

  • Architecture sketch, types of servers, types of requests from clients
  • Programmatic client requests

Machines and datacenters

  • Machines and bandwidth, datacenters, N+2 redundancy, network QoS
  • New domain names, DNS load balancing

Volume estimates, capacity, and performance

  • HTTP traffic and bandwidth estimates, launch “spike,” traffic mix, 6 months out
  • Load test, end-to-end test, capacity per datacenter at max latency
  • Impact on other services we care most about
  • Storage capacity

System reliability and failover

  • What happens when:
    • Machine dies, rack fails, or cluster goes offline
    • Network fails between two datacenters
  • For each type of server that talks to other servers (its backends):
    • How to detect when backends die, and what to do when they die
    • How to terminate or restart without affecting clients or users
    • Load balancing, rate-limiting, timeout, retry and error handling behavior
  • Data backup/restore, disaster recovery

Monitoring and server management

  • Monitoring internal state, monitoring end-to-end behavior, managing alerts
  • Monitoring the monitoring
  • Financially important alerts and logs
  • Tips for running servers within cluster environment
  • Don’t crash mail servers by sending yourself email alerts in your own server code

Security

  • Security design review, security code audit, spam risk, authentication, SSL
  • Prelaunch visibility/access control, various types of blacklists

Automation and manual tasks

  • Methods and change control to update servers, data, and configs
  • Release process, repeatable builds, canaries under live traffic, staged rollouts

Growth issues

  • Spare capacity, 10x growth, growth alerts
  • Scalability bottlenecks, linear scaling, scaling with hardware, changes needed
  • Caching, data sharding/resharding

External dependencies

  • Third-party systems, monitoring, networking, traffic volume, launch spikes
  • Graceful degradation, how to avoid accidentally overrunning third-party services
  • Playing nice with syndicated partners, mail systems, services within Google

Schedule and rollout planning

  • Hard deadlines, external events, Mondays or Fridays
  • Standard operating procedures for this service, for other services

Copyright © 2017 Google, Inc. Published by O'Reilly Media, Inc. Licensed under CC BY-NC-ND 4.0