I love runbooks. No fluffy words, no deep dives. Just the information required. Everyone should be able to follow the instructions if they are alone in the middle of the night. When you are paged at 3 a.m. your sleepy brain is equivalent to a new hire with zero context.
Also, I think they have the hidden benefit of making you think. To write it you have to identify all the possible failure scenarios of the system and come up with the mitigation and recovery steps. Without a runbook, most of them would go unnoticed until they are a real problem in prod.
A practice I find helpful in my team is reviewing an artifact every week in the on-call handoff meeting (runbooks ,dashboards, and even the alarms). This ensures we are collectively aware of the content and updating it.
I'm curious to hear if people in small startups have experiences with those. Personally, we have nothing organized in engineering, only on the support team's side.
Thank you for a great article :) Totally agree that runbooks makes Engineers lives a lot earier especially if they are up to date like you beautifuly mentioned.
Thanks for the mention, Raviraj!
I love runbooks. No fluffy words, no deep dives. Just the information required. Everyone should be able to follow the instructions if they are alone in the middle of the night. When you are paged at 3 a.m. your sleepy brain is equivalent to a new hire with zero context.
Also, I think they have the hidden benefit of making you think. To write it you have to identify all the possible failure scenarios of the system and come up with the mitigation and recovery steps. Without a runbook, most of them would go unnoticed until they are a real problem in prod.
A practice I find helpful in my team is reviewing an artifact every week in the on-call handoff meeting (runbooks ,dashboards, and even the alarms). This ensures we are collectively aware of the content and updating it.
Great post once again!
Good points.
That’s a nice practice your team follows. These things can rot pretty quickly.
Great article! Thanks for sharing. And thanks for recommending my newsletter. 🙏😊
Thanks for the mention, Raviraj!
I love runbooks
I'm curious to hear if people in small startups have experiences with those. Personally, we have nothing organized in engineering, only on the support team's side.
Waiting for the article tomorrow :)
8 years ago if we had a runbook then it would generally say - investigate & fix :)
Thanks for this man! Great article!
Thank you for a great article :) Totally agree that runbooks makes Engineers lives a lot earier especially if they are up to date like you beautifuly mentioned.
I would like to self promote my growing newsletter too An Engineer's Echo: https://open.substack.com/pub/basmataha199?utm_source=share&utm_medium=android&r=2tl0x2
Please have a look and I would appreciate if you leave me any feedback 😊
Thanks!
I will take a look :)