sys_exorcist

When an issue may be occurring, an automatic alert system will ring someone’s pager (hopefully not mine!). Nearly all of our problems are caused by changes to our systems (either human or automated), so the first step is playing the “what is different game.”

My favorite professional development is informal chats with my teammates. There is always something they know that I don’t. There are plenty of formal classes available as well (unix internals, deep dives into various programming languages). We spend a lot of time giving talks at universities and have a booming internship program.

Coffee.

SRE teams typically focus on a single service like Search, or a piece of our infrastructure. My team works on Storage infrastructure like Colossus and Bigtable (pdf)