What a Day 2!

Today was the first day of conference talks, as opposed to the excellent training and workshops of yesterday, and wow, my brain is full!

My day started out with an unfortunately blameful post-mortem of myself and my devices (pro-tip: if you rely on devices to wake you up, make sure they are charged enough to do so), but after that there were so many learnings to be had. I spent my morning in Grand Ballroom A, starting with "You Code Like a Sysadmin" which was a great discussion of the code that we ops folk write. We spend a lot of time judging ourselves for writing code that isn't "good enough" for whatever reason- either we don't have a development background or we had to hack something together just to get it working, and many of us have been ridiculed, either by ourselves or others, for the quality of our code. But our code, regardless of how pretty or elegant it may or may not be, works. There was some audience participation in this talk where people came up to the mic and shared the worst code they'd written (confession time: my worst code was when I first (accidentally) started as a sysadmin and inherited 200 snowflake servers with absolutely nothing automated about them and I cobbled together some bash monstrosity before learning that configuration management was a thing that existed, and wow that was a mess) and the common theme was that IT WORKS. The other moral of those stories was to keep coding and keep learning. We keep coming to conferences because we realize that there's always more that we can learn, and even if we don't think of ourselves as good at writing code, we can keep learning about that as well. 

After that I got to hear more learnings from Dave Cliffe at PagerDuty about what to do when the shit hits the fan (did I mention that I'm going on-call soon?). One of the big lessons here was that teams should always be learning from their on-call experiences and working to improve them. We all hate getting alerts, but there's a lot that we can learn from them. Were they useful and actionable? Did they provide enough context? Alert fatigue is a real problem, and we can learn from the alerts that are sent out how our monitoring systems are doing, and work to improve them - not just for the sake of the sites or products that we're monitoring, but for the health and happiness of those of us wearing the pagers as well. I really liked the idea of having an incident commander who takes charge when the fan is getting bombarded with excrement and handles things like communicating status updates (both internally and externally) and shielding the people who are doing the heavy lifting of fixing things from distractions. 

The official conference lunch was very interesting in that hors d'oeuvres and small plates were being circulated around the vendor exhibition hall, giving attendees and vendors a great chance to interact while filling their faces. Since my morning device failure robbed me of my breakfast, I needed a bit more food, so a few of us, including organizer Pat Cable and the awesome Standalone Sysadmin went across the street to grab some SushiOps, which was a nice followup to the PhoOps of last night's dinner. 

Right after lunch I took my food coma to Doug Hughes' mini-tutorial on identifying bottlenecks with strace and truss, which was so engaging I found myself almost wishing I had some performance issues to track down right then. There was so much to take in there I was very glad for the slides being on the helpfully-provided USB key for future reference. To finish up the day, Carolyn Rowland and one of my fellow Etsians Avleen Vig talked about dealing with the interruptive nature of our work as sysadmins and how we can make this kind of environment as productive as possible. We learned things like how interruptions cost us on average 2.1 hours a day of productivity, and how Avleen currently has 76 tabs open in Chrome (makes me feel a lot better about my 37!). A lot of research has been done into interruptions and how they affect our brains, but there are things we can do. We can shut off all notifications on our phones that we don't need to do our jobs, we can work with our teams and organizations to give us times of day or even full days where we can focus on work sans meetings, and we can get our little jolts of dopamine from finishing small sub-tasks of projects instead of just constantly refreshing Twitter (I write as I tab over to Twitter to check it - clearly I just need some time for all the knowledge to sink in!).

Between that and the BoFs I grabbed some drinks with a few people and talked about Ops School, and thought to myself how awesome it's been to get to talk in person with so many smart people- some I know IRL, some I've only known on the internets, and plenty of new faces as well. The hallway track really is a great part of this conference. I now find myself torn between enjoying more of that and getting some sleep so my Jawbone will quit judging me so much, but either way, I'm looking forward to another great day of talks tomorrow.