Looking to land a new IT job? Move your resume to the top of the list by avoiding these three common pitfalls
12 days of DevOps: Expert tips to help teams succeed
Is your holiday wish to begin or improve a DevOps effort? Read on for 12 ways to help your teams shine
On the seventh day of DevOps, ensure the environment creation is repeatable
If you are doing manual environment creation, please stop. Manual environment creation will be the death of you. It is not scalable. When your team needs to create 500 or 1,000 different environments, doing them manually doesn’t work. Step back and look at automating environment creation, because it unlocks a lot of wasted time and allows teams to focus on the important things rather than on the infrastructure aspects of what you’re trying to do.
If you can get something to be repeatable, you’ve got a signal that you can ask your organizations to respond to. If a team is able to do something at a higher frequency and make a process easier to triage in smaller batches, the organization will realize it needed the stable signal. The fact that something didn’t work, or the fact that it works at all, when it’s not repeatable and automated generally takes heroic efforts. As laudable as it is when people step in and do these kinds of things, it’s actually counterproductive in the long run.
If something hurts in DevOps, you need to do it more frequently, and in smaller batch sizes. If you want to get to the point where your enterprise can release new software products more frequently and reliably, the aforementioned tasks cannot take days or even hours to complete – they need to be done in minutes. But it takes time to get there. But once you do, you’re ready for the next step, which is starting to see the benefits of what you created.
On the eighth day of DevOps, work on cultural transformation
Start cultural transformation by asking everybody in the development organization to respond to this signal. Now you’re asking them to build quality in at the beginning instead of waiting or responding to defects later. By asking them to change how they develop so that they can bring code in on a regular basis without breaking the existing functionality, get the development team used to the idea of adding smoke tests. If you want to know what tests to start with, ask your manual QA team what things they see that make it completely useless for them to be testing an environment that’s so bad you shouldn’t have it in the first place.
Require your development organization to keep those tests running with every single check-in. Do that over and over until it becomes the standard and then add some more tests, but add tests at a rate at which the organization can be successful.
Organizations tend to jump to the eighth day too soon. When that happens, organizations often have an insurrection from the developers because you’re slowing them down and breaking their ability to get work done based on problems that haven’t been fixed and that are outside of their control. Leadership can lose credibility when that happens, and it’s hard to get that credibility back once it’s lost.
On the ninth day of DevOps, make triage easier
As leaders, we want to make it easy to get the right behavior out of our teams; in this case, the development teams. So what can we do to make triage easy for them, as we’re gating code and making sure it can’t go through?
Talk to the developers about what do they do when they debug and triage. Figure out if there are things that can be automated, and pull that information together so it’s really easy for them to do the right behaviors, and enable more productivity in terms of triage.
Soon they will get used to keeping builds green and tests passing. But when they fail, ask what they’re doing, how they’re approaching it, and what they need for debugging and triage. See if there’s any other way to pull that information together for them automatically so it’s just straightforward and easy for them to do the right thing.
On the tenth day of DevOps, increase gating at the speed of Dev
Now that your development teams are used to responding to the smoke tests and you’ve gained higher efficiency from the triage, start building in more quality by raising the gate and adding tests to make the process more stable on an ongoing basis. Do this at a rate at which the Dev organization can succeed. It’s a balancing act: If you start with five tests, then add 5,000 and nobody can get a green build, you’re going to have an insurrection again. This part of cultural transformation needs to be gradual, and you need to start increasing the barrier and the gates for quality at a rate that the organization can handle.
You want to get a culture that doesn’t break things. And when things do break, ask: Where should we have caught this? What is the fastest, cheapest, easiest way that we could have caught this problem? It takes a village and a mindset to be vigilant in your approach to problem-solving and problem prevention.
[ How does your culture stack up? Read 6 DevOps culture mistakes holding you back. ]
On the eleventh day of DevOps, start shifting issue discovery left
We started as far right as possible because at our level of influence, keeping the environment stable and working well will have the biggest impact on the broadest number of people. Now we’re starting to figure out: What are the causes of these bugs, and can we catch them sooner? Can we catch them closer to the developer so they can learn in real time? The idea is to then take what the team has done in this environment and move it to the left, starting with the Dev components that are causing most of the challenges and the issues in the organization.
This is where analytics can start to help: You can look at a particular component in the code that’s causing issues or a particular type of environment that you’re having problems with, and devote more resources to swarm the problem.
Most often, the problem is because we didn’t write a unit test or as simple a test as we could have. The cheapest place to fix a bug is on the whiteboard, before you even write the code, get the architecture right, and so on. Sometimes it simply means the teams need to write more robust unit tests. No matter the solution, it will become part of the process to continue shifting issue discovery left and getting motivated to ensure that these issues are fixed as early as possible.
On the twelfth day of DevOps, break out the eggnog!
Once you’ve helped implement all of the previous steps and your team has become good at practicing them, you get to spend Day 12 having fun! Why? The ability to shift issues left in the software delivery process provides an opportunity to look back and appreciate the progress your team has made.
When you face those difficult days when people are complaining or there’s a little bit of an insurrection, you’ll see that the problem that you’re having today is because you didn’t do something correctly previously. The good news is you’ve just discovered another area where you need to go back and address areas for improvement.
Running through this continuous improvement process – going back to Day 1 whenever you hit a snag – helps ensure that you have a reliable environment set up in continuous improvement. You have tests that you can run repeatedly without failing, and environments that you can deploy to repeatedly without failing.
If at any point those things are no longer true, then try something different and do it again, and make sure that the environment is stable and that the deployment is stable. Ensure those tests are running and are not flaky. This process does work.
Gary and I have seen hundreds of organizations do this and do it well. It is not easy; it takes time. It is not something you can do in a week or even a month. For large, complex organizations with hundreds or thousands of developers and hundreds or thousands of products, these transformation journeys can take months or even years.
Remember that it is not a step forward every day. Sometimes you have to take a step back to make progress.
[ Want more advice from your DevOps peers? Read DevOps lessons learned: What I wish I knew sooner. ]