DevOps: Beware 3 paths to the dark side

DevOps: Beware 3 paths to the dark side

Three common mistakes can lead your team into the dark side of DevOps - where nothing goes right and everybody blames each other. Here’s how to avoid them

93 readers like this

When DevOps is done correctly, it’s a sight to behold. People are working together, tools are being applied at the right times in the right places, apps are moving through the pipeline, and all the stakeholders end up happy with the outcome. 

But anybody who’s built a DevOps practice knows that things don’t always go according to plan. Everybody in the process usually has the best of intentions, but sometimes teams can run into problems executing on their DevOps strategies. They encounter the dark side of DevOps – where nothing goes right, everybody’s blaming each other, and nobody has a solution.

[ Some common DevOps wisdom falls flat. Read 7 pieces of contrarian DevOps advice. ]

How do these “dark side” situations crop up? And what can you do to get back in the light? Here are a few scenarios to consider.

1. Us against them

Companies in virtually every industry have tried to reorganize departments to get people talking to each other more.

There’s been a lot written in recent years about the benefits of breaking down silos. Companies in virtually every industry have tried to reorganize departments to get people talking to each other more, sharing more information, streamlining the decision-making processes, and aligning toward common goals.

This kind of alignment is especially important in DevOps. But it’s not always easy to achieve.

Traditionally, the Dev side and the Ops side have been at odds because they approach their jobs with different styles. Developers are creative types who want to push the limits on code and plow through any problems they encounter. The operations people are more buttoned-up and resistant to change.

Then there’s another issue: Dev and Ops pros don’t only have to get along with each other. They must work in concert with the business side of the organization as well. Getting all sides to work together – ridding the organization of the “us against them” mentality – is critical to DevOps success.

[ Read our related article: 3 DevOps roadblocks and how to beat them. ]

Organizational issues create tension, resulting in difficult, friction-laden handoffs between teams. The problems often show up in the software delivery process and in the software itself in the form of quality gaps, inconsistent user experiences, and other issues.

Organizations need to acknowledge that culture will have a direct bearing on DevOps success, and commit to taking on those challenges. Organizations need to align, communicate, and be inclusive.

2. Poorly defined KPIs

If the KPIs aren’t appropriate for the task, the initiative may fall short of actual success.

The way a team defines success ultimately determines whether the team is successful. A team can appear to do a great job based on the KPIs it created. But if the KPIs aren’t appropriate for the specific task, the initiative may fall several steps short of actual success. As with any project, effectively communicating the metrics of a project is critical to success. It’s equally important to think through the KPIs being set up for each project and not just recycle the same ones every time.

Going back to working across departments and breaking down barriers, the KPIs that the developers are using should also tie to a business KPI, such as the number of new customers acquired or the percent of revenue increased. This will show velocity and help the entire business work more effectively, rather than just running at a speed that’s not manageable across departments.

For example, if a quality engineering team wants to measure speed, it might create a metric based on the number of tests executed per sprint. Humans are driven by incentives, and the natural tendency of the team could be to add more and more tests without archiving the tests that are outdated. What if the team tried to operate with fewer tests? Fewer tests would lower the test cycle time. So instead of sheer quantity, a team should focus on the coverage and effectiveness of tests it’s doing.

[ Read also: DevOps metrics: Are you measuring what matters? ]

3. Deploying code out of step

Sometimes teams get ahead of themselves and run applications that haven’t been properly tested. Teams should treat their CD pipelines as applications themselves by checking for and eliminating bugs and errors in CD pipeline code. When teams don’t treat CD pipelines as applications, there is a significant cost and/or risk involved.

[ Need a primer on continuous integration/continuous delivery? Read What is CI/CD? ]

Teams sometimes run in parallel during CI/CD, which can speed up their automated testing and shorten feedback cycles. To avoid this, make sure deployment is not executing as a parallel step. That misconfiguration will cause your code to deploy at the same time the tests are running, regardless of their exit codes, pretty much defeating the purpose of automated testing before a deploy.

To avoid these issues, teams should build, manage, and create staging environments. First, build from dev to test to deployment starting from the left-hand side of the process. While teams may not be able to define an ideal automation solution completely on the first pass, going left to right encourages teams to focus on what the life cycle of an artifact looks like.

Second, teams should define their CD pipeline using an everything-as-code or pipeline-as-code approach. And finally, establish staging environments or test pipelines to validate the CD process.

Following these steps will give your teams an opportunity to iterate changes against application code, do some initial scans, and collaborate to define your CD pipeline.

Working together, setting the right KPIs, and coding in the correct order aren’t the be-all-end-all when it comes to avoiding DevOps problems. But these steps can help. Keeping these things in mind can help bring some light back into your organizations’ DevOps practices.

[ Read also: DevOps for doubters: How to deal with 9 kinds of people who push back.]

Brian is currently a DevOps evangelist and practitioner at CloudBees, where he helps the community and customers in implementation of Agile, CI, CD and DevOps practices. Prior to CloudBees Brian spent 22-plus years as a software professional in multiple domains including QA, Engineering and Management.

7 New CIO Rules of Road

CIOs: We welcome you to join the conversation

Related Topics

Submitted By Donna Tuths
September 28, 2020

In response to the COVID-19 pandemic, CIOs and other C-suite leaders must transform how they think about “experience” – for both customers and employees – or risk losing them.

Submitted By Ginny Hamilton
September 28, 2020

As CIOs and IT leaders adapt to leading fully-remote teams, they may inadvertently adopt bad habits that could hurt culture long term. Watch out for these behaviors.

Submitted By Kassie Rangel
September 25, 2020

Looking to land a new IT job? Move your resume to the top of the list by avoiding these three common pitfalls


Email Capture

Keep up with the latest thoughts, strategies, and insights from CIOs & IT leaders.