Part five of a five-part DevOps series.
Part 1: What's creating demand for DevOps?
Part 2: The role of the cloud
Part 3: Tips for making DevOps a reality
Part 4: Surprises born out of DevOps
Part 5: Measuring success in a DevOps culture
PANELISTS:
Pete Buonora, Enterprise Architect, BJ’s Wholesale Club
Lee Congdon, CIO, Red Hat
Tom Soderstrom, Chief Technology Officer, Office of the CIO, Jet Propulsion Labs
Merv Tarde, VP of IT, CIO, Interstate Batteries
Rajesh Wunnava, Global IT Leader and Digital Business Strategist, Warner Music Group
INTRODUCTION: The Enterprisers Project assembled a group of seasoned IT leaders in an interactive exchange to discuss how they're using DevOps in their organizations. Here are highlights from the conversation.
Measuring DevOps Success
TEP: Two final questions: What kinds of results are you seeing since you’ve begun to move to a DevOps culture? And how are you trying to measure success?
Lee Congdon: For us, the number one measure is speed from the time the agile team starts a project until it's in production. We want to be ahead of the business so that if the business isn't ready, we let them decide how fast they want it to move.
Tom Soderstrom: I agree completely with that. Another success measure is that they see it costs less overall because we don't take heavy ideas and develop them all the way because of the prototyping idea, and we get more profit. So reducing overall costs of development. The other thing we really want to measure is customer satisfaction. If it doesn’t go up we'll figure out what we did wrong.
Merv Tarde: We're basically the same but we have two things to drive us. One is excellence, which comes before speed, and the other is using the Net Promoter Score concept. So we're getting everything measured that we can by that type of net-promoter scoring methodology from our stakeholders on how well we do things. Whether it's speed, excellence, or getting things out in a certain timeframe, we use short surveys—real quick, you know, no more than a minute—[we put] these things out and measure how well we're doing.
Tom Soderstrom: When you do those short surveys, what kind of response rate do you get?
Merv Tarde: I know if we keep it short, within a minute or two, we're getting response rates of probably 70 to 80 percent, which we consider to really be pretty good. But if we give something out that takes ten minutes or more, it goes down drastically, well below 50 percent; probably more in the 20 to 25 percent range. People don't have the time.
Tom Soderstrom: One other question I have is around A/B testing. Is anybody yet doing this in their enterprise?
Lee Congdon: We did it a lot at Capital One when I was there. It's just starting here. We've got a new version of our website coming online soon and that will have extensive capability to do [A/B testing]. One piece of serendipity is that I walked past one of my development organizations today and they had [A/B testing] up on the white board, so I would say we're in the very early stages, but we will be implementing it later this year on a much broader basis.
Pete Buonora: Especially with more innovative projects, we're definitely starting to implement some split testing and we are looking to put some of that into the process. I have been trying to inject some of the concepts from the Lean Start Up and that's one of the core concepts—along with the feedback loop and putting it in front of the customer and getting a direct feedback before you can move on to the next step and thinking about whether to “pivot or persevere”. So we use some concepts that are slowly working their way into the enterprise through an organic fashion.
Tom Soderstrom: That's great. All the things you mentioned are great ideas.
Merv Tarde: We are not doing this that formally, but it is something that after this conversation I will be looking into, that's for sure. We use something similar, but it’s in our agile group and eBusiness group and that’s it.
TEP: Thanks, everyone! Great conversation. And glad to hear DevOps is going so well for everyone.
Catch up on our previous DevOps Roundtable conversations here:
Part 1: What's creating demand for DevOps?
Part 2: The role of the cloud
Part 3: Tips for making DevOps a reality
Part 4: Surprises born out of DevOps
Part 5: Measuring success in a DevOps culture
Photo by Flickr user FredericRivollier.
