Three years ago, when Gustavo Brey took over IT for the social and health services government agency in Argentina, he inherited development and operations teams that did not like working together, and it showed. Each time a deployment went wrong (which Brey says happened 80 percent of the time), each team would point fingers at the other team instead of trying to solve the problem.
We talked to Brey, IT Manager for Instituto Nacional de Servicios Sociales para Jubilados y Pensionados, about what he did to turn things around (these days 90 percent of deployments are a success). Here’s what he said:
The Enterprisers Project (TEP): How were the development and operations teams working together before you moved to a DevOps model?
Brey: When I started here, I felt like there was a war between the two teams. Every time there was a problem, one side pointed to the other instead of trying to get the problem fixed. They weren’t even taking time to see what caused the problem.
TEP: What were some of the issues causing problems between development and operations?
Brey: All of the developers used to test their code in a different environment. And when this code has to move to production or test environment, they couldn’t make it work because they were using different versions of libraries. So the first thing we did was to standardized the environment. We provided each developer with a localized environment that was the same version of libraries and version of the middleware as the infrastructure environment.
Another problem we had was that the deployment process of every application was failing. We have around 100 deploys per week and 80 percent of those deploys had a problem. Every time we had a deployment, I had to pray it would be OK. So two years ago we automated deployment. We started implementing Jenkins, which is an open source tool we use for deployment. We started moving all the deployment process in an automated fashion.
Every time there was a problem in production, we were always alerted by users. We never knew something was wrong unless we had a call or a web form. Every time we found a problem, the two teams started saying that the other team was responsible. Those were the main problems we had at that time.
TEP: What are some of the other ways you helped the teams work better together?
Brey: We started a DevOps meeting every Monday morning where we have people from DevOps teams discuss the plan of the week. After that meeting we have an architectural meeting and we decide all the policies and practices. We plan the week and the next month in terms of new applications and how we are going to architect them. Since starting the meetings, standardizing the environments, and automating the deployment process with a ticketing tool, we’ve gone from 80 percent of our deployments having a problem to 90 percent of deployments being a success.
TEP: Congratulations! That’s quite the improvement. How did things change after that?
Brey: As we tackle these things, both teams have a better relationship. They don’t fight anymore. They communicate. Developers always see operations as lazy. And operations always think developers will break the build. And that won’t change. But if you can have tools to bring them to the table so that they don’t have any regret or any shame, it makes it easier for them to communicate. And for me in my role, I no longer need to know if there is a deployment or not. We have thousands of commits per day. We have about 30 deploys per day. Most of the deploys are automated. All the developers have the chance to deploy their code on a different environment except for production because our standards don’t let the developer deploy to production. But the procedure we use to deploy in development and QA and production is the same, the difference is who approves it. That gives me more time.
Instead of looking at the problem that is happening today, we have time to look at new things or to look at the new DevOps or infrastructure or architectural changes I want to make. We are looking at Docker and OpenStack. We want to move forward to create some sort of a path to make the operations side web-based. I don’t think we need developers talking too much to operators. They have to provision all the infrastructure they need as a service. But we have to work on the most important things at the very beginning.
ALSO READ
- Gene Kim on how to get started with DevOps
- The secret ingredient for successful DevOps
- Focus DevOps on high-impact areas
Gustavo Brey is an Information Systems Engineer graduated from the National Technological University (UTN), with over 15 years of experience in IT in different industries, holding varied positions. At present, he is the Information Technology and Communications Manager at PAMI (INSSJP) where he has been promoting the technology upgrade of the Institute with the aim of improving the Argentine health care system. He has been teaching courses on Software Architecture for 10 years and has created the IT Project Architecture subject at the UTN. Gustavo participates in different IT conferences addressing topics like health, creativeness and free software, lecturing about experiences and research in Agile Methodologies, Programming Languages, Software Architecture, Technology and Programming for Children.
