DevOps in SAFe – Success Possible with Cultural Change First

DevOps is the new buzzword in many companies. Although the concept has been there for a while it is still something new, something that we feel we need, something that we try to implement. And the sooner the better! Yet we still fail to recognize what DevOps has actually become and that the implementation is not only technical but rather cultural.

---

What is DevOps?

Let’s start with Wikipedia: “DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality. DevOps is complementary with Agile software development; several DevOps aspects came from the Agile methodology.”¹

Ok, so in the beginning, there is Dev and Ops, and the gap between these two competencies, then we move to SDLC (Systems Development Life Cycle), and the technical approach to DevOps. Only once you read further, there’s even a small mention about the culture: “Other than it being a cross-functional combination of the terms and concepts for ‘development’ and ‘operations’, academics and practitioners  have not developed a universal definition for the term ‘DevOps’. Most often, DevOps is characterized by key principles: shared ownership, workflow automation and rapid feedback”.

So, here we talk about shared ownership in addition to automation and feedback. Finally, a bit better, but still the emphasis is absent (the underlying is mine).

All of this is coming from the origin of DevOps, how it all started, and when we take a journey somewhere 20 years in the past all of this is correct.

When I started my first job as DevOps in Thomson Reuters this is exactly what we did. There were around 100 developers on-site in Gdynia, with another 300 working from remote locations and an Ops team in Boston.

With our small team of 3, we started setting up a continuous integration environment on Jenkins, code repository on SVN, multiple automation scripts and virtualisation of environments. Upgrading to new technologies and fun stuff like scripts to upgrade solutions from something called JamNT to MsBuild. There was no discussion about any mindset. We had a job to do and that’s what we did, as DevOps, however as DevOps engineers.

So when we move to DevOps engineer description, since very often that is the synonym of DevOps, we can read pretty much the same. For many people DevOps is a person that will deal with many troubles we find in technologically advanced pipelines.

“A DevOps Engineer works with developers and the IT staff to oversee the code releases, combining an understanding of both engineering and coding. From creating and implementing systems software to analyzing data to improve existing ones, a DevOps Engineer increases productivity in the workplace. They understand the software development lifecycle and have a clear understanding of various automation tools for developing digital pipelines (CI/ CD pipelines)”.²

This is very often the reason why, when we try to address the problem of DevOps, we talk about a person, or a team of people, who have the expertise and experience to implement all of the above into our pipeline so that we are faster, more agile. Unfortunately, the problem is that we are focusing on an individual or a group of people with a limited capacity to solve problems, and although I met some very good DevOps engineers, who were extreme experts and solution seekers, they were still one person who could not be everywhere and do everything.

Let’s fast-forward to the present… DevOps has been developed by a community of like-minded people who had understood automation and virtualisation is not enough to be truly agile. It’s how we work together as a team that really matters. It’s the ownership and the collaboration between different team members and between teams alike. In short it’s the culture.

CALMR

SAFe’s approach to DevOps is contained within the CALMR (Culture, Automation, Lean Flow, Measurement, Recovery) concept. And it starts with the Culture of Shared Responsibility. It is the teams taking responsibility for the end-to-end process. We understand that every team member and everybody on the Agile Release Train (ART) has a specialised set of skills and responsibilities, however, we also collaborate to enable smooth flow through the pipeline. Sometimes it means enabling others who work before us in the process, sometimes it means creating something that makes the life of people after us in the process a bit easier.

Only afterwards supporting this principle comes the:

• Automation, which automates as much as possible of the pipeline;
• Lean Flow, which keeps the batch that go through the pipeline as small as possible;
• Measurement, which is about metrics, those inside the pipeline (environments) and outside of it (customer feedback);
• Recovery, which is about rolling back if something happens.

How do you do it then?

Well, I guess there is no single exercise or workshop that one can do to change the culture. It is yet another journey that every setup has to take and find its own path.

Once I witnessed an interesting practice implemented by a company that was trying to apply DevOps practices and techniques. They were working on multiple value streams and had various continuous delivery pipelines however, there were several Scrum teams working in a cross functional setup – every team could work on everything, more or less at least. They were helping each other on a continuous basis, and although there was no specific scaled agile setup implemented, they were fostering the mindset of a culture of shared responsibility.

How they did it however was quite eccentric. Everyone was located on a single floor and open space, however quite spread around the building. In one location that was central to everyone they installed a red siren light and hooked it up to one of the computers.

Moreover, they hooked it up to their Jenkins pipelines and whenever someone or something broke the pipeline the light came on illuminating the floor so everyone knew there was a problem. Another thing that was happening was that automated scripts were blocking all merges to the main branch until the issue was fixed. That meant, there was an “all hands on deck” situation and everyone wanted to fix the issue as soon as possible. Was it a bit drastic? Maybe, and it certainly required preparation and couldn’t be implemented everywhere all the time. However, what I saw there was not a red light on the floor, but the intrinsic need to help and collaborate with each other, the mindset that we are all in this together and if something is broken – it needs to be fixed, with all means necessary. The average time for fixing a problem was 10 minutes.

To put a different point of view of another setup where we had DevOps engineers, however there was no mindset. In one of the projects, there were multiple teams and ARTs, working on a single solution, with multiple DevOps teams, responsible for different aspects of the Pipeline. There was no light, there was no solution, there were processes that clearly stated who is responsible for what, yet often the issues were laying around with so many people involved however nobody actually took responsibility for anything. Development teams pushed code, on a daily basis, without any consideration if what they are pushing is actually working or not. Sometimes they made hundreds of merges on a daily basis. Continuous integration was so slow that there was no immediate feedback if what one did locally was working in a higher environment. If you were lucky, maybe the next day. The complexity and lack of culture was so bad that at times there was no successful build and deployment for 6 weeks! Constant conflicts and escalations, instead of collaboration and ownership.

And there are other practical examples that you can get inspiration from and figure out your own way to build on the culture and have fun at the same time. One of the funniest I’ve seen was on a smaller scale, with one team, in a single room. In the middle of the room they set up a nerf gun and hooked it up yet again to some scripting tool like Jenkins and when one of the developers broke the build, the gun targeted that person, and started shooting at him, making everybody a good laugh.

While doing SAFe transformation alongside your DevOps implementation at some point it is very good to go through SAFe DevOps workshop. Some companies sell this as training, however it is more useful when the workshop is run with a full ART, all the teams and preferably all stakeholders.

Facilitated by Scrum Masters, each team maps out their Continuous Delivery Pipeline to the best of the knowledge and ability. We have different stakeholders present who can fill in the gaps, where teams lack expertise. This is the way to dig into details and different activities that start with continuous exploration, move through continuous integration and deployment, and culminate with release on demand. We make sure to understand who is participating in each of the activities and estimate what is the cycle time and lead time of each of them. Once done, we mapped out the current state. Since one of the key aspects of being agile is improvement, we move to the future state, so the pipeline we would like to have. Adding additional improvements into the “postcard from the future” we start creating an Improvement Backlog for the ART. While further estimating the benefits of each of those improvements, we can already move into prioritising the backlog. However, in my mind, the immediate change that one can get from the workshop is the understanding and team spirit that can be achieved through it.

DevOps is a mindset

If you are responsible for or are part of a DevOps implementation, I advise to focus first on the culture. Make it fun and engaging. Enable others to make necessary decisions to take care of the automation, virtualisation, lean flow, metrics, and rollback, while you work with the entire Continuous Delivery Pipeline to take ownership of the end-to-end process and help each other every step of the way.

Sources:

1. https://en.wikipedia.org/wiki/DevOps
2. https://business.linkedin.com/talent-solutions/resources/talent-engagement/job-descrip-tions/devops-engineer