DevOps trends (and effect on quality)

What's Up In The DevOps World

Traditional Development teams and Operations departments normally have completely different motivations and deliverables. Development keeps changing things based on user or market demands for new features (and fixes of existing features). Operations try to keep everything as constant as possible in order to deliver maximum availability and stability of software services. These two contradicting goals create a bottleneck when it comes to delivery of business value. Some people, albeit misguided, call this bottleneck “Quality Assurance”.

What is DevOps?

Process methodologies, such as Extreme Programming, Agile, Lean, Six Sigma and others, promised higher quality and faster time to market, and they delivered. Once management got a taste, they obviously wanted more: better, faster, cheaper. 

But how do you get more out of a process, that may already be streamlined? How do you accomplish a rate of 10 releases per day, which to most organizations probably seems like pure fantasy?

The proverb “It takes a village to raise a child” highlights the fact that in order to raise a child to be a successful adult you need many different specializations working closely together as one unit. What if the entire IT organization is the village and the product is the child?

Everyone is equally responsible and accountable for the product.

DevOps means that there are no walls, no gates, no transitions, and no ceremony between Development and Operations. They are seamlessly integrated (when viewed from “above”) into a single, value delivering, IT entity. 

There may be individuals who specialize on “one side or the other”, but even those individuals interface seamlessly and directly without the need for an intermediary. Most importantly ... DevOps means that everyone – from Jr Analyst, to Mid Dev, to Sr. Test, to Director of IT – is equally responsible and accountable for the product from inception through retirement. This means the Devs are just as likely to be maintaining the system in Production as the Sys Admins are to be doing configuration testing in the Dev Environment.

In any organization, the first time someone mentions shorter delivery time lines, or throws around phrases like DevOps, most people tend to react negatively. This probably stems from lack of understanding. DevOps is not about cutting corners in order to push code out faster, and is definitely not about lowering headcount. 

DevOps is about prioritizing keeping the software deployable over working on new features. At any moment the boss could walk in and declare: “We are releasing, now” and nobody would bat an eye, nobody would even break a sweat. That is what this is about.

Obviously keeping the software instantly deployable requires a large amount of automation, but it is dangerous to only focus on the tools. The focus should instea be on people coming together and working towards making the entire process rapid – reliable – repeatable. All the way from inception of a feature to deployment to production and everyone in between.  

It is also important to realize that all this will not happen over night, and may never be “done”. It will will likely be an ongoing process of continuous improvement à la Kaizen.

The role of QA

In a pure DevOps world, Quality Assurance is no longer the gatekeeper between development and operations – the bottleneck. QA is an enabler (remember everyone contributes to raising the child). 

On one side, QA works together with development in trying to push more of their tests into the continuous integration system. Tests must have zero human intervention and generate their own test data. 

On the other side, QA works with operations to collaborate on monitoring tools; perhaps to continuously run smoke tests in Production. Chances are that operations is developing scripts for backup and restore of production systems, rollback of deploy, or scripts for disaster recovery.

Testing such procedures presents an interesting and perhaps new challenge for QA. Another idea is that test automation should have a higher effort at a lower level: the API.

Assuming the API is doing everything correctly (see the article What makes API Testing special?) then the UI is likely having issues. But your back end APIs and Data layer should be fine. 

Concentrating automation efforts on lower levels allows tests to be significantly less brittle. For higher level testing of the UI, which may change much more often, manual exploratory testing is a better use of resources. Any problems discovered can quickly be corrected due to higher deploy frequency. 

Bottom line is: QA can and should contribute to the product; bake QA test scripts as a deliverable in the final product. (More on this in the article Test First article.)


There are organizations where the DevOps methodology faces resistance.

DevOps assumes that just about everything, and certainly testing, is completely automated. As all testers know, there are certain things which cannot (or should not) be automated. 

Exploratory or ad-hoc testing may at first appear to have no room in DevOps. Also, long-running scenarios would appear to be problematic - for example end-of-month accounting procedures in banking. Such situations requires collaboration with other professionals than just the IT department. 

The principles of Context-Driven Testing come to mind: you do the best you can with what you have. But keep in mind that a high degree of low-level test automation will free up test resources, who can perform additional exploratory testing, both pre- and post-deployment.

Testing in Production, which DevOps embraces, may look like an immediate block for industries with high amount of regulation: banking and medical. A possible solution to consider is the idea of “Blue-Green Deployment”, where you build up an copy of the production environment, hardware and all, in parallel to the real production environment. Once everything has been tested to satisfaction, deployment becomes mostly a matter of switching a router.

DevOps is still a relatively young method, partly experimental. As such it has received its fair share of negativism. But you can approach it with an open mind and select the bits that work for your organization and bring them together into a greater whole.