DevOps has come a long way from ‘it’s the code’ or ‘it’s the environment’ puzzles. As the concept became familiar, thought leaders like Qentelli are giving emphasis on closing the DevOps feedback loop (listening and responding). Further course of CI-CD requires near real-time detailed insights for each phase to make improvements in the application delivery. Properly administered feedback results in offering new dimensions to act upon such as new events, feedback, data, customer demands or variables for businesses.

In this article, we will talk about signs for missing feedback loop0 and how-to plug-in feedback at all stages of DevOps. Without further ado, let us get started –

Observe the obvious

Teams assume few things are obvious in DevOps like making speed the goal, creating too many toolchains, keeping security, and testing at the end, but the assumptions bust when things go wrong. Observing the obvious signs of testing, tool standards, and unified communication helps in closing the feedback loop to improve software metrics.

For instance, a top game development company assessed that industry would witness the mobile revolution in second decade of 2000s, but it came early. They were the market leaders but witnessed a major fall as none of their games were mobile-ready. There was a gap in identifying market forces and how it impacted internal business plans. The application development space is changing rapidly that what holds relevance a year ago or even five minutes before is irrelevant now. Teams not observing signs are leaving behind in the race.

Organizations should look for such obvious gaps to identify broken DevOps loop. These signs can serve as a beacon to navigate the path of DevOps to avoid mediocre software deliveries.

#1 –Treating DevOps and Business Separately

The meaning of DevOps does not convey properly as it says Dev and Ops–combining development and operations. But in true essence, it brings business into software lifecycle picture. If your business teams are not part of your next feature release or prioritizing tickets, it is a red flag for DevOps program. Business teams are treasure of information about customer demands and reviews. Engaging them in the software development lifecycle will fill in the gap of serving customer needs and releasing and fixing the right features. Business teams can translate Developed versus Released features, Automation, Security, Compliance & Governance integration into customer acquisition and retention costs, resulting into right rankings for development.

It’s the Software Economy. Application releases decide how the balance sheet and P&L statements of organizations look like. Early feedback from management helps in tailoring application delivery as per the business goals.

#2 – Unidentified Communication Barriers

Communication and collaboration are the true enablers of DevOps. Prescriptive rules do not drive communication, instead it is a philosophy for cross-functional teams to work together for software deployment, support, and maintenance in production and beyond that.

As per the Pulse of Profession Report 2018, inadequate or poor communication accounts for the top 5 reasons for the project failure in any organization.

Organizations constantly reinforce different modes of communication such as stand up meetings, smiley boards, chat rooms and open spaces. But does it create transparent and open communication culture? The answer is No!

The goal of DevOps is to have development and operations work together collaboratively for better application releases. But if your developers still throw surprises at Infra/Ops team this is a sign of an open DevOps loop. If there are many instances like this, it is an alarm to fix communication gaps. Management and other technology leaders must proactively act to establish communication rhythm and sell DevOps across the organization.

#3 – Late identification of issues 

Organizations are trying hard to identify and resolve errors in the early stage of development lifecycle but are achieving limited success. Teams identifying bugs at later stages of development is a sign of missing feedback.

It is unavoidable to commit mistakes but spotting, preventing, or resolving them in lower environments is possible using feedback. DevOps teams should align testing with the development to catch errors in an early stage and avoid any mismatches downstream.

The formal DevOps feedback loop should reflect customer perception of poor quality. Say, for example, customers perceive hundred defects that does not impact user-experience as great rather ten issues impacting usage of important features as poor quality. Tracking customer perception of quality and duly developed testing strategies exploits the feedback around perceived quality. This helps in resolving bugs as per the customer perception of quality. Above all customer experience is Emotion, Convenience & Outcomes and not underlying application technology or infrastructure performance.

#4 – Lack of automation

For most of the organizations, starting automation initiative is a fine mess. If your most of the development processes are still manual, this is a sign of missing DevOps loop.

Automation is heavily associated with DevOps. DevOps stresses on creating a highly automated environment from planning to production and post-production monitoring. Lack of automation in DevOps breaks the feedback loop as–

• Errors are detected in later stages
• Increased risk of a deployment causing downtime affecting application
• Delayed roll-back in case of issues

Non-automated processes cannot be put under a continuous frame of capturing feedback from delivery pipelines and set up notification triggers. Based on the processes, applications and architectures, organizations need to identify critical steps from where important and timely feedback distributes to the team.

#5 – Faster releases but no insights about critical KPIs

DevOps adoption is increasing, but DevOps improvement has reached a plateau stage. If the only improvement area in your releases is deployment speed, then this is a sign of missing DevOps feedback. Faster releases with compromised quality are giving sub-optimal benefits to teams. Improvements in Customer centric and quality metrics such as number of production defects, Mean time between failures, Mean time to recover and Cycle time show about the efficient processes and time taken to provide actionable feedback to developers about their codes.

The more, the time taken for feedback, the more time is wasted in delivering value to customers. The feedback can be faster if developers are running unit tests early in the pipeline saving time for them.

The feedback should include customer experience metrics tied to improve overall DevOps metrics. Customer experience metrics need to be shared across the organization, reviewed by a set of identified cross-functional team members to discuss how organizations can work on them to meet most of the end-user expectations. Cross-functional teams should create a product roadmap to break user stories small enough to release them fast and provide value to users. Feedback and involvement from business teams avoid backlogs of non-value-added features later in the development.

#6 – Monitoring should extend to feedback loops        

In DevOps, just monitoring without taking relevant actions is a sign of missing feedback loop. Feedback is the extension of Monitoring. Monitoring infrastructure, applications and logs do not give a clear and comprehensive picture of DevOps as a whole.

In current times, DevOps teams are monitoring different areas of DevOps pipeline resulting into a siloed picture of the DevOps pipeline. The way to DevOps feedback starts with alerts and monitoring extending into feedback for all the areas of DevOps pipeline and recommended improvement areas. Some areas to track for creating feedback in a simple delivery pipeline are–

Commit Stage – Commit notifications, build fail/pass results, unit test results and code metrics

Testing Stage – Performance testing results, having threshold criteria for performance, and comparing them

Deploy Stage – Production performance monitoring and deployment reports.

Qentelli’s Intelligent Analytics Dashboard TED tracks the entire DevOps lifecycle and provides actionable insights for potential flaws. TED is technology and tool agnostic and can be integrated with diverse toolchains used by organizations.

The Nature of DevOps Feedback Loop

The right DevOps loop must bear these characteristics–Faster, Relevant, Actionable and Accessible. Engineering teams need to set rules for acting on different feedback and own the complete code quality checked in by engineering teams.

Remember the classic example of SlideShare DevOps failure where one small reorganization in database brought whole site down for over 60,000 users. Had it been a timely and properly targeted feedback at the developer level they would have avoided the whole situation?

Another example comes from IBM in early 2000s. IBM started their DevOps journey to speed up their software releases. But unable to achieve so, they assumed that the problem is with code deployment and automated it. Still, did not achieve results. Later they discovered the problem was with operational and development environment. Again, establishing a relevant DevOps feedback loop would have given the expected results.

This is the early 2000s scenario, companies failed, changed strategies and now, they are maturing in DevOps. But still things that seem obvious go wrong with the broken feedback loop. The DevOps measurement and feedback loop places a set of rules to roll back to the original state or not to introduce any new changes without reaching the threshold of the test code coverage. A customized feedback loop and process is necessary for every organization to act as a control center to alter the course early when things go wrong.

How to set up a continuous feedback loop for DevOps

Continuous Feedback Loop–Delivering Quality Application at Light-speed

DevOps teams run into a bottleneck when feedback is not timely, pulling the whole release timeline backwards. A formalized feedback loop brings Agility, Quality and Customer-centricity.

The concept of monitoring has been injected into the phases with a wide variety of tools, but these tools do not provide an end to end insights about pipeline health. The extension of this monitoring is broader continuous feedback loop that drills down to the overall pipeline health. DevOps feedback spans across all the phases such as Planning, Development, Integration, Testing, Deployment, Monitoring.

• Continuous Planning – Post-deployment application usage, customer reviews, incomplete user stories, user-behavior.
• Continuous Development – New user stories, new compliance, and security requirements, build tests.
• Continuous Integration – Build fail/pass, test environment, latest code version, artifacts.
• Continuous Testing – Root cause analysis of tests failure. Result of mini regression suite.
• Continuous Deployment – Application feature and usage, end-user interaction and the areas of improvement.

Constructing Continuous Feedback Loop

The proactive monitoring across stages like application health, log monitoring, code quality monitoring, etc is a thing of the past. The modern DevOps team requires platform/solution to have a panoramic view from planning to post-deployment. The basic premise of DevOps feedback loop is simple–Collect real-time information based on the code movement in the pipeline, provide this information to the teams responsible for it, establish processes to fix issues and improve continuously. This should be introduced focusing strongly on customer journey. On a broader level, DevOps feedback loop requires–1. Setting up Quality Check Gates 2. Requires an Intelligent Engineering Dashboard tools such as The Engineering Dashboard (TED) 3. Establishing and adopting processes to ensure immediate remediation of issues.

1. Setting up Quality Check Gates

Building DevOps feedback loop requires setting up quality check-gates at each level with specified threshold at each stage for code acceptance. This requires brainstorming from the various teams such as customer service, product managers, business analysts, QAs, Infra teams, Developers. Customer experience metrics and industry statistics can also be helpful in determining these threshold numbers for the first time. Quality gates ensure–

• Code shipped into production have the highest quality standards including security, compliance, and governance
• Code is halted if a certain threshold is reached and respective teams are notified to fix the issues
• Better release confidence to developers
• Better collaboration between teams
• Quick roll-back strategies to the previous stage

For organizations starting to build their feedback mechanisms, it is a work-in-progress to get the specific threshold at each phase correctly in one shot. The key is to remain progressive and observant in recording and iterating them.

2. Intelligent Engineering Dashboard tools such as The Engineering Dashboard (TED)

There are readily available tools in the market such as NewRelic, HP LoadRunner focused on the single segment of the DevOps pipeline like application performance monitoring, load testing. IT leaders have low visibility into the quality and health of DevOps pipeline.

The acceptance of DevOps has created an over-importance of metrics and how to collect them instead of interpreting for wowing customers. Teams do not look beyond the defined metrics to measure the efficiency of processes, tools, projects, and teams. This creates a gap in identifying new and innovative metrics. CIOs are seeking products/solutions to examine and improve efficiency and quality of the software.

Continuous feedback loop requires tools like TED, a home-grown accelerator of Qentelli to provide complete insights around the health and quality of the DevOps process. This gathers information right from planning phase to code commit till production and end-user monitoring.

TED collects near real-time data from different quality check gates and integrates with existing tool stack. Teams can configure alerts and notifications for different teams to act upon them. TED creates transparency by showing the required information (depending on your team structure and rules for accessibility of information) to the right people. It shows the relevant information to developers so they can fix the code, or infra team so they can see the downtime, or business analysts about the requirements gathering for the upcoming sprint.

3. Adopting processes to ensure immediate remediation of issues

Continuous feedback loop ensures identifying errors earlier and if they slipped to the production responding faster to resolve them without customers getting a hint. This also involves understanding the root and cause of issues and proactively avoiding them. The third aspect of DevOps feedback involved major changes around culture. Amplifying DevOps feedback loops require creating a dynamic and agile culture to have comfort of changing the course whenever required.

Feedback maximizes DevOps returns

The well-administered feedback loop revolutionizes complete app lifecycle from planning to deployment all integrated to see problems before they go live. Read how we did for a Premium Business Networking Portal.

Born-digital decacorns are leading the ‘releases per day’ game but every organizations can realize the true benefits of DevOps with the continuous feedback loop. Intelligent feedback help businesses address their persistent quality loopholes and build dynamic solutions.

Qentelli’s approach of establishing intelligent tool like TED benefits organizations to improve DevOps processes using data captured across software lifecycle. Are you ready to maximize your DevOps efforts? Start the conversation with Qentelli at info@qentelli.com to deliver value to your customers faster.