As I started writing this piece, I talked to Software Engineers and DevTestOps team members with varied years of experience in the industry. This gave rise to two schools of thought for what to measure in Quality Engineering and which model fits the bill. The first school stresses collecting every data possible and then analyze for the correlation while the second talks about collecting only relevant software engineering metrics.
I believe both have their pros and cons. The first school is helpful to find innovative software engineering metrics and new correlations, the second is the base to remove the monotonous patterns leading to the same defects and downtime.
As a leader with the responsibility to build ‘world-class’ digital solutions, how many times did you daydream about getting those magical Software Engineering metrics? Quite often, Right! While leaders are caught with big visions, engineering teams are consumed in engineering metrics and model discussions for measuring Software Quality.
This article will draw references from the second school of thought. We will further discuss what are the hot metrics and models in 2021. It is very important to understand why and how to choose the right metrics and models for Software Engineering.
Build consistency in Development – Applying Software Engineering metrics brings consistency in processes, tools, and technologies. Common definitions of Defects, Issues, Acceptable Criteria, Scope Changes in testing and Coding, establishes a common line of communications for teams to adopt Metrics-Driven processes.
Automate Data Collection Process – Your organization can apply the relevant model and capture engineering metrics based on data lying in your systems to discover trends, drive improvements and uncover underlying causes of defects. There are various tools available for automating the data collection across the entire software lifecycle. Understanding these data trends helps in identifying the right metrics and improving them to achieve higher quality products.
Data for Distinguishing Defects – Do not treat all defects equally. So how do your teams decide which defects are worth their first-hand time and efforts? The data collected in your automated systems can build correlations and find out which defects can cost more to the company (in terms of money or brand value). The automated systems give faster feedback for teams to tweak their QE Practices and achieve threshold metrics to get the best quality product out in the market.
Software Engineering Models and Metrics
Measuring in digital ecosystems requires quality models and related metrics systems to give the right value about the processes, projects, and products. Our Premium Business Networking client began to focus on the quality of applications and developed a one-stop dashboard to give insights about on-demand environments for Testing and Build Promotion. A finance & consulting firm is spending 50% lesser time on obtaining Test data, improved Quality Assurance cycles by 10 times with 5 times faster test execution.
Check out similar Client Stories.
There are tons of Software Engineering and QE models. We will discuss some of the models and approaches that are still relevant and used by organizations.
1. Quality Management System (QMS) – As per definition, QMS is defined as a formalized system that documents processes, procedures, and responsibilities for achieving quality policies and objectives.
2. Total Quality Management (TQM) – TQM brings in the new facet of customer satisfaction along with managing processes, procedures, and responsibilities. The eight pillars of TQM are Customer-focused, Total Employee Involvement, Process-Centered, Integrated System, Strategic and Systematic Approach, Continual Improvement, Fact-Based Decision Making, and Communications.
3. Kaizen Model – It relies on the Continuous Improvement in Continuous Integration and Development cycles. This applied to projects, products, employees, and organizations to improve business practices.
4. Six Sigma – It talks about removing the causes of defects and minimizing variability to establish consistent processes without any deviations or errors.
5. Voice of Customer (VOC) – VOC is a process to capture the customer feedbacks, requirements, and integrate them into existing processes to start and scale the improvement journey.
You need to figure out what works best for your teams, products, and organization. It can be a single model or a combination of two or more models. Moving on to Engineering Metrics, we will discuss three major buckets of Engineering Metrics – Project, Product, and In-Process.
Best Practices alone will not guarantee quality in your applications and can even result in late-identified and high costs issues if not applied correctly. The good idea is to catch the defects as early as possible and capturing the right metrics at the Project level ensures Quality Assurance from the beginning.
Right Project level metrics captured from the beginning can help testers, managers, and executives to measure and improve day-to-day Development and QA operations. Some popular Project-level metrics are:
1. Requirements and requirement coverage: Requirement metrics measure the business requirements against what your developers need to achieve at the outset of production. Outlining requirements at the beginning helps teams in setting their course for Development and Testing. Additionally, software teams get clear direction about development and testing requirements to maintain their trajectory as the development picks pace.
2. Cycle Time: It is a metric borrowed from lean thinking and manufacturing disciplines, but it is very much relevant for measuring the efficiency of the software development process. Cycle time is the time taken from when an item is accepted and picked by the dev team through to go-live; Lead time also includes the time that an item is pushed into the backlog and is waiting to be picked.
Cycle time says a lot about your team coordination and collaboration, code reviews, tools used, CI-CD integration, automated tests, and deployments, etc. An optimized cycle time reflects an optimized software development process, a team with a shorter cycle time can be termed as more efficient than one with a longer cycle time.
3. Productivity: The productivity metric considers two factors 1. Size of the software you deliver and 2. Efforts to build the software (Cost, time size, time taken and other factors to be considered). Measuring productivity metrics helps in calculating the right team size for your project, finalizing cost-effective tools and techniques. This metric helps in cost-savings for small and medium-sized companies and large organizations can form the right team size to act agile.
4. Rate of Requirements Change: Requirements framed at an early stage are bound to change and it is a part of the development process. Most of the teams are attuned to the process but ever-changing requirements impact the estimation, accuracy, release time, cycle time, and quality of the product. Small changes do not create much disturbance in the ongoing release cycles, frequent changes can be frustrating and time-consuming for developers leading to a bad quality product. Measure your rate of requirements change and aim for the highest stability possible in the requirements.
Few other metrics are considered crucial depending on business/project objectives, teams can measure and monitor them. Some of them are – Estimation Accuracy, Staffing Change Pattern, Scrum Velocity, Cost, Scope, Risk.
Product Quality is directly associated with Customer Satisfaction and Adoption. Software Engineering Metrics are a cross-cutting dimension in terms of making a good product based on continuous feedback, improvement, and iteration. Due to DevOps and other technologies, Product Development is catching up faster than usual, making it imperative for teams to measure Product Quality continuously.
Here are some Product Metrics that you can measure to 'build-better' products:
1. Reliability (Defect Density): Reliability or Defect Density metric helps your team to understand the code quality per unit. The lesser defects per unit, the higher the product quality. Measuring reliability metric gives the confidence of better quality from the beginning as developers can receive immediate feedback on how their codes are functioning. The defect density per thousand lines of code shows how effectively your developers are coding and how processes like peer code reviews and unit testing are working.
I would like to briefly touch on some important Reliability Metrics:
Mean Time to Failure (MTTF)
MTTF is the time interval between the two-consecutive system/application failures. MTTF is consistent for systems with large transactions. Teams with reduced or consistent MTTF are designated to have stable systems, meaning stable code and new changes.
Mean Time to Repair (MTTR)
MTTR is the average time to track the error, identify causes, and fix it. MTTR shows the agility and maturity of teams. It is an effective measure of Operational Resilience or how quickly systems bounce back to a normal state.
Mean Time Between Failures (MTBF)
This is the average time between repairable failures of an application/system/product. This can be used to track both the availability and reliability of a product. The longer the time between failures, the higher the system reliability.
2. Availability: In some studies, Availability is considered as the most important Customer Satisfaction Metric. And we can understand why, No one wants a system to crash when a user is in the middle of filing the new financial reports on the last day of the Financial year or process payment of the next insurance premium on the due date.
Reliability metric can be collected using internal systems and resources for the latter one, your team can adopt three approaches: collect the data directly from a small set of customers, collect the data via your customer service process, and conduct special customer surveys. Availability metrics are directly proportional to your Customer Satisfaction and hence measuring them can give teams real break-through if they are looking to achieve that magic number of 99.9999%.
At Qentelli, we have achieved this magic number for some of our clients by employing Availability Engineering in building digital products and applications. Talk to us about your next Quality Engineering initiative
3. Usability: The usability metric measures the degree to which end-users use the product effectively, efficiently, and satisfactorily.
As you continue to release new features in digital products and applications, there is an increased need to assess Product Usability. If as a technical leader, you want to present numbers about how many customers are liking a newly released feature, this is your metric.
This is linked to your User Interface and how quickly users can adapt to its usage. For example, if a social media tool allows a user to schedule its posts with 7 clicks, the software is simply not optimized for usability. A similar tool lacking in showing consolidated clicks, views, and reactions on social media post lacks user-friendliness in presenting the information.
You can track Usability by collecting data such as User Analysis, Site Visits, Task Analysis, Usability Benchmarks, and Designs. The right approach for your team is to define Usability targets from Day 0 and work towards achieving them.
4. Overall Customer Satisfaction: Your Product Metric dashboard is not complete without an Overall Customer Satisfaction metric. This metric gives the collective picture of your product, all other Engineering metrics, your team development efforts, uptime, usability, etc.
The ideal mix and match of data sources lie in your product user-base and the nature of users. Your team can determine customer satisfaction by customer surveys, a customer reported defects/month and resolution time, nature of defects, collecting data from the customer service team, and social channels.
A good customer satisfaction rate means customers trust your product. For a Digital Organization operating in a competitive market, failure to address the falling customer satisfaction rate can lead to a loss of market share. Metric can help your team in pinpointing the exact point of failures and fall in customer satisfaction and resolve the issues to alleviate satisfaction graphs. Faster the feedback, the faster the resolution leading to an increase in satisfaction rates.
Becoming ‘Metrics-Driven at the Core’ can create more sustainable value for your Engineering initiatives. Customers are updating expectations for organizations everywhere. Any organization that wants to sustain the digital race must accommodate the new rules of engagement with digital solutions. Companies like yours need to be metrics-driven, proactively plan for acting on metrics to be more profitable than peers.
Talk to us about what Engineering Metrics you are tracking, and we can help you in measuring new metrics relevant for your projects and teams.
Tracking in-process provides strong monitoring and control to ensure that your teams react and respond to deviations at an early stage. Additionally, teams can alter processes producing regular defects to bring more control over software quality.
1. Reliability Growth Pattern: You have a new CI-CD process with integrated testing and security tools. As you are confident about these processes and tools, your team reports longer system testing time and more failures. So, what has gone wrong? Reliability Growth Pattern can pinpoint the flaw in the process.
This metric measures and predicts the improvement of reliability programs in the testing process. The growth model represents the reliability or failure rate of a system as a function of time or the number of test cases.
Your team should benchmark plotted versus testing time with the magnitude of spike related to significance and volume of changes. If there is an increase in plotted time versus actual time, as an IT leader you might want to revisit the testing processes and tools.
2. Defect Patterns During Testing: The type of defects plotted versus actual found during testing is an important metric to further evaluate your release time and thus it is an important in-process metric.
These defects must decrease as the product moves towards release. If it remains the same, re-look into the testing coverage, is it sufficient for the current release or does your team need more testing?
If the defects have gone substantially high, your testing effectiveness must increase. Bring in Automation or other sophisticated tools based on team, project, and budget. This metric will help you win the buy-in from the management team for Test Automation.
3. Backlog Management Index: Backlog Management Index is the accumulated difference between defects closing and defects arriving. Tracking Backlogs are important for assessing testing progress and customer rediscoveries of the defect.
A lot of factors influence Backlog Management Index like if your team is working on a legacy system, there are high chances of discovering more defects and taking more time to resolve them. With the pressure of new releases, this backlog can increase over time.
Your process management approach should be three-pronged: With an effective and strong testing plan in place, ramp-up your testing processes to achieve early and faster feedback. Aim for targeted improvements by analyzing data from the trends and their timings. Manage backlog reduction by prioritizing the defects based on severity and priority. Try to get the values within the pre-determined levels.
These are in-process metrics to bridge the long-term gap between processes. Every project is unique, and we recommend working with QE specialists in figuring out the right metrics to track your development process.
One more SLA metric is Percent Delinquent Fixes that determine how long does it take to fix the defect by the number of fixes delivered.
A note for Management Team
As a positive push to software engineering metrics implementation, leaders can promote the piecemeal approach. Ensure your Software Engineering champions foster transparency and achieve buy-in from Developers. Empowering your team with automated tools and learning about the latest technologies will go a long way. Do not fall for ‘Flight Magazine Syndrome’ and expect results to flow in, always look for opportunities to reward team rather than punishing approach (a usual mindset with Metric Implementation).
After working with more than 100 clients, our QE consultants and Innovation team developed a product, The Engineering Dashboard (TED) to bring the right metrics together for better Quality Management. Know more about TED. TED tracks all metrics including Process, Product, and Project and presents a consolidated view for stakeholders enabling faster decision making.
Want to discuss more on Metrics concerning your development process and Quality Engineering plan. Feel free to write to us firstname.lastname@example.org