Gathering substantial Technical Debt can weigh your business down, affecting your business agility and value delivery. To keep Technical Debt on balance we have to be able to measure it. In addition, it is important to separate Technical Debt from Quality Issues, the difference being that Technical Debt is intentional and Quality Issues are not. This distinction is important from a business perspective, since Technical Debt gives a trade-off in delivery time, which is of value to the business. A Quality Issue gives no value to the business, it is only a burden. Read more about this in my article “Is Technical Debt Good?”.
Since Technical Debt is intentional, then we are aware it exists and we can record it. At the point of taking on the debt we probably know the cost of the temporary implementation and we also probably have some idea of the cost to fix it at a later date.
Several articles suggest using a Technical Debt Ratio (TDR) to measure Technical Debt. This creates are ratio between the total cost of product development (Development Cost) and the cost of the fix (Remediation Cost). This provides a ratio in the form of a percent.
For example:
Development Cost = 9000h
Remediation Cost 150 h
TDR = (150/9000) * 100 = 1.6%
The reasoning behind using this ratio is that we can relate the debt according to how large the product is, instead of using absolute values such as hours. For example, 100 hours may be a large debt for a small product, but acceptable for a large product. TDR provides a metric for measuring Technical Debt, but its accuracy and reliability as a cost indicator lies in how we calculate the Development Cost and the Remediation Cost.
The business should understand and quantify the total cost of remediation. While TDR may be useful, it is the Remediation Cost that we use to calculate resources or calendar time required to fix the debt. So, we need to keep tabs on a ratio to give us a relative measure, and the Remediation Cost to provide an absolute measure. We will want to apply this not only to Technical Debt but also to Quality Issues. This allows us to assess the cost of quality issues, debt and gain a total set of metrics for technical quality.
Finally, to produce good metrics, we need a way to manage the Technical Debt and Quality Issues so we can track, prioritize and resolve issues.
Development Cost and Remediation Cost
Before we move into how we manage Technical Debt and Quality Issues, we should consider how we define Development Cost and Remediation Cost.
In many articles, I notice that lines of code are used as a way of calculating Development Cost and Remediation Cost. The number of lines is multiplied by a time value to gain an idea of cost. In my opinion, this does not provide an indicator of the cost of development. Firstly, development is not just about code, it’s about the time taken to design, collaborate, test, review, document, etc.… It is very difficult to turn lines of code into time or cost, the process of developing code often means that the developer has several attempts at the code before settling for a given implementation. So, the lines of code represent the final result, not the time or cost spent to achieve the result. We also need to account for many of the soft factors of development, for example, competencies, motivation or work situation, which also affects cost. This is difficult to assess from lines of code.
Note that while lines of code and other metrics (such as complexity, coupling, inheritance) are valuable indicators about the quality of the code, and they may be able to provide a ratio in terms of quality, however, I’m not sure they say much about the cost. They just give a measure of the current state of quality, which is difficult to turn into a real term cost.
TDR uses Development Cost to create a ratio for Technical Debt, where Development Cost is the total cost of developing the product (although different articles appear to have different views on this). The thinking is that we can relate Technical Debt to the cost of developing the product, to create a relative value. For example, a product with a Development Cost of 20 000 hours and Technical Debt of 2000 hours, provides a TDR of 10%, and so does a product with a Development Cost of 1000 hours and Technical Debt of 100 hours.
To provide an accurate ratio, TDR relies on the accurate calculation of the Development Cost, but how do we work out that cost? The finished code in quality and quantity doesn’t say much about the cost which was incurred to create the code. Consider any product, a car for example, we may be able to inspect a car and see that the car is top quality but we cannot know the effort it took to make the car without knowing a lot more about the process and resources used to produce the car. If we cannot estimate the Development Cost accurately, what does that say about the meaning given to the ratio?
The Technical Quality Backlog Since we want to manage issues primarily concerned with quality, it is useful to create a specific backlog for Technical Debt and Quality Issues. In the backlog, we categorize items as Technical Debt or Quality Issues. This can form part of the product backlog if you are working with agile practices, so long as we categorize the items.
When we take on debt or discover a Quality Issue, we create an issue in the backlog with the Remediation Cost. We can, for example, use hours as our measurement of cost. If you are working with agile practices this will be no different from estimating the work required for other items in the backlog. From the backlog, we will be able to calculate the total number of hours required to remove our Technical Debt and our Quality Issues.
At regular intervals, a Technical Quality Review (TQR) is held, for example, every two weeks, or at the end of an agile sprint. The TQR allows the team to review each item in the backlog, update the remediation times as required, and prioritize items for work. The result of the review is an up-to-date Remediation Cost for all items in the backlog. An important aspect of analyzing Technical Debt and Quality Issues is to keep a record of the Remediation Cost over time since this data can be used later to plot trends and analyze technical quality.
Use Investment instead of Development Cost
From a business perspective, we know that Technical Debt and Quality Issues harm product development, and we want to measure how this affects value delivery. Sometimes we want to focus on delivery at the cost of quality, but we don’t want to end up in an impossible situation where the quality of the product is so poor that it seriously affects delivery. Instead of solely looking at the code base, perhaps we can gain more insight into costs if we consider comparing Quality Issues and Technical Debt against investment. We normally have an idea of the current investment placed in the maintenance and development of the product, and should even have access to historical data on investment. It is also reasonable to assume that there is an investment plan or budget for the product. So, if we know how much investment we have available, it is useful for the business to know how much of the investment is required to resolve Quality Issues and Technical Debt. This will also enlighten the business as to how much of the investment goes to value delivery. So, using the information in the Technical Quality Backlog we can start to plot the Quality Issues and Technical Debt against investment. For example, say we have a yearly development budget of 60 000 hours. This would mean we can perform a TQR every month against 5000 hours of investment.
The above graph shows how we can track Quality Issues and Technical Debt remediation costs in hours. This puts the cost in real terms so we can work out how many resources are required to resolve technical quality issues, and how many hours can be allocated to producing value-added features. While this is useful as a graph and analysis of cost, this does not perhaps give a ratio that can be used as a relative indicator. However, we can apply the same formula as TDR but use Investment instead of Development Cost, to gain a Payback Ratio.
Payback Ratio (%) = (Investment Amount / Remediation Cost) * 100
This can be calculated for Quality Issues and Technical Debt, these together provide a ratio for the cost of maintaining technical quality.
The above graph looks pretty much the same as the graph showing Remediation Cost, with the exception that it can be compared to other products to assess the Payback Ratio. If the technical debt or issues are not resolved, they will continue to eat at the budget and affect value delivery. Of course, we can increase investment to reduce our Payback Ratio, but that requires a decision to inject investment. This should raise the question as to why debt has not been paid off within the current budget and require analysis of what is causing the issues.
It is also important to stress, that both a ratio and cost are important measures that complement each other, and both should be tracked.
Tracking Payback
It is perhaps also useful to track how much debt and issues are being resolved. If we see that we are using the investment to resolve debt and issues, but continue to see a rising Remediation Cost, this would be an indicator that serious action is needed to address quality issues in product development.
In the graph above we can see the resolved debt and issues represented as a percentage of the actual investment used to resolve technical quality. We can also see that although an effort is being made to reduce debt and issues, it is not having the desired effect on our Remediation Costs.
Resolved debt and issues can be calculated by logging the working time in the Technical Quality Backlog. This, however, does require discipline from those carrying out the work, so that the actual time used to resolve the issues is logged.
Debt Creep
Debt Creep is what can get businesses into trouble, for example, we might take on what seems like a small debt and then, 3 months later the Remediation Cost for the debt is 3 times the original value. This is Debt Creep, and we need to understand how the cost of our debt change over time, this can also be applied to Quality Issues. Estimating Debt Creep is challenging, as it is difficult to determine just how future development will affect a particular debt. This is why a regular TQR is used to continually assess, update remediation times, and prioritize items for work.
The following are examples of activities that affect Debt Creep:
- Further modification to the Technical Debt (further workarounds)
- Creating dependencies to the Technical Debt (increases the scope of change)
- Changes in the team regarding skills or competence (degradation of knowledge)
- Changes in the designs and implementations surrounding the debt (the rest of the product progresses, but the Technical Debt remains unchanged)
In an attempt to estimate Debt Creep we can use the historical trend we have recorded from the Technical Quality Backlog, and then apply that trend forward in time. As can be seen in the graph we also apply this to Quality Issues.
We can estimate the future debt and quality issue cost by taking an average deviation of the historical results (for example last five results) and plot this accumulatively at several intervals into the future.
This will perhaps give an indication of the Technical Debt/Quality Issue trend based on current information. The weakness in this method is it does not account for the future state of product development, for example, if development enters an intensive period, if more investment becomes available to the project, or if there are plans to replace parts of the product. However, as an indicator, this is useful for drawing attention to what can happen if we do not gain control over our debt and issues.
No Silver Bullet
While it would be nice to have a simple solution to estimating the costs of Technical Debt and Quality Issues, we have to estimate many different factors in a Remediation Cost to gain an accurate estimate. Quality metrics and automated tools may help in assessing the quality of a technical product and provide valuable information to an estimation, but these metrics alone are difficult to translate into real costs.
We can measure Technical Debt and Quality Issues by maintaining a backlog, in the same way, we maintain backlogs for other items in software development. Keeping track of Quality Issues and Technical Debt separately provides the business with an important indicator. Technical Debt is intentional and the business gains value through an early release time trade-off, while Quality Issues are unintentional and provide no value to the business. The cause of Quality Issues is very different to Technical Debt, for example, we may have to resolve problems with processes, organization or skills. Keeping a Technical Quality Backlog up to date with the Remediation Costs for both Technical Debt and Quality Issues provides a basis for estimating the Remediation Cost. This provides a cost for maintaining technical quality based on the ability and judgement of the team managing the product. We can use investment and Remediation Costs from the Technical Quality Backlog, to calculate a Payback Ratio and an absolute cost. This provides the business with an indicator of how much investment is being used to deliver value and how much is being used to maintain technical quality.
Architecturality 