It seems everybody loves metrics and has their own set of favorites. In the field of software development many metrics have come into favor, and later fallen out of favor. Without doubt, they can be used to interesting numbers and pretty graphs; but do they really provide value?
To determine net value, one must examine the costs incurred along with the benefits received to determine the ROI [Return on Investment]. For most corporate situations, this directly translates to money; for others the investment and benefit may be measured in other terms, but the calculation of ROI still translates to a quantifiable measure of “Was it worth it?”.
A big part of the challenge in determining ROI is that it is rarely known what will actually happen, or what the metrics will reveal. In many ways this is similar to the situation with insurance policies, medical checkups, and property inspections. Under the best of circumstances they are all expenditures that provide no actual return on the investment. Under other conditions they are critical to having a manageable situation rather than an unmitigated disaster. Prudent individuals and organizations select a sub-set of all of the available items (hopefully) based on an risk analysis and rarely (if ever) make use of every possible offering.
Selecting a set of metrics and analytics will benefit greatly by performing a set of calculations based on a similar set of costs/risk/benefit criteria. Since these calculations are often predictive in nature, a probability of occurrence needs to be assigned to the elements in the calculations. While the end result will still be a subjective judgment call, having these numbers will greatly facilitate focusing on those items most likely to provide benefit and discarding those that are likely to result in nothing but sunk costs.
Lets break down the costs into a number a few categories :
Next we can look at a breakdown matrix of possible timelines (only a subset are covered at this point)
Of course these are large bucket summaries, and much better information can be gathered by assigning numeric values to the costs along with probabilities and values to the outcomes. It is also recommended to do this over a period of time, with individual numbers assigned to each period [perhaps per iteration/sprint], and the results aggregated over a longer interval [perhaps over a year].
Repeated experience has indicated that the most effective approach is to drive down the costs associated with acquiring and maintaining the data and then gathering as much data as practical. This greatly reduced the chances of lost opportunities (due to lack of data) and provides the ability to perform the analysis as indicated.
In a future post, I will dig deeper into some of the data, analytics/metrics and the ever present danger of misinterpreting the results.