Cost framework for COTS evaluation

We focus on factors that the user should consider when deciding whether to use COTS software. We take the approach of using the common denominator, cost. This is done for two reasons: first, cost is obviously of interest in making such decisions, and second a single metric (cost in dollars), can be used for evaluating the pros and cons of using COTS. The reason is that various software system attributes, like acquisition cost and availability (i.e., the percentage of scheduled operating time that the system is available for use), are commensurate quantities. That is, quantitatively "a low acquisition availability". These units are not multiplicative. However, if it were possible to translate availability into either a cost gain or loss for COTS software, we could operate on these metrics mathematically. Naturally, in addition to cost, the user application is key in making the decision. Thus one could develop a matrix where one dimension is application and the other dimension is the various cost elements. We show how cost elements can be identified and how cost comparisons can be made over the life of the software. Obviously, identifying the costs would not be easy. The user would have to do a lot of work to set up the decision matrix but once it was constructed, it would be a significant tool in the evaluation of COTS. Furthermore, even if all the required data cannot be collected, having a framework that defines software system attributes would serve as a user guide for factors to consider when making the decision about whether to use COTS software or in-house developed software.

Certainly, different applications would have varying degrees of relationships with the cost elements. For example, flight control software would have a stronger relationship with the cost of unavailability than a spreadsheet application. Conversely, the latter would have a stronger relationship with the cost of inadequacy of tool features than the former. Due to the difficulty of identifying specific COTS-related costs, our initial approach is to identify cost elements on the ordinal scale. Thus, the first version of the decision matrix would involve ordinal scale metrics (i.e., the cost of unreliability is more important for flight control software than for spreadsheet applications). As the field of COTS analysis matures and as additional data is collected about the cost of using COTS, we will be able to refine our metrics to the ratio scale (e.g., the cost of unreliability in COTS systems is two times that in custom systems).
The cost elements for comparing COTS software with in-house software are identified below. This list is not exhaustive; its purpose is to illustrate the approach. These elements apply whether we are comparing a system comprised of all COTS components with all inhouse components or comparing only a subset of COTS components with corresponding in-house components. Explanatory comments are made where necessary. Mean values are used for some quantities in the initial framework. This is the case because it will be a challenge to collect any data for some applications. Therefore, the initial framework should not be overly complex. Variance and statistical distribution information could bc included as enhancements if the initial framework proves successful. These quantities are the number of failures per year that .cause loss of productivity and availability of the software system.

Cost Elements
In some applications, some or all of the above quantities may be known or assumed to be constant over .the life of the software system. Using the above cost .elements, we derive the equations for the annual costs of the two systems and the difference in these costs. In the .cost difference calculations that follow, a positive -quantity is favorable to in-house development and a 'negative quantity is favorable to COTS.

Cost of Acquiring Software
Difference in annual cost = C,Q) -CiQ)

'Cost of Upgrading Software
Difference in annual cost = U,Q) -UiQ)

Cost of Software being Unavailable for Use
Annual cost of COTS software being unavailable for use = ( l-Ac(j)) * P(j).
Annual cost of the in-house software being unavailable for use = ( 1 -Ai(j)) * P(j).
Then, TC,, total difference in cost in year j, is the sum of (I), (Z), (3), and (4). Because there is the opportunity to invest funds in alternate projects, costs in different years are not equivalent (i.e., funds available today have more value than an equal amount in the future because they could be invested today and earn a future return). Therefore, a stream of costs over the life of the software for n years must be discounted by k, the rate of return on alternate use of funds. Thus the total discounted cost differential between COTS software and in-house software is: In this initial formulation, we have not included possible differences in functionality between the two approaches. However, a reasonable assumption is that COTS software would not be considered unless it could provide minimum functionality to satisfy user requirements. Thus, a typical decision for the user is whether it is worth the additional life cycle costs to develop an in-house software system with all the desirable attributes.