Identifying and managing risks for automatic test systems

Risk is the possibility of suffering loss. In a development project, the loss describes the impact to the project which could be in the form of diminished quality of the end product, increased costs, delayed completion, or failure. This paper discusses the methods and techniques for identifying and managing the risks involved with automatic test systems development.


INTRODUCTION
"First, risk concerns future happenings. Today and yesterday are beyond active concern, as we are already reaping what was previously sowed by our past actions. The question is, can we, therefore, by changing our action today, create an opportunity for a different and hopefully better situation for ourselves tomorrow. This means, second, that risk involves e , such as in changes of mind, opinion, actions, or places .._ rhird,] risk involves -e, and the uncertainty that choice itself entails. Thus paradoxically, risk, like death and taxes, is one of the few certainties of life." [Reference: Charette, R. N., McGraw-HWlntertext, 1989.1

Software Engineering Risk Analysis and Management,
Considering these key risk factors (future, change, and choice), what risks might cause the Automatic Test System (ATS) development project to go awry? How will changes in requirements, development technologies, target computers, and all other entities connected to the project affect timeliness and overall success? What methods and tools should you use? How many people should be involved? How much emphasis on quality is enough?
Risk management is a management practice with processes, methods, and tools for managing risks in a project. It can provide a disciplined environment for proactive decision making to: Assess continually what could go wrong (risks) Determine which risks are important to deal with Implement strategies to deal with those risks Measure effectiveness of implemented strategies

IDENTIFYING RISKS
Risk identification is a process of transforming uncertainties and issues about the project into distinct (tangible) risks that can be described and measured. Identifying risks involves two activities: Capturing a statement of risk -Involves considering and recording the conditions that are causing concern for a potential loss to the project, followed (optionally) by a brief description of the potential consequences of these conditions; the objective is to arrive at a concise description of risk, which can be understood and acted upon.
Capturing the context (circumstances) of risk -Provides a brief, concise description of the condition and consequence of the risk; involves recording the additional information regarding the circumstances, events, and interrelationships within the project that may affect the risk; the objective is to provide enough additional information about the risk to ensure that the original intent of the risk can be understood by other personnel, particularly after time has passed.
Various methods and tools for capturing a statement of risk and the context of a risk are: Independent Verification and Validation (IV&V) is a system engineering process employing a variety of software engineering methods, techniques, and tools for evaluating the correctness and quality of the ATS throughout its life cycle. IV&V is a "tool" to efficiently and effectively manage ATS development risks, and an effective method for reducing costs, schedule, and performance risks involved with ATS development.
IV&V is performed by an organization that is technically, managerially, and financially independent of the development organization: Technically -Requires the IV&V organization to examine those areas of a project, which the program deemed most critical or risky.
Managerially -Responsibility is vested in an organization separate from the development and program management organizations.
Financially -IV&V budget is allocated by program and controlled at high level such that IV&V effectiveness is not compromised.
The paradigm of IV&V is to locate members of the IV&V organization at the ATS development and program management sites where physical presence is necessary for the exchange of information. With the developer, problems are worked at the lowest level with the aim to resolve problems at the earliest possible moment, thus requiring a small impact to correct.
Employing IV&V during the acauisition, develop ment, and maintenance processes can combat ATS development risks by evaluating the correctness and quality of the ATS throughout its life cycle.

ACQUISITION PROCESS
During the acquisition process the IV&V organization will assist program management in scoping the effort of IV&V required for the ATS development project.
The IV&V scope is determined so as to maximize the management of risks while staying within the program's budget allocated to IV&V. This is performed by determining the objective of the IV&V effort (e.g.technical complexity, safety, mission critical, security), the criticality and importance of the ATS, and the IV&V budget, including the required test facilities and tools.
Once these steps are completed, the IV&V effort is tailored to fit the ATS development program's need and a schedule for each IV&V activity is defined along with the IV&V Project Plan. The IV&V Project Plan documents how risks will be managed on the project: the processes, activities, milestones, and responsibilities associated with risk management. It provides the details of each major activity in the IV&V process and how it is to be accomplished, and documents how the IV&V process is to be measured and improved.
Including the IV&V organization at the beginning of the acquisition process can lend itself optimum benefits. One large benefit would be the selection of a reliable, highly qualified organization to develop the ATS. The IV&V organization can assist the program with determining the criticality of the ATS to be developed, the end cost, and the performance history of ATS vendors. This can be accomplished by: Developing a specification for function and performance of the desired ATS, along with defined measurable characteristics whenever possible.
Selecting three or four candidate ATS that best meet the specification.
Developing a comparison matrix that represents a head-to-head comparison of key (critical) functions.
Evaluating each ATS based on past product quality, vendor support, product direction, reputation, etc.
Contacting current users of the ATS and ask for opinions.
In the final analysis, the selection of the ATS development organization should be based on the outcome of this selection process.

DEVELOPMENT PROCESS
As pan of the IV&V effort during the ATS development process, a continuous review of associated documentation, processes, and technical reviews and audits is performed (Phase Independent Activities), and the software/system engineering processes and products are closely monitored for any weak areas (Phase Dependent Activities). Technical reviews and audits are assessed for the acceptability of the development products under review and the extent to which milestones and life cycle goals are met. Assessing the reviews and audits increases the program's visibility into the development of the ATS and provides prioritized recommendations for corrective actions to maintain a focus on the problems posing the highest risk.

Phase Independent Activities
Documentation Review -Review technical consistency; technical adequacy (requirements are unambiguous); completeness; appropriate level of detail; traceability to and consistency with higher level documents.
Process Analysis -Analyze process documentation (Configuration Management Plan, Development Plan, Test Procedures) to verify the processes are being implemented in accordance with the plans.
Technical Reviews and Audits -Attend and participate; present any IV&V issues/concerns; ensure actions are tracked and closed.

Phase Dependent Activities
Requirements Phase -Ensure correctness, completeness, accuracy, testability, and consistency of hardware and software requirements.

MAINTENANCE PROCESS
The maintenance IV&V activities reapply the IV&V activities from the development process, but do so in the context of the existing hardware and software. The maintenance IV&V activities address any modifications, enhancements, or additions to the hardware or software after the ATS has been delivered to the program. Enhancements may include replacing existing ATS computer operating system with a PC. Additions may include adding a VXI card and software drivers to the ATS. If the proper IV&V activities are performed during the development process, any risks identified during the maintenance process should be easily mitigated.
success, enables more efficient use of resources, promotes teamwork by involving personnel at all level of the ATS development project, and provides tradeoffs based on priorities and a quantified assessment. When performed in parallel with the ATS life cycle, IV&V provides for the early detection and identification of risk elements. The program is then able to take action to mitigate these risks early in the life cycle, reducing development and maintenance costs.