Essentials for team based rehearsals and the differences between Earth Orbiting and Deep Space Missions

The mission readiness environment is where spacecraft and ground systems converge to form the entire as built flight system for the final phase of operationally-themed testing. For most space missions, this phase starts between nine to twelve months prior to the planned launch. In the mission readiness environment, the goal is to perform sufficient testing to exercise the flight teams and systems through all mission phases in order to demonstrate that all elements are ready to support. As part of the maturation process, a mission rehearsal program is introduced to focus on team processes within the final flight system, in a more realistic operational environment. The overall goal for a mission rehearsal program is to: 1) ensure all flight system elements are able to meet mission objectives as a cohesive team; 2) reduce the risk in space based operations due to deficiencies in people, processes, procedures, or systems; and 3) instill confidence in the teams that will execute these first time flight activities. A good rehearsal program ensures critical events are exercised, discovers team or flight system nuances whose impact were previously unknown, and provides a real-time environment in which to interact with the various teams and systems. For flight team members, the rehearsal program provides experience and training in the event of planned (or unplanned) flight contingencies. To preserve the essence for team based rehearsals, this paper will explore the important elements necessary for a successful rehearsal program, document differences driven by Earth Orbiting (Aqua, Aura, Suomi-National Polar-orbiting Partnership (NPP)) and Deep Space missions (New Horizons, Mars Atmosphere and Volatile EvolutioN (MAVEN)) and discuss common challenges to both mission types. In addition, large scale program considerations and enhancements or additional steps for developing a rehearsal program will also be considered. For NASA missions, the mission rehearsal phase is a key milestone for predicting and ensuring on-orbit success.


INTRODUCTION
NASA and its industry partners develop space missions that expand knowledge of the Earth and its environment, the solar system, and the universe. As part of the mission readiness phase, each project works toward a fully tested and verified flight system consisting of mature spacecraft and ground elements. During this part of the development life cycle, each project is presented with a unique set of challenges stemming from: mission class, cost, launch window, ground system maturity, workforce experience and operational complexity.
These obstacles manifest themselves in different ways across missions but need to be successfully addressed to achieve mission success. To ensure the final flight system is prepared to support flight operations, a mission readiness test campaign is realized to operate spacecraft and ground interfaces as a system. As a sub-section of this campaign, a Mission Rehearsal (or Operations Readiness Test) test phase is conducted that exercises personnel and operational processes to demonstrate the flight team's ability to meet mission objectives, within the final flight environment. For NASA to ensure mission success and maintain its leadership in space, the agency is committed to successful mission readiness campaigns and ensuring that all missions are trained to operate at the lowest acceptable risk. Figure 1 below shows NASA's risk posture for different mission classes based on payloads.  [1] https://ntrs.nasa.gov/search.jsp?R=20150019650 2020-05-08T15:54:48+00:00Z

ESSENTIALS FOR ANY MISSION REHEARSAL PROGRAM
As will be described later, there are several key factors that are unique to developing a Mission Rehearsal (MR) program based largely on mission type: Deep Space or Earth Orbiting. This section discusses those program elements that are common and independent of mission type. These are: test case development; test objectives; success criteria; command, communication and control authority; roles and responsibilities; and the process for executing an exercise.
A Rehearsal Director is an independent agent for the NASA Project who typically reports through the Mission Manager up to the Project Manager. As an independent entity, given the rehearsal exercise number agreed to by the various stakeholders and approved by Project Management, the Rehearsal Director and his team identify a set of test cases to rehearse based on critical events in the mission timeline. Critical events are usually propulsive maneuvers, instrument commissioning, any first time events or coordinating activities that involve multiple teams. Specific critical events are: -Launch -Solar Array deployment, orbit insertion, planet flyby; -Delta-V, Trajectory Correction Maneuvers, Gyro/Star Tracker calibration; -Instrument power on, deployment and initial configuration/commissioning; -Collision Avoidance process; -Shift handovers Next, a set of high level objectives to exercise are introduced and modified for clarity with all stakeholders. Some core objectives are: -Exercise the mission timeline using operational processes and flight products; -Exercise the mission timeline change process; -Train on communications system and exercise the voice protocols; -Exercise team interaction during nominal and contingency situations; -Exercise anomaly identification and resolution process; -Train on the flight configured ground system; -Exercise with system hardware configurations As each rehearsal exercise is defined, unique objectives will become clear. As a Rehearsal Director, briefing the stakeholders on new objectives or clarifying the core is necessary in order for teams to properly prepare and support the exercise. Communicating early and often during the rehearsal meetings guarantees everyone is in synch with rehearsal plans.
Once objectives are cross referenced to test cases, the success criteria or demonstrated proof that must be witnessed for each objective to be met is further detailed and documented. These criteria are used to evaluate how well the flight team performs during the rehearsal exercise and is used to determine whether an exercise needs to be rerun. In this paper, when we refer to "flight team," we mean spacecraft and payload engineering teams, government managers, ground controllers, flight operations and mission planning and scheduling personnel.
In preparation for the first exercise flight approved products and a mature mission timeline should be available. In addition, an agreed command chain needs to be established. Who will approve flight products for uplink; how changes will be communicated and approved; how the mission timeline is modified; and how flight activities are planned must be understood and potentially developed as part of the mission rehearsal program.
In conjunction with the command chain, the roles and responsibilities to be fulfilled by all flight team leads, subsystems and major stakeholders need to be defined and agreed to. An exercise needs to establish this early or exercises will be impacted by having many stakeholders behaving or following non-flight operation processes. This item is always complicated to manage as I&T personnel merge into flight operations, and continue to follow processes that lead to operational risk in flight.
A forum in which to discuss rehearsal development and preparation activities is highly useful to focus the flight team on rehearsal exercises. This is comprised of all the major stakeholders to discuss preparation activities and readiness to support. A smaller Rehearsal Anomaly Team (RAT) is also established to develop anomalous scenarios for evaluating team processes during contingency operations.
A process by which to brief the flight team of an upcoming rehearsal (e.g. Kickoff) is used to communicate details of the exercise and how the Rehearsal Director intends to manage the exercise in case of unforeseen events.
All these elements, the key Mission Activities, Objectives, required maturity of products, Roles and Responsibilities, and the development process are all captured in a Mission Rehearsal Test Plan which becomes the governing document from which to prepare each exercise and to further develop individual exercise plans.

MISSION READINESS PARADIGMS
Given enough time and money, flight system ground testing would never end! It is universally accepted that the more you test, the more you learn and thereby reduce risk. It is always beneficial to test hardware, software, flight products, and interfaces, in all expected flight configurations, and analyze results, assess changes, then re-test or design new tests all in the spirit of avoiding surprises in flight. However, due to schedule and budget constraints, the development of an efficient spacecraft integration and test (I&T) schedule is needed to balance the amount of testing across all element levels against on-time spacecraft delivery to the launch pad. As a result, the amount of mission readiness testing (and strategic placement in the schedule) is bound by the launch readiness date.
Another factor in developing an Integration and Test (I&T) (aka, Assembly, Test and Launch Operations -ATLO) schedule is mission classification. For large missions (class A, B) [1] with a low level of acceptable risk, the amount of testing generally tends to be abundant and extensive with many groups involved ensuring all aspects of the spacecraft and ground elements are fully verified.
Additional independent groups (e.g. NPP Systems Integration and Test, Aqua/Aura Mission Readiness Test Team) are involved to ensure testing across the elements are re-tested and verified before the additional compatibility and operational system level testing occurs. In terms of Mission Rehearsals, there are more formalized and rigid plans put into place for the amount of pre-tests (i.e. dry runs) before the teams are exercised and in the level of support for coordination with the elements. In general, for large missions the number of planned Mission Rehearsals and total days available to perform operational exercises is higher in number than on programs with a higher level of acceptable risk (class C, D) [1].

Differences between Deep Space missions and Earth Orbiting missions and regarding I&T and ATLO
Most Deep Space missions have a fixed launch window, which tends to scale down the scope of mission readiness testing. Also, due to schedule constraints, Deep Space missions typically have to develop an incompressible test list, that is, a list of all tests that must be performed successfully before launch. The remaining tests would then be performed as schedule allows.
For Earth Orbiting missions, launch date flexibility may allow for more generous mission readiness test time. Furthermore, Earth Orbiting missions typically adhere to a more standard critical path method which allows for more generous testing opportunities.

CHALLENGES BY MISSION TYPE (DEEP SPACE, EARTH ORBITING)
A. Considerations due to: ground system development, system maturity, program priorities Experience across several NASA mission types (Deep Space and Earth Orbiting/Large and Small) identifies the approach and philosophy toward ground segment development as a big driver on challenges that will be encountered during Mission Rehearsal development. A simple upfront decision to use the same core ground system for Telemetry, Command and Control (TCC) in the I&T and flight operations environments pays huge dividends in product development and team proficiency. Given the relative re-use and maturity of the ground segment dictates how much focus will be placed on this crucial element at the Project level and thus, impact the scope for the rehearsal program. On the other end, the decision to "re-invent the wheel" commits a program to enormous amount of effort and financial resources to deliver a ground segment ready for launch.
As an example, on the MAVEN mission, where the core TCC was used for I&T and flight operations, the re-use of this reliable and stable ground system with the inclusion of mature institutional elements at JPL (DSN and Navigation), led to exercises that could stress these Ground Data System (GDS) interfaces less due to previous mission readiness activities. On MAVEN, the rehearsal program concentrated more on flight team product readiness (i.e., do we have all products identified and ready for flight approval), exercises that stressed process and mission timeline execution. For a Scout mission under this scenario, the levels of exercises were proportional and well balanced to the needs of the flight team.
On the other end of this model, the NASA/NOAA Suomi-NPP mission used a different TCC during I&T and flight operations, with the TCC being a part of a newly introduced billion dollar GDS. This system took extensive resources to verify before the rehearsal program could kick off. Given the lack of maturity and the complexity of many of these products, services and interfaces, the rehearsal program had less stable product interfaces and relied on more extensive product review prior to the start. In addition, the complexity inherent in dual agency collaboration spawned more rehearsal program scrutiny and last minute changes to the exercises. As a result, rehearsal days were increased, which required additional resources for preparation activities. As an example, the instrument power-on and calibration activities were exercised several times in nominal and nonnominal conditions. Furthermore, an intensive collision avoidance exercise was performed with the NASA process for product exchange and decision making, after initially agreeing to use the NOAA process. Table I below summarizes extreme ends experienced in developing a rehearsal program as a function of ground segment development approach. It is shown for comparison and as a guideline for missions at the extreme ends of the ground development paradigm. For future missions, programs need to perform their own assessment and justify the right level of exercises for their mission readiness and rehearsal programs.
For these examples, thoughtful decisions were made with all stakeholders, resulting in successful rehearsal programs for each mission. For the most part, Earth Orbiting missions have a flexible launch date which allows operational products to be finalized prior to launch and used during mission readiness tests. These flight products are eventually used during the first thirty (30) to ninety (90) days on orbit as part of satellite checkout and instrument commissioning. Then routine observations are loaded to the spacecraft on a daily or weekly basis to perform science operations.
In terms of the rehearsal program, the Earth Orbiting missions perform all their exercises pre-launch, as the final mission readiness tests before shipment to the launch site. Like all missions, this last push to the launch site is a mad rush so the impact on rehearsals is the same: be ready to rehearse! But for Earth Orbiting missions, once on orbit, rehearsals are no longer a distraction to the team that has gained their experience from the program and can now focus completely on flight activities and product changes based on flight experience. A rehearsal program for Earth Orbiting missions is highly interactive due to real-time data availability and the instant feedback from ground commands. Given ground and space communication assets, Earth Orbiting missions have the "luxury" to monitor all critical activities, command and control activities from the ground and to be notified in near real-time of events onboard the spacecraft. Flight operations have tight control over daily activities and the operations philosophy is to perform the same set of functions daily. During a rehearsal, the flight team remains on console for a majority of the exercise and learns to fly with real-time displays.
On Deep Space missions, a bulk of the work is started prelaunch, with the products for the initial flight timeline targeted to be finalized, while the remaining lower-priority products are completed to the highest degree allowed, used during mission readiness spacecraft level verification tests, and then put "on the shelf" for finalization post-launch. As Deep Space missions can take several months or years to cruise to a distant body, the operational product development cycle extends into the post-launch cruise phase. Launch date is not an end-point for product development; the success of the I&T program will influence the level of product maturity at launch. On MAVEN, a successful I&T program resulted in very mature and complete products; on New Horizons, delays in autonomy testing prevented the use of the simulation environment for the final maturation of commissioning products prior to launch.
Deep Space missions perform rehearsals pre and postlaunch. Deep Space missions prolong the need for rehearsals as their products are not all final at launch and rehearsal time pre-launch is allocated to spacecraft level verification tests. This approach causes a Deep Space flight team to juggle between two critical activities: 1) day to day flight activities and 2) planned development, test and finalization of operational products. In addition, the team works to prepare for rehearsal exercises while flying their mission! A rehearsal program post-launch runs the added risk of: 1) flight anomalies reducing time available to exercise, 2) competing for simulation resources to test flight products while performing rehearsal dry runs and 3) competing for resources in completing risk reduction tests and products updates for planned or unplanned special events (i.e. "fly by", Mars Orbit Insertion, Targets of Opportunities, etc). An important consideration tor a Deep Space mission rehearsal program is the concept of Round Trip Light Time and how daily ground interaction is kept to a minimum. Due to long latencies between command uplinks and verification, and low data rates, rehearsals are seldom console type events; exceptions are the launch and planet insertion exercises, where a team is available to follow a timeline of activities happening on-board. Finally, rehearsals on Deep Space missions focus more on the process and cadence used by flight operations to generate develop and approve products for uplink. The planning and coordination in support of these activities is a key objective.

C. Considerations due to: flight team maturity (operational process maturity)
A mission is only as good as the personnel that run it and execute it day to day. Typically, Earth Orbiting missions are less complicated in the sense that they repeat the same set of on-orbit observations; Deep Space missions consist first of a lengthy cruise/checkout phase, followed by a flyby or planetary orbit insertion, and then the routine items performed by Earth Orbiting missions. Deep Space missions provide more operational complexity and require a dynamic team.
A difference in philosophy is worth noting here that can be used to measure the quality of a team in the context of ground system readiness, it's relation to operational product readiness and the personnel used to accomplish these activities.
The philosophy for Earth Orbiting missions is that the ground system adheres to a strict ground system development and test program leading to formal delivery and acceptance by a government agency team. Once completed, formal operational product readiness efforts can commence that result in final products approved for flight (and rehearsals). The typical flow generates products that are born of various hands such as: 1) a ground development vendor team delivering the TCC and ground tools; 2) acceptance testing of TCC software by an independent test team or Flight Operations Team (FOT); 3) system experts providing narrative procedures for flight product development; 4) development and testing of operational products by the FOT; 5) testing of interfaces by another independent team; and 6) conversion of products (e.g. databases) for satellite level testing due to different TCC at the plant and control center! In this scenario, the FOT is never in control of their ability to develop products.
With regards to personnel, Earth Orbiting missions, whose planned development work load is substantially less risky after launch, rely on integrated teams with various flight experiences, augmented between six to twelve months prior to launch. These individuals stay together for launch and commissioning, after which most of them disband as they journey in search of the next mission. Only a minimal crew stays behind for satellite monitoring and operations.
In Deep Space missions most middle men are cut out since the spacecraft vendor and FOT are usually one and the same. But the curious item is that due to their reliance on ground system heritage and reuse, their initial launch products are readied and approved for flight BEFORE the final ground system TCC and tools have been accepted! How does this system work? With a team that: 1) develops operational products at the spacecraft vendor level, which 2) is using high heritage ground systems and tools, 3) that is knowledgeable in the ground system and final products (no narrative), 4) whose process is to start from the prior mission and report on non-conforming items for the current mission to improve the ground tools, while 4) avoiding the complexity and overhead of different external teams involved. Avoiding different TCC's at the plant and control center also helps! In general, the FOT and I&T team are highly integrated in Deep Space missions.
As for personnel, Deep Space missions have an inherently riskier approach stemming from their need to launch in a specific window as the next launch opportunity could take years or decades. These teams are highly functional, motivated and experienced. What makes a successful Deep Space mission is a mature and experienced FOT, augmented during the cruise phase.
Regardless of the mission or team, or when exercises are conducted (pre-or post-launch), a successful rehearsal program will ensure the teams involved can demonstrate: 1) proper configuration management of all products used and environments (flight and simulation); 2) that a process for product development is followed and includes testing and analysis of results; and 3) a formal product Change Request process is in place to track new product development updates during the mission.

LESSONS LEARNED AND GENERAL ADVICE
Some lessons that are applicable to any rehearsal program are documented here as a guideline for executing a successful rehearsal program while avoiding common pitfalls.
As Rehearsal Director, assess the mission and understand what is unique about it; all missions are unique onto themselves, but first time activities and team dynamics are frequent weak areas. Deep Space missions usually perform complex navigation activities to enter planetary orbit or for gravity assist maneuvers. Earth Orbiting missions must perform frequent collision avoidance maneuvers but have fairly routine operations once commissioned. Being able to scope the effort along these lines allows sufficient resources to be applied in preparing the rehearsal program. Determine the flight team size (small is on the order of 50 members, medium is 75 and large is anything above 80) which implies the amount and level of communication necessary to prepare for an exercise. Again, by "flight team," we mean spacecraft and payload engineering teams, government managers, ground controllers, flight operations and mission planning and scheduling personnel. Review with the Project the number of rehearsals to be supported by the stakeholders and determine early if this number is adequate; a contract modification may be necessary. Review the system end to end and ascertain the maturity of the interfaces; if these interfaces are new or unfamiliar to the stakeholders involved on either end, more mission readiness tests may be needed which could impact the rehearsal schedule or affect the efficiency of the team during exercises. Learn if the key participants have a background in supporting NASA or if they have been exposed to commercial ventures. The rehearsal content for NASA missions is geared toward maximum risk aversion which may be an extreme for other cultures entering NASA.
To ensure a smooth transition into flight operations, always start at the top! Establish the decision making organization with roles and responsibilities for the flight team/stakeholders early on. Large size teams have many layers of management and decision makers that need to be re-trained for or kept out of the flight operations environment. This circumstance can be found in either Deep Space or Earth Orbiting missions. Engage Project Management and the Mission Manager on down to understand their role and the role/functions to be carried out by the team. Formalize this four months before the first rehearsal so that it can be communicated, discussed and agreed to by all flight team members. There are always surprises and territorial aspects that arise when folks from different organizations come together for the first time in an exercise. Work this aspect early and, in addition, have a room layouts created to document where team members will reside during each mission phase and during specific operations. Have this agreed to as well to avoid last minute reconfigurations that will not be supported by the hardware or software! To determine how well this holds up, it is always a good idea to do a Launch exercise as a first rehearsal in order to flush out leadership and room layout details.
Ensure the rehearsal program content is based on as many critical activities contained in the mission timeline that can be exercised with the flight team. If a list of exercises with content is available, review it for completeness and as Rehearsal Director assess whether more exercises are necessary. Deep Space missions are more prone to working with a limited amount of test time due to launch window constraints; Earth Orbiting missions will also be forced to launch though they are given more flexibility. If it is found that more exercises are needed, report this finding with supporting rationale to the Project immediately. Do this as early as possible to get commitments in place and add to the integrated master schedule. If the total number of exercises cannot be extended (e.g. typical of missions on a tight launch window), make the most of the given rehearsal opportunities. Propose making a 2 day exercise 3 days or extending the hours from eight to twelve or around the clock. Always exercise as much of the mission timeline as possible, centered around critical events and processes (such as, shift handover).
Worth noting here is an evaluation of the simulation environment by the Rehearsal Director. Experience shows that across all mission types, there is no simulation environment that can mimic all flight conditions and behave as the actual flight hardware. All hardware, software and hybrid simulators have limitations. As a Rehearsal Director, take the simulation environment "as is," but request a limitation list for all non-flight like profiles. The simulation engineer should be tracking these as part of future fixes or has already given up on their fidelity. Whichever the case, review the environment during a launch and ascent test case; good practice is to have some flight system leads support a dry run on the operational ground system and solicit their feedback on the simulated environment. It may not improve the conditions but it will serve to set the level of expectations for the simulation. Whenever possible do recommend critical simulation upgrades (e.g. control of the data on/off) but stay within reason of time and money. For upgrades already planned, monitor these deliveries and ensure they do not impact time scheduled for a formal rehearsal dry run. Finally, if data routing outside the control center is part of the rehearsal program, be sensitive to how upstream delays to the data will impact planned objectives under nominal and contingency scenarios. As a Rehearsal Director controlling the data stream is paramount to controlling the pace and direction of the simulation. Having a room full of flight team members waiting to make a decision without data may not be the most optimal use of everyone's time! So consider that network and communication link interfaces are best exercised in other mission readiness tests. If the scheduling process is to be exercised as a background objective, on a non-interference basis to the main simulation, then this is worthwhile.
Finally, work the rehearsal program diligently and efficiently. Once again, Deep Space missions are prone to move exercises earlier while this is rarely the case for Earth Orbiting missions. Develop a minimum dry run time and work it into the simulation facilities schedule. Always be prepared to rehearse earlier than expected; at one or two weeks from the planned rehearsal make sure all is ready to proceed as planned. To ensure readiness to rehearse, stay focused on the stakeholder's ability to support the exercise. The last two weeks prior to a flight team exercise, team leads need to provide status on their readiness to support. Assess the maturity level of products and processes; discuss with the leads their level of confidence and seek feedback from the Project on their readiness to proceed. All of this will be communicated in a formal Kickoff but it's always best to uncover any surprises before this. If a rehearsal date cannot be met, it should never be due to the Rehearsal Team being unprepared.

FINAL OBSERVATIONS: EARTH ORBITING AND DEEP SPACE MISSIONS
Certainly one size does not fit all. A rehearsal program that worked well with a mature ground segment, an experienced team and well documented products will not be adequate given a new ground system, a newly formed flight team and products developed with limited flight experience. The best manner to ascertain what to focus on and what to include in a rehearsal program is to assess these factors and ensure they are properly and sufficiently tested in the Mission Readiness phase. Know your mission in order to effectively tailor the rehearsal plan.
As stated previously, a Rehearsal Director given six to twelve months prior to the first exercise can: 1) identify all stakeholders and ascertain their unique needs; 2) develop specific objectives to build an integrated and highly functional team; 3) focus and develop the roles and responsibilities for all team members with input from the NASA Project; and 4) assess product and process maturity in pre-cursor mission readiness tests. A Rehearsal Director can then document these test items into a cohesive plan for final review and after modifications, execute team based rehearsal program to ensure low risk flight operations.
For large scale programs, a Rehearsal Director will need help to develop extensive plans and to communicate and report often to management. Given large programs' very low (minimized) risk policy, it is further suggested to augment the rehearsal team by a few members. Under this scenario, at a minimum, a Deputy Rehearsal Director assigned to help develop careful planning and requirement validation objectives is necessary. A person that can undertake the planning for instrument simulations and coordinate anomaly testing is also very crucial for an effective rehearsal program with large teams.