Experimental Dependability Evaluation of Memory Manager in the Real-time Operating System

The paper presents results of experimental dependability evaluation of the Phoenix-RTOS operating system. Experiments are conducted using a self-developed testing environment and a kernel fault injector. Dependability evaluation is the last stage of a system development process. Results will be used in the future research to propose enhanced error detection techniques embedded into the operating system kernel.


INTRODUCTION
Dependability evaluation of operating systems is the focus of many research jobs. Developed methodologies are based mainly on software-fault injection technique. In papers [2], [8] new methods of dependability evaluation for microkernelbased operating systems are proposed. In paper [5] experimental environment for operating system with monolithic kernel and results of experimental dependability evaluation for Linux kernel are presented. The paper [6] presents model of error propagation between device drivers and operating system services. New measures characterizing error propagation (exposure and diffusion) have been introduced. Results of experimental estimation of introduced measures have been presented. In parallel to performing the presented works focused on dependability evaluation of popular operating systems, new architectures of mission critical operating systems have been proposed. The main goal of these architectures is to maximize efficiency of error detection and to eliminate fail silence violation. In paper [3], [1] the timetriggered architecture and results of experimental dependability evaluation for the architecture implementation have been presented. The main drawback of presented architecture is the requirement of specific application decomposition. It significantly decreases number of possible system implementations and practically disables application porting.
Presented works, related to dependability evaluation of popular operating systems, are focused only on selected modules and system services provided for applications. Proposed operating system models do not include functional dependencies between subsystems and error propagation is evaluated using simple metrics. Presented results have no influence on development of new methods and algorithms for error detection and propagation barriers embedded into operating systems. New algorithms for kernel-level error detection and fault tolerance have been only proposed for MARS operating system.
Phoenix-RTOS operating system developed by the author is an open-source real-time operating system for widely used event-driven applications. System provides all services (like resource protection, threads and processes, IPC, IP networking and graphical interface) required by industrial applications. Development process of every subsystem has been extended with dependability evaluation experiments. Results of these experiments allow to propose new error detection and fault tolerance algorithms integrated with the operating system kernel. Detailed error propagation analysis will be used to propose propagation barriers. This paper presents results of experimental dependability evaluation of the memory management subsystem of Phoenix-RTOS real-time kernel Section 2 presents briefly Phoenix-RTOS operating system and memory management. Section 3 shows the used error model and introduced measures characterizing the system behavior after the fault injection. Section 4 describes experimental setup and developed fault injector embedded into the operating system kernel. Section 5 presents the analysis of results of conducted experiments. Section 6 presents conclusions.
nology in years 2005-2006 as the successor of the prototype operating system Phoenix [7]. System has been developed as a result of work which goal was to prepare the prototype for real industrial applications. Phoenix-RTOS is the time sharing real-time operating system intended for embedded systems. System executes threads located in separated address spaces associated with processes. The system has been implemented for IA32 based PC computers used in embedded applications. Ports for ARM7 and PowerPC architectures are available. Phoenix-RTOS has been developed for real industrial needs which is the health monitoring appliance of one of a Polish medical equipment vendor. This paper is not focused on Phoenix-RTOS architecture therefore only the memory subsystem will be briefly presented.
Virtual memory management subsystem consists of five parts: hardware abstraction layer V hal , page allocator V pg , kernel virtual space allocator V kmap , kernel heap allocator V kmal and process virtual space allocator Vmap. Memory manager functions are presented in the table 1.
Hardware abstraction layer V hal implements functions managing the virtual memory mappings and operating on MMU data structures.
Page allocator manages available physical memory using a page map structure. The map describes the accessible memory regions dividing it into segments consisting of page groups. Possible segment size values correspond to succeeding powers of 2 starting from 0. Each entry of the map points to a list of segments with a given size. The first entry points to the list of segments consisting of one page only. The N th entry points to the list of segments of 2 N −1 pages. After the system initialization the memory is divided into segments with maximum sizes. During the allocation process, allocating function (vm pageAlloc) looks for the first segment with a size greater or equal than requested one. When the segment size is greater than the requested size it is divided into two smaller segments. One of them is returned to the map. This process repeats until the obtained segment size value is closest the requested value. When segment is released (using function vm pageFree) the kernel merges it with a succeeded or proceeded segment on the list. Described memory allocator (named buddy allocator) is typical for the most operating systems and has been presented in [9].
The kernel virtual space allocator V kmap manages kernel virtual space V K . An allocated segment is not accessible for the kernel and it should be mapped into the kernel address space if the kernel intends to use it. Mapping function vm kmap looks for the first virtual space region at which a segment can be mapped. After mapping, the kernel marks this region as used. The last stage of mapping is the invocation of a hardware dependent pmap enter function which fills structures used by MMU. Opposite function vm kunmap removes segment from kernel virtual space, marks the region as free and inaccessible using pmap enter function. Kernel space allocation is not a common mechanism in operating systems because many of them assume that the virtual address space is much greater than the physical memory size and the kernel space can correspond to the physical memory.
The kernel heap allocator is used to manage the allocation of memory chunks which size is less than the size of the page (minimal size of the segment). The main data structure of this allocator is the size[] array. Array entries point to segments divided into smaller chunks with specified size. The relation between the entry index (i) and the chunk size is given by the function 2 4+i . When the table entry is empty (no chunks are available) the new segment is allocated, mapped into the kernel address spaces and divided into chunks. Heap allocator consists of allocating function vm kmalloc and releasing function vm kfree. Each of theses function depends on the segment allocation and mapping functions.

ERROR MODEL AND MEASURES
The error model assumed in this paper is an error having impact on correct execution of an instruction by processor. This model is similar to the model presented in [5]. Singlebit errors are injected to have impact on instructions of functions belonging to the memory manager. Each injected error can be characterized by following attributes: • trigger -An error is injected when instruction is fetched by processor.
• location -Errors are injected to code of selected functions at random locations. Errors are injected to any byte of an instruction.
• type -One single-bit (bit-flip) error per byte of an instruction is injected.
• duration -Two types of errors are considered -transient errors which are removed after the erroneous instruction execution and persistent errors which exist during the experiment execution time. Type of error is selected randomly, but the probability of selecting persistent error is 25%.
Outcomes from fault injection experiments are classified according to observable kernel states presented in table 2.
To characterize what impact errors injected into the function f have on the operating system the special measure, called operating system resistance R, was introduced. Resistance R for function f is the vector of percentage shares of every observable state during the injection campaign for this function f (1).

Sexc
Injected fault has been detected using CPU exception.
S hang Kernel hang ups and goes to a non-operational state. Restart is required.
The corrupted instruction has been executed and no visible impact on the system has been observed either by the workload application and operating system kernel.

S f sv
The corrupted instruction has been executed and requests generated by workload application are not properly serviced (i.e. two succeeded memory allocations returns the same heap chunk). This is the fail silence violation state.
S tm Injected error is not executed in the time assumed for the experiment.
Other metrics characterizing error detection timing were introduced.
• average error activation time T act -the time elapsed from the beginning of the experiment to the moment of execution of the corrupted instruction; • standard deviation of the average error activation time σT act ; • average error detection time T det -the time elapsed from the beginning of the experiment to the moment of error detection; • standard deviation of average error detection time σT det .
Error propagation is characterized for each function using a propagation ratio measure P rop(f ) (2). Experiments has been performed for workload generated by thread calling routines used for validation of implementation of a memory management subsystem. These routines activate all functions of the memory manager.

EXPERIMENTAL SETUP
Dependability evaluation of the operating system requires different methodology than the dependability evaluation of applications running under the control of an operating system. Application running in presence of faults injected into its address space has no impact on operating system stability and all faulty conditions can be properly handled using debugging subsystem routines. Methodologies and tools developed for the dependability evaluation of user applications have been presented in [4].
Dependability evaluation of an operating system requires the existence of separate computer which controls fault execution and gathers outcomes of each experiment. This computer is called an experiment controller. Errors are injected by the fault injector embedded into the operating system kernel which communicates with the controller. Each fault injection corresponds to one experiment. Many experiments associated with a selected function constitute a campaign. The experiment process is presented below: • Select the location of the fault and other fault attributes

• Set the instruction breakpoint address
Before setting the breakpoint instruction stream is disassembled to establish the location of the instruction containing the error location. Disassembled code is sent to the experiment controller. This step of the injection experiment is called the 'inject' phase.
• Wait for the instruction execution When a selected instruction is fetched by the processor injector modifies the selected memory location and enables the step mode of instruction execution. This step is called the 'preinject' phase. Disassembled instruction stream and CPU context are sent to the controller.

• Execute the corrupted instruction
After the execution of the corrupted instruction the experiment enters into the 'postinject' phase. In this phase the injector sends CPU context and disassembled instruction stream starting from address of the program counter and restores the original value of the memory byte if the transient error is simulated.

• Wait for the fault manifestation
An experiment enters into the last stage which is the fault manifestation.
• Restart the system using watchdog Outcomes from experiments are stored as files consisting data coming from each experiment state. The example file is presented below. This file describes the fault injected 12 ms after the system initialization. This injection modifies the second bit of byte belonging to the movl instruction at address 0xc0060c8e. The corrupted instruction has been executed 125 ms after the system initialization. Fault modifies offset of the operand moved to the EBP register. After the instruction execution the EBP register stored incorrect value but this error had no impact on workload and the operating system (fail silence state S f s ). After the timeout system was restarted and the next experiment was performed.

EXPERIMENTAL RESULTS
During the memory subsystem tests, 1503 experiments have been performed. Outcomes from experiments are divided into groups corresponding persistent and transient errors.
For each function of the memory subsystem introduced measures are calculated. Table 3   resistances R for persistent errors injected into the memory management functions. Table 4 presents resistances for transient errors.

presents operating system
The column named N presents the number of experiments conducted during the campaign. The number of experiments is relatively low in comparison to the number of function instructions but the goal of experiments was not to evaluate dependability of the memory manager in details. The goal of the experiments was to present the methodology and framework for a quick dependability estimation of implemented kernel functions using introduced measures. Such estimation allows for selecting functions requiring introduction of propagation barriers or enhancing error detection mechanisms. From development process perspective it makes no sense to evaluate in details dependability of every function because implementation is changing during the system life-cycle.
Results show that many of injected faults are not executed. Most faults are ignored in functions vm kmalloc and vm kunmap. This is a consequence of workload nature. Typical execution of vm kmalloc requires no page allocation and the branch instruction at the function begin omits execution of a large part of code. Similar scenario takes place in vm kunmap. When a kernel address space region is released, the kernel tries to concatenate it with other free regions on the list. If no candidate for concatenation is found, branch instruction moves execution to the end of the function. Least faults are omitted in function vm pageFree. The function code contains many branches but they omit only short parts of the code.
The other important fact should be noted. Results show that software error detection techniques like assertions verifying correctness of input arguments passed to a function are insufficient. No error has been detected using this mechanism. Paper [6] is focused only on injecting faults into input arguments of functions and his author concluded that wrapping is a good technique for prevention of fail silence  violations and increasing the number of detected errors. Authors assumed that measures introduced by them could be helpful in finding vulnerable operating system parts which potentially demand enhancing argument validation. Presented results show that such methodology is insufficient. The percentage of errors detected using assertion and exceptions is extremely low when faults are injected into the base system component which is crucial for overall system stability.
In presented results there are no outcomes associated with fail silence violation state S f sv . This is very difficult to verify outputs of memory allocation and mapping functions. If the mapping function fails and validating function tries to reference the mapped segment system hangs or processor raises the exception. This is the possible explanation of such situation. This explanation is convinced by the relatively high number of system hangups. Most hangups were observed when faults were injected into mapping function which operate directly on MMU.
Tables 5, 6, 7 present detection times and propagation ratio for each tested function. Injected errors have been detected practically after activation. In some cases only the latency between activation and detection was about 1 s. Highest error propagation was observed for vm kmalloc and vm kunmap functions. This is the consequence of function complexities and dependencies with other memory manager functions. Presented results for persistent and transient errors are similar. When fault is injected it is practically manifested after the first execution of corrupted instruction. This conclusion is convinced by the measured short times between error activation end error detection.

CONCLUSIONS
The paper presents methodology for dependability evaluation of Phoenix-RTOS real-time operating system developed by author for real application needs. Dependability evaluation is the last stage of implementation of every Phoenix-RTOS function. Outcomes from fault injection experiments allow to determine the kernel resistance (efficiency of error detection mechanisms) for errors introduced into the selected function and the function ability to propagate errors. In opposite to popular real-time operating system, which authors concentrate only on system stability, this is the novel approach which allows to propose new kernel-level error detection mechanisms and propagations barriers. Obtained results related to the memory manager (presented in section 5) allowed to select functions demanding the enhancement of error detection mechanism and introduction of propagation barriers. According to proposed methodology the goal of the introduction of error detection enhancements and error propagation barriers is to obtain (for every VM function) specific error resistance vectors with dominant p S exc and p Serr components.
Furthermore obtained results show that techniques based on wrapper, proposed in the paper [6]), are insufficient when errors are injected into the functions of crucial operating system parts.
Obtained results will be used in next stage of Phoenix-RTOS development to propose the enhanced error detection techniques embedded into the operating system kernel. The last stage of the development process will be the dependability evaluation of the health monitoring system based on Phoenix-RTOS.

ACKNOWLEDGMENTS
This work was supported by KBN grant 4T11C049 25.