Augmented Reality-extended Humans: Towards a Taxonomy of Failures – Focus on Visual Technologies

Augmented reality, e.g. immersive visual technologies, augment the human's capabilities. If not properly designed, such augmentation may contribute to the decrease of the human's awareness (e.g., d ...


Introduction
Augmented reality-extended humans refers to humans, who can see, hear, perhaps touch, smell and taste more than the non-extended ones by receiving extra information through augmented reality (Krevelen and Poelman 2010). For example in transport system, additional information regarding surrounding environment can be displayed on the windshield of the car to extend driver capabilities in driving safely (Abdi, Abdallah, and Meddeb 2015).
Providing extra information through visual augmented reality can improve driver's performance, but meanwhile it can enforce additional cognitive-processing load (Schwarz and Fastenmeier 2017) or distract driver, if it is not properly designed. Failures related to using visual augmented reality technology or more specifically immersive visual technologies are not considered by current human failure taxonomies. In this paper, first, we review state-of-the-art human failure vocabularies and taxonomies with the lens of the well-established terminological framework on dependability (Avizienis et al. 2004). Then, we provide a novel organization of the fragmented taxonomic domain knowledge by means of a feature diagram that systematizes their inherent commonality and variability. Finally, we extend the feature diagram by considering failures describing the deviating behavior of augmented reality-extended humans, focusing on visual technologies. The final outcome serves as the foundation for failure logic-based analysis tools for (image-centric) socio-technical systems.
The rest of the paper is organized as follows.
In Section 2, we provide essential background information. In Section 3, we review human failure taxonomies, with the state-of-the-art dependability-focused lens. In Section 4, we propose our human failure taxonomy. In Section 5, we discuss about our achievements. Finally, in Section 6, we draw our conclusions and sketch future work.

Background
In this section, we provide the background information on which this work is based on.

Feature model and feature diagram
A feature is a prominent or distinctive characteristic of a family of systems that can be understood or seen by end-users (Kang et al. 1990). For example, transmission and horn in a family of bicycles. Feature modeling deals with the illustration of common and distinctive features of a family of products. Families of products are also known as product lines (Schobbens et al. 2007). Feature diagrams are a broadly used specification language for modelling features. A feature diagram consists of a multi-level tree, where nodes are features and edges are used to decompose features into more detailed features. There are different kind of features such as mandatory, optional and alternative (Kang et al. 1990). The legend in Fig.1 summarizes the subset of the concrete syntax of feature diagrams, used in this paper. The feature diagram, in Fig. 1 Proceedings of the 29th European Safety and Reliability Conference exemplifies the usage of feature diagrams for a family of bicycles, characterized by four features, where transmission feature is mandatory, horn is optional. One gear or multi gears, which specialize transmission, are given in alternative.

Basic Concepts on Dependable Systems
In this subsection, we recall essential dependability-related terms, introduced by Avizienis et al. (Avizienis et al. 2004).
System is "an entity that interacts with other entities, i.e. other systems, including hardware, software, humans, and the physical world with its natural phenomena". System function is "what the system is intended to do" and correct service "is delivered when the service implements the system function". Service failure or failure is "an event representing a transition (a deviation) from correct service to incorrect service." Error "is the part of the total state of the system that may lead to its subsequent service failure". Fault is "the adjudged or hypothesized cause of an error". A failure may manifest itself in different forms that are called failure modes. In literature (Pumfrey 1999), service's failure modes have been categorized based on: 1) provisioning (omission, commission); 2) timing (early, late); 3) value (course, subtle).

Visual augmented reality technology
Visual Augmented Reality (AR) technologies (Krevelen and Poelman 2010) superimpose computational and virtual content upon the real world view of the users. We summarize some of the effects of using augmented reality from various research papers: (1) Drivers may detect risks and respond more quickly (Wai-Tat, Gasper, and Kim 2013); detect hazards in low visibility (Schall et al. 2013).
(2) Drivers' perception to side lanes vehicles may be augmented (Wai-Tat, Gasper, and Kim 2013) and the drivers' speed in perceiving (Phan 2016) may be increased.
(3) Driver's situation awareness (Wai-Tat, Gasper, and Kim 2013) may be augmented. Note that situation awareness is "the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future" (Endsley 1995). For example, when a pedestrian is in front of the car, the driver first perceives the pedestrian, than estimates the time for crossing (comprehend) and then decides about the action (projection). Therefore, increased situation awareness shows improvement in perceiving and deciding functions. (4) In visual augmented reality technologies, GPS, lidar and infrared sensors provide more information from outside of the vehicle for driver and extend human sensing/ detecting/ perceiving in addition to providing surround sensing capability (Phan 2016). (5) AR causes stronger visual attention allocation during decision making phase (Eyraud, Zibetti, and Baccino 2015) and attention is directed to roadway hazards (Schall et al. 2013). (6) AR provides additional information for decision making and helps in learning and preparation of decision makers. Spatial problem-solving may be increased and comprehensive decision making is facilitated (Deshpande and Kim 2018

Revisited Human failure taxonomies
In this section, we review the most used human failure taxonomies with the dependabilityfocused lens. More specifically, in compliance with Section 2.1, we use the term "failure" for human deviations from expected behaviors and not the term error, as it was done before the birth of the dependability community. We also distinguish failures from failure modes, by prefixing failure modes with "FM". Moreover, we use quotations when we cite the definitions and italics when we deemed necessary to complement the definitions with explanations taken from the Oxford dictionary (Simpson and Weiner 1989). Categories such as mistakes already mean failures. Thus, we do not repeat the word "failures".

Norman Taxonomy
Human failures based on (Norman 1980) are: (1) Mistakes are failures in "formation of intention". Mistake meaning is an act or judgment that is misguided or wrong.
(a) Decision making mistakes "arise when the situation is misclassified, or when inappropriate decisions and response selections are made". (b) Description mistakes are failures "in the retrieval and use of memory information". Description means a spoken or written account of a person, object, or event. (c) System induced mistakes are failures induced by the system that human is working within that. Induce means Succeed in persuading or leading (someone) to do something.
(2) Slips are failures in "performance of the intention". Slips are pass or change to a lower, worse, or different condition, typically in a gradual or imperceptible way.

Reason Taxonomy
Reason (Reason 2016) divides human failures into three categories, which are: (1) Slips and lapses are "failures in either the execution or the storage stages of an action sequence." Lapses are brief or temporary failures of concentration, memory, or judgment. Slips and lapses are sub-divided into three categories including: recognition, memory and attention failures. (ii) FM-Non-detections are "failures to detect a signal or problem". Detect means discover or identify the presence or existence of. (iii) FM-Wrong detections are "wrongly detecting problems or defects that were not actually present".
Based on the definitions given in 2.2 (Pumfrey 1999) and based on the above recalled definitions, we can conclude that: misidentification is manifestation of a recognition failure as a value failure; non-detection is the manifestation as an omission failure; and, finally, wrong detection is the manifestation as a commission failure.
(b) Memory failures are failures in "information processing stages including input, storage and retrieval". (i) Input failures occur when "insufficient attention is given to the to-be-remembered material and it is lost from short-term memory." Input as a verb means Put (data) into a computer that here it is put into short-term memory. (ii) Storage failures occur when "the to-be remembered material decays or suffers interference in long-term memory". Forgetting intentions is a storage failure. Store means keep (something) for future use. (iii) Retrieval failures occur when "known material is not recalled at the required time". Retrieve means get or bring (something) back from somewhere. (c) Attention failures are failures that occur "when attention is captured by something unrelated to the task in hand". Attention means notice taken of someone or something.
(2) Mistakes are failures in "process of making plans". Mistakes can be rule-based or knowledge-based.
(a) Rule-based mistakes are failures in "applying a problem-solving rule that is part of our stock of expertise". (b) Knowledge-based mistakes are failures in "finding a solution 'on the hoof'" and occur in novel situations that there is not any rule to solve the problem. 'On the hoof' means without proper thought or preparation.
(3) Violations are "actions that involve some deliberate deviation from standard operating procedures". Violate means breaking or failing to comply with (a rule or formal agreement). Violations can be routine or exceptional.
(a) Routine violations are when the user often do the violation as a habit and it is tolerated by authority. Routine means a sequence of actions regularly followed. (b) Exceptional violations are when the user violates but it is not his/her typical behavior pattern.

Rasmussen Taxonomy
Rasmussen et al. (Rasmussen 1982)'s human failure taxonomy stems from the analysis of mental processes, which consist of three levels of cognitive control behaviors: x Skill-based refers to activities that are routine and humans do them automatically. x Rule-based refers to activities that need identification and recall from memory. x Knowledge-based refers to activities that are exploratory and unfamiliar. (Rasmussen 1982)'s taxonomy includes: (1) Detection failures: "Operator does not respond to a demand".
(2) Identification of system state failures: "Operator responds but misinterprets the system state." (3) Decision failures: Decision means a conclusion or resolution reached after consideration.
(a) Selection of goal failures: "Operator responds to properly identified system state, but aims at wrong goal (e.g. operation continuity instead of safety)." (b) Selection of system target state failures: "Operator selects an improper system target state to pursue proper goal (e.g. he decreases power to 80% instead of shutdown)." (c) Selection of task failures: "The operator selects a task, an activity which will not bring the plant to the intended target state." (4) Action failures: Action means the fact or process of doing something, typically to achieve an aim.

HFACS Taxonomy
The Human Factor Analysis and Classification System (HFACS) (Shappell and Wiegmann 2000) taxonomy is based on Reason (Reason 2000) taxonomy. HFACS includes: (1) Decision failures occur when the intended action is performed intentionally but the plan is not appropriate for the situation. These failures can be divided into three categories: (a) Procedural failures also known as rulebased mistakes occur "during highly structured tasks of the sorts, if X, then do Y." Usually management does not condone this behavior". (b) Non-violations are setting a goal inconsistent with proficiency, capability or readiness of the individual/team. (2) Attention failures are failures "to attend to relevant information that was present or accessible". (3) Sensory failures are failures in physical capabilities for sensing the needed information. Knowledge (Perception) failures are when "the operator didn't have the pre-existing baseline knowledge or skills required to adequately or correctly interpret the situation." (4) Perception failures are when "All relevant sources of information were attended to but an incorrect perception was formed due to ambiguous or illusory information, or due to processing biases that shape our perceptions and filter the available information." (5) Communication/Information failures are failures "in communication or information exchange between machine (display) and human, or human and human." (6) FM-Time Management are failures "to use appropriate and effective time management strategies." (7) Knowledge (Decision) failures are when "the operator didn't have the pre-existing baseline knowledge or skills required to form an appropriate or correct response to the situation. These are failures in knowing what to do rather than failures in implementing the response." (8) Ability to Respond Failures are when "the operator does not have the physical capability to make the response required to perform the task." (9) Action Selection Failures are failures "in the decision process due to shortcomings in action selection, rather than misunderstanding or misperception of the situation. These are failures to formulate the right plan to achieve the goal, rather than a failure to carry out the plan."

Our proposed taxonomy
In this section, we try to harmonize and organize the existing taxonomies as a product line and propose a feature diagram, called AREXTax, for modeling their commonalities and variabilities to present their evolution over time. For space reasons, our feature diagram is constituted of two sub-feature diagrams: one focusing on the human's functions and one on the failure modes potentially associated to these functions. In addition, we present an extension in order to deal with augmented reality.

Human functions taxonomy
Based on the six taxonomies, we retrieve and organize the human functions in Table 1. The rational for the fields of Table 1's columns is: 1) the function extracted from taxonomies; 2) subsection number of the related taxonomy and the failure that the function is extracted from; 3) failure modes (FM) of the function. For example, as it is explained in Subsection 3.2:1.a, recognition failure is a failure in the identification function, thus, in the first row of Table 1, identifying is extracted from the recognition failure. We also explained that misidentification is manifestation of a recognition failure as a value failure, so we add 3.2:1.a.i to the third column of first row. According to the definitions of the functions we define the hierarchy of them in the table. For example as mentioned in Subsection 3.2:1.a.ii detecting means identifying the presence, so we consider detecting as a subpart of identifying. Then, we extract human functions that are augmented via augmented reality. For example, when a driver uses visual augmented reality technology, he/she will detect more quickly through this technology and this AR-detection is an extended function of the extended-human.
According to subsection 2.4 we can extract human functions that are affected by using augmented reality. For example, in Subsection 2.4:2 it is stated that augmented reality augments driver perception to side lanes vehicles, so the affected human function in this case is perceiving. This function is shown in third row of Table 2. Then we present the human functions feature diagram in Fig. 2 that shows functions deciding/ making plan, acting and executing are three common functions in all six taxonomies. It means that in Table 1 we have failures from all six taxonomies for these three functions or the functions that are subparts of them.
In addition, we extracted some more functions based on visual augmented reality application. These features are shown by dotted lines in Fig. 2. For example, based on 2.3:4, we can consider GPS/lidar/infrared sensing as augmented functions, which transform humans into extended humans. By using AR information regarding surrounding the car and blind spots and displaying them on the view of driver (Rickesh and Naveen Vignesh 2011), he/she can sense, detect and perceive these additional information. Thus, these functions are extended as surround detecting/sensing/ perceiving.

Failure modes taxonomy
In this subsection, we show that (Pumfrey 1999) categorization is still valid and failure modes are still the same, shown in Fig 3. All FMs (failure modes in the third column of Table 1) are the features of the categories mentioned in Subsection 2.2. For example, according to definition mentioned in Subsection 3.2:1.a.i FMmisidentification is a wrong function and based on (Pumfrey 1999)

Discussion
According to (Hansman 2003), there are a number of requirements that a good taxonomy should meet. In what follows, we discuss to which extent our taxonomy meets those requirements. The proposed taxonomy is accepted, because it is structured and it is built on previous accepted taxonomies. It is comprehensible, because it is understandable by experts and those with interest in the field, since we split it based on human functions that are clearly defined. It is difficult to prove that the taxonomy is complete, but we can claim that it is complete to some extent because the covered taxonomies help to categorise the human failures based on human functions. It is deterministic, because we can determine human failures according to the related functions. However, sometimes, it is hard to discriminate if the failure is in detection or perception functions.
We cannot claim that it is mutually exclusive because each failure is not categorised into a single category. It is repeatable because we defined the procedure and by repeating the classification the result will be the same. In addition we used terms complying with previous and state-of-the-art works to remove/reduce the ambiguity. In some cases, in previous taxonomies, same terms were used with different meaning or same meaning with different terms. We reduced the ambiguity by using stateof-the-art-terms and showing how previously used terms were related with state-of-the-art terms. All the terms (including failures modes) are defined both according to the definitions mentioned in the related taxonomy and also according to Oxford dictionary. It is also unambiguous because the functions are clearly defined. Related to the usefulness of the suggested taxonomy, we do not have evidence yet. It should be evaluated by the community.

Conclusion
In this paper, we have reviewed the state-of-theart on human failure taxonomies and provided a taxonomy of taxonomies, given as a feature diagram, to visually show their evolution in time. Then, we extended the taxonomy for visual augmented reality-extended humans.
As future work, with growing domain expertise, we aim at defining cross-cutting constraints to relate human functions with failure modes. In addition, we plan to use this taxonomy as the foundation of a failure logic-based analysis tool for socio-technical systems and validate it in industrial settings.