Modifying a manufacturing task for Teamwork between humans and AI: initial data collection to guide requirements speciﬁcations

,


Introduction
In the last decade, there has been a large growth of AI systems applied to the manufacturing industry Li et al. (2017Li et al. ( , 2014; Buchmeister et al. (2019). The vast availability of data, ongoing advancements in learning algorithms, and a growing acceptance of machine learning applications are driving this growth, leading to a fast changing landscape of human-automation interaction in Industry 4.0. Pan (2016). in This context a human-centered strategy is essential to maximise a smooth transition in those changes and to maximise the potential benefits to encourage a good interaction between operators and AI on the shop floorWilson and Daugherty (2018).
This changes pushed some organisations to turn into team-based structures where their employees are supported by a different systems to cope with the growing complexity of the operational environment Katzenbach and Smith (2015). For a good team-based environment Salas et al. (2005) proposed a a framework of the key dimensions of what teamwork is. They define a team as a group of "two or more individuals with specified roles interacting adaptively, interdependently, and dynamically toward a common and valued goal"  Salas et al. (2005). In our context this concept can be extended by incorporating AI agents and enhanced automation as one of the possible team member. This concept explains that team leadership, mutual performance monitoring, backup behaviour, adaptability, and team orientation are the core components that should be included for a practical teamwork. In this context we will be focusing on the mutual performance monitoring. This component refers to the monitoring of the fellow team members to maintain an effective awareness to detect slips, mistakes, or lapses prior or shortly after occurrence McIntyre and Salas (1995). This paper presents an example of a use case where the basic key elements for this type of human-AI teaming have to be set in motion and the methodology used to achieve them Mocan et al. (2022). This work consists of the following steps: • Identification of relevant operator tasks that are currently not accounted for in the automatic data monitoring and quality controls of the process • Analysis with descriptive statistics of the available data from the manufacturing process • Identification and collection of data regarding recurring causes of process deviations and downtime in operations (which may require to account also for the missing human tasks) • Requirements specification for data to be collected and analysed to support a better automation-human collaboration on the case study

Use case description
The current interim lessons learnt came from the attempted application in a manufacturing company that produces high-precision machining of large-sized metal components. The selected process is milling of large parts, which are timeconsuming with execution times lasting up to 10 hours. This process intermixes automated and manual tasks. While the automated tasks can be precisely timed, the manual tasks have a huge variability in execution time (multiple hours). The automation of the milling machine is blind to the actions of the human and the human is blind to the execution time of the automated task, as the machine does not give feedback about upcoming human interactions. Therefore, automated as well as manual tasks happen without having a shared mental model or a communication feedback loop. The case study is trying to orchestrate the interplay between humans and automation and use AI to optimise their joint outcome.

Standard process description
In general the milling processes changes with each article. However, we can try and describe some basic high level steps as follows: Part set up and clamping: The metal parts are placed on the machining tables by the operator using a ceiling-mounted crane system with the capacity to carry pieces of several tons of weight. The parts are secured with a system consisting of chains, slings, closures, and moorings. Moving the parts and securing them to the table are the activities with the most relevant occupational risk profile within the task. Operators must ensure the good working order of the systems, (i.e.checking that slings, moorings, and chains are in good conditions, etc.) Depending on the shape of the metal part, a specific support stand is chosen to clamp the parts for machining. The clamping is a manual activity not mapped that can take several hours and can occur on one of the two turning tables while the machining is ongoing on the other (see Figure 1). The timing of this task is not currently recorded anywhere and represents a source of variability in the overall machining of the product (e.g. the overall estimated machining time associated with the Article 1 may be around 4.1 hours but the manual set up of the turning table could require from 1.5 to 3 hours). Support from an AI algorithm to suggest time to perform this manual task while the operator is not required to attend the automated task on the milling machine can constitute a significant benefit in terms of reduction of down time and variability of execution times. Computer Numerical Control (CNC) program execution: When the part is correctly clamped, the operator has to choose the correct program on the CNC interface. Previous to the execution of the CNC program, the operator must ensure that the correct tool heads are installed in a storage area of the milling machine their automatic use. The conditions of the tools are the following: • The milling machine storage area can stock up to 80 different tools. • Manual and automatic tools are labelled with a different series of numbers. • The probe head is always placed manually to avoid damage due to its sensitivity. • The tasks for the milling machine are defined in the program, but not the details of the manual tasks required to assist in the automatic process (e.g., changing manual heads).
Part machining: When the correct CNC program has been started, the milling or machining of the part starts. However, this automatic process needs sporadic operator intervention. The description of the general machining steps are as follows: • Probing: -The operator has to mount a touch probe. -The probe is self-calibrated by the machine using a reference. If there is any problem with the touch probe, the machine will raise an alarm. -The probe then check if piece is well located in the table by touching key reference points on the piece. If the piece is not properly placed, operator is notified and has to correct its position. Furthermore this process will then detect if the dimensions of the piece are out of tolerance (e.g., because of a defective casting), the machine will stop and raise an alarm. This would create a significant downtime since it require de-clamping and removing the defective part. This is why operators takes preventive measures of the piece when clamping it to minimise this eventuality.
• Milling: -After the probing, the operator re-activates the automatic activity blocks (it resumes the program execution). -The machine starts milling automatically the extra material from the piece, it follows the program also changing the tool heads as required. In some cases for tools that can not be changed automatically the operator may need to intervene Fine milling: Once the piece has been milled, the next step is to refine it in order to adjust the piece to the adequate tolerance values. In this step the operator will have to assist the process by manually mounting a special boring head. Then, the part is milled, measured again, and repeat this sequence until the tolerance value has been reached.
Part dismount: Once the CNC program has finished, the machined part should be ready for dismounting. Ideally, when one part is finished the milling machine should be able to move to the other turning table to start the milling of a new product. Meanwhile the operator can disassemble and unload the piece using manual tools and a overhead crane. This interplay can create down time as there is only one operator attending the milling machine and ideally he/she needs to allocate time prepare the other turning table while the automatic task not requiring his/her attention are ongoing. This is where an AI agent to support this scheduling may be of use.

Operational task description
The next step in the methodology required the mapping of the actual operational tasks carried out by the operator. This information can then be used to build a better map of the operational steps including also the operator tasks so as to design an interface that allows a communication between the human and the machine to facilitate the interplay between their joint simultaneous activities and the one they can each carry out independently. This will optimise execution time and decrease the down times in which the machine is waiting for the operator's input. A key ingredient here is knowing the details of what tasks operators need to perform OHara et al. (1994).
The methodology used to map out the activities carried out by the operators on the shop floor consist on the following: Talk Aloud Protocol (TAP). This is a data collection approach in which participants are requested to speak aloud while performing a certain task, describing what they are thinking as they complete the exercise. The subject is instructed to speak aloud whatever thoughts come to mind, offering a simultaneous account of thoughts while avoiding interpretation or explanation of what is being done. The TAP uses verbal reporting and raises thoughts into consciousness to collect information about an individual's cognitive processes Ericsson and Simon (1984). Think aloud verbal protocols provide detailed information about reasoning during a problem-solving or decisionmaking job. This technique has been applied in a broad range of situation, e.g. training purposes Vie and Arntzen (2017).
Eye-tracker. In short, an eye-tracker device consists on a video recording instrument focused in the eye movements of the wearer (observer). The observer's gaze pattern gives useful informa-tion based on where and what the observer is looking at when studying eye movements Duchowski and Duchowski (2017). The information collected with such a device can range from attention, fatigue level, perception, consciousness, and cognitive processes Yarbus (2013). Most studies have employed objective approaches to study the relationship between oculomotor behaviour and cognitive processes throughout diverse visual tasks. The popularity of the eye-tracker studies has grown with the improvement of their technology. Mobile eye-trackers provide freedom of head movement as well as commuting in the work area and other properties of natural vision, which results in a superior approach for researching visual attention and perception in a real-world setting Kiefer et al. (2012). Even some models offer the possibility of recording audio and video at the same time. This feature becomes handy when combining it with the TAP.
Semi structured interview and subjective workload assessment. As apt of the data collection semi structured interview were used to gather information about the current working practices and desires of the operators in relation to what they would like to change. Concurrently the NASA task load index (NASA TLX) was also deployed as a tool for measuring the subjective mental workload (MWL) experienced by the operators involved Hart and Staveland (1988). The tool can be useful to offer a comparable assessment of the MWL of a participant while they are performing the task as it is now, and as it will be in the future after being modified by the intervention. The operators therefore allowed us to observe them in action using a talk-aloud protocol with the support of a wearable eye-tracker device incorporating an audio recording tool, and were later also available to address some of our questions and complete a NASA TLX questionnaire that we currently do not report as it will be used only to compare the "as is situation" with the possible "to be" scenario. The protocol was revised by the project ethics committee.

Manufacturing process data analysis (descriptive statistics)
This quantitative data analysis of the manufacturing process is aimed to complement the information collected from the operator tasks to point possible deviations in the process times. Data sources and objectives: As a part of the manufacturing system, information about the positions, temperatures, and other multiple sensors from the machine is stored with a sampling frequency of approximately 1 second and combined with inputs from the operator, such as working order and article Id, and are subsequently displayed in a graphic interface. The graphic interface is accessible from the company intranet. However when the machine is shown to be stopped in the Cycle Time and Execution State, the reasons for the stop are not recorded or specified. This also represents an opportunity for improvement as discussed later in this document.
The manufacturer provided two data sets, the fist one including the operator and reduced machine data between 2022-01-23 and 2022-01-29, and the second one including the data between 2022-02-18 and 2022-03-11. They can be combined, since there is no significant difference in the manufacturing process or in the features presented.
The analysis performed was aimed at describing the data regarding operator inputs joined with some of the machine data. However, since a manufacturing process for a product may require up to 8 hours, the data set provided did not contain a significant amount of complete cycles.
The analysis only focuses on two types of products, which are the one most frequently produced by the manufacturer at the moment, and consists on descriptive statistics with graphical analysis to set the base for a future resolution of the scheduling problem. The relevant features used in the data analysis were: • Date of the event • ExecutionState: 0 (Ready), 1 (Paused), 2 (Stopped) and 3 (Working). • ArticleID: (input of the operator) The type of article that is being manufactured or, if the machine is not working, the task that is being made (such as maintenance, meetings, etc.) • WorkingOrder: (input of the operator) The identifier (number) of the product that is being manufactured (0, if no piece is being worked at the time). • ProgramName: The CNC programs used by the machine. Art1 Prog1 and Art1 Prog2 for article 1, and Art2 Prog1 for article 2. • ToolNumber: The number of the tool head that is being used by the machine.
Data processing method: The Manufacturing data was processed using Python programming language and covered the following steps: (1) Concatenation of the different files in format .csv to reconstruct the combined dataset.

Operational task sequence: the missing ingredient
A summary of a task sequence (the finishing process for one of the pieces) is displayed in figure  2. Each of the tasks enumerated should have additional detail to describe the operational sequence of the tasks that the operator executes and, therefore, know the implication of the operators in each stage of the manufacturing process. The task sequence was changed to provide a deeper level of detail for the operators task descriptions. This, alongside the eye tracking analysis (3.2) and an analysis of the manufacturing process data (3.2) allows a better understanding of the operators roles and contributions in every stage of the process and help to identify the stages where a better interplay between automation and the human element can be made, such as asking for extra operator input regarding why the machining may have stopped and/or providing the operator with a view of what phase the the automatic process is at and when is his/her intervention needed next.

Eye-tracking analysis
As described in Section 2.1, an eye-tracker system can provide useful information on what the observer is looking at when studying eye movements. This information can be analysed to generate an operational description of the tasks.
In Section 3.1, the first task listed in the sequence is Coupling the boring head, and it is a manual process. The event starts when the operator uses the crane remote control to load the boring head (since it is quite heavy) and ends when the employee position the head near the milling machine where it needs to be attached. Once the operator attaches the boring head to the milling machine the finishing task can start.
The data in Table 1 provides information about the behaviour of the operator while the employee spend most of the time looking at the milling machine (43719 ms/12 visits), followed by the boring head (6872 ms/8 visits). The employer switches his attention mainly between the milling machine and the boring head. Occasionally he also focus on the crane remote control and the milling machine control panel.
Milling initiation. Once the operator has manually adjusted the boring head settings, he enters the milling machine cabin (start of the activity), inputs the proper values and starts the milling process (end of the activity).
According to Table 2, the employee started the activity using the control panel and then the main control screen for introducing the proper values for refining the milling of the piece. We can see how he spent most of the time switching his attention from the main control screen (10038 ms/4 visits) to the main control panel (6492 ms/10 visits), and to the door window of the milling cabin (13444 ms/4 visits), in order to monitor the execution of the milling process.

Data process results for the manufacturing data
For this section, some examples of the graphs that can be obtained from the data are included. Plots of one of the articles, and one of the programs of the first article are the ones displayed, though the study could be extended to all the programs of the two different articles. Figure 5 visualise the time series of the first dataset (from 2022-01-23 00:00 to 2022-01-29 23:58), that places all the activities (as different events) and pieces on a timeline. It shows how the articles are alternated and how the schedule of the different pieces occurs. It also reveals how important it is to remove the "0" working orders: for example, between the 2022-01-25 and the 2022-01-26, there is some downtime. it is recognisable because is a time that does not correspond to any article or activity, but it still has a program and a tool associated to it. This is normally considered an uncommon event registered by the workers. Erasing 0 working orders also allows to remove the downtime at the start and at the end of the week, when the factory is not open (e.g., on the last hours of the 2022-01-28, where the workers finish the last piece before the weekend and do not initiate the process of another). Figure 6 shows the average duration of the   processes for the different articles, if "0" working order is excluded. The variability shown in the others article ID shows that the duration of these articles is very variable. They are associated to either other pieces less commonly produced by the manufacturer, or other activities performed by the workers, e.g., preventive maintenance.  Figure 7 shows the average duration of the programs when ArticleId is Article 1. Art1 Prog1 and Art1 Prog2 are designed for this article. It reports some programs that are not designed for the corresponding articles (e.g., Art2 prog1 for Article 1). This could mean that there is a delay  ity of the program length for Article 1, which is considerably larger than the delay mentioned before and is worth studying. Although the working part of the programs should be the same (once the machine is active, the programs usually have a fixed duration), part of that variability could be explained by the differences between the cast pieces (they have tolerances and sometimes more material is to be removed) or the override that the workers occasionally enforce in the machine to reduce the milling time.  Figure 8 shows the average duration of the usage of each tool number where ArticleId is Article 1 and ProgramName is the second program for Article 1, which includes the finishing stage mentioned in the above sections. A high variability can be appreciated in Tool Change None, which is the value assigned when the machine has no tool. This item also shows the highest proportion of 2 STOPPED and 1 READY Execution States, in which the machine is not working. This could be useful information for the manufacturer, since the study of the causes of this variability and downtime can lead to a reduction of the overall production time of the pieces. This information is knowledge possessed by the operators that is currently not yet collected by the process and supports the idea of the need of a thorough task scheduling as shown in section 3.1 Additionally, the intervals of time with the highest variability are the one associated to instances where the tool heads need to be changed manually and/or for situations where the machine has no tool head mounted. These two instances also contribute the higher proportion of unproductive time if compared with the tool heads that are changed automatically. This is also something worth exploring to find a way to improve and make the manual changes more consistent.

Conclusions
This work presents the description of the general tasks of a manufacturing company in charge of machining metal parts. Such a description cover some of the basic element needed to lay down the basis for better requirements specification to modify the current interplay between the automation and the human operator assigned to the process. The analysis only focused in two of the articles produced, however the steps followed and the findings could be generalised to other products too.
It must be noted that the data has been labelled to describe when the operator needs to be directly supervising the milling machine and not only when he physically has a task to perform (e.g. change a tool head). The company currently collect specific times only for the task performed by automation, while the manual tasks and activities are unaccounted for. this creates a gap in the shared model of the interplay between the machine and the human. A thorough operational description of the manual tasks, as the one provided in this report, would benefit the time calculation of such tasks. The authors envision a potential for AI collaboration. AI algorithm in fact can supporting better communication between the Human agent and the Automation by providing info about the times when the operator does not need to directly supervised the milling machine and suggest a scheduling for other simultaneous activities that the operator can perform such as preparing the other turning table for the second piece and/or be able to perform some maintenance on the tools. To achieve this possibilities we need to focus on mapping and collect data about the human tasks currently unaccounted for. If they were to be reported and represented similarly to what is already done on the automation side, and if the operator were willing to support the missing information regarding the causes of stoppage and downtime, it will lead towards a better teamwork between the automatic agent and the human agent providing what Salas et al.Salas et al. (2005) refer to as necessary teamwork coordinating mechanisms: a two way communication and a shared mental picture of the situation (e.g. the operator will be able to see when is needed and when is not and be given suggestion for other tasks he/she may need to attend to, while the automation will be able to monitor what task the human agent is attending to, how long they take and be given information regarding causes of downtime when it occurs).