Insights into the Introduction of Digital Interventions at the shop floor

It is well-known that the introduction of innovative digital tools in manufacturing due to Industry 4.0 has far-reaching effects on an organizational and on an individual level. The H2020 funded project FACTS4WORKERS - Worker-Centric Workplaces in Smart Factories - aims to develop user-centered assistance systems in order to demonstrate their impact and applicability at the shop floor. To do so it is important to understand how to develop such tools and how to assess if advantages can be derived from the created ICT system. This study introduces the technology of a workplace solution that is linked to a specific industrial challenge. Subsequently, a 2-stepped approach to evaluate the presented system is discussed. Heuristics, which are an output of project "Heuristics for Industry 4.0" are used to test if the developed solution covers critical aspects of socio-technical system design. Insights into the design, development and holistic evaluation of digital tools at the shop floor should be shown.


INTRODUCTION
There have been increasing research activities during the last decade in studying the introduction of digital tools and changes in work practices at shop floor level in production environments [e.g. 1, 2, 3]. Büttner et al. [4] have made an overview of the studies done in the area of Augmented Reality (AR) and Virtual Reality (VR) concepts, especially their utilization to support and facilitate industrial use-cases. They also propose an interactive and community-driven tool for the visualization of the design space, which impose additional requirements for the systems of the future [4]. Thus, the challenges by using AR/VR applications in smart manufacturing environments call for further research. and attention recognition, and how these devices provide critical notifications perceived by workers in an industrial environment to keep workers away from situations, which are harmful or economically suboptimal. Further, Funk et al. [6] have made a comparison between the instructions of head-mounted displays (HMD), tablets, baseline paper instructions and in-situ projected instructions with an abstract Lego Duplo assembly task. The results of the study [6] indicate that assembling parts is considerably faster, workers make less errors and have less cognitive load when using in-situ projected instructions compared to HMD instructions, and locating positions is considerable slower using HMDs. Thus, the authors [6] believe that the handsfree possibility of in-situ projection will have great potential for instruction systems at workplaces, because tablet instructions may interfere a two-hand assembly tasks and HMD instructions have problems being accepted by workers. In addition, Kritzler et al., [7] present another study on a digital tool, i.e. RemoteBob, for supporting on-site workers by remote experts during ad hoc maintenance tasks. RemoteBob was implemented as a proof of concept in a lab environment and the study provides hands-on experience to industry experts on how the proposed concept could function in practice [7].
This study is a part of the on-going FACTS4WORKERS (F4W) project 2 , which develops and demonstrates workercentered solutions that support the inclusion of increasing elements of knowledge work on the shop floor of smart factories. The F4W project's primary goal is to develop, pilot and evaluate a worker-centered solution which is designed for industrial shop floor workers by using new models for work optimization and utilization of production systems. The main objective of the project lies on increasing problem-solving and innovation skills, Job Satisfaction of workers who are intended to evaluate the pilots, and increasing productivity. Interventions through digital tools are expected to increase these project objectives.
This paper reports about a digital intervention at the shop floor (section 3) and introduces two approaches for the solution's evaluation (section 2). First, the FACTS4WORKERS Evaluation Framework has the goal to demonstrate and evaluate the impacts of performed interventions, which is introduced in section 2.1. The framework is developed as a combination of several tools and methods, taking existing ones from literature as a base, tailoring them and defining new approaches when considered for measuring these changes and for demonstrating that these changes result from the digital interventions.
Existing Job Satisfaction tools do not completely cover all the factors we consider important for measuring the project goals or they do not measure the worker's feelings about the Information System being used [8,9]. In the F4W project, we consider the introduction of new information system solutions and new work practices as aspects that reciprocally affect each other and thus should be seen as one entity. Existing job satisfaction literature does not consider these changes in worker practices and information system solutions as one intervention. In addition, 2 Project Homepage: http://facts4workers.eu system acceptance and success models explain user satisfaction and system use as dependent of system and information quality and as determinant of net benefits but do not show how to measure them. The assessment of system quality (HMI interaction in particular) does not include shop floor workers. The presented evaluation framework is a more detailed approach to evaluate the acceptance of a system. Second, heuristics, which originate from the ongoing project "Heuristics for Industry 4.0" (hi4), are presented in section 2.2 and utilized to deepen the understanding of the developed information system solution in regard to the most critical aspects of socio-technical system design and to identify possible flaws or shortcomings (section 4.2).

F4W Method and Strategy
The F4W evaluation framework was introduced in [10]. The framework takes existing Information System success models [11,12,13] as base and extends them aiming to measure the impacts of an information system intervention at production environment shop floors. The framework and the strategy (see Fig. 1) for using it were tested last year (2017) when first prototypes of the solutions were deployed [10].
F4W evaluation goals are addressed by the impact analysis, in order to assess which is the impact of interventions on individual (workers autonomy, relatedness, task variety, competence and protection) and organizational dimensions (efficiency and quality), and the quality validation, where the different artifacts are evaluated to determine if the (system, information and interaction) quality meets the users' expectations. An example of using the results is presented in [14]. This paper uses the results of the evaluation performed at an industrial partner and shows how they are used for determining the achievement of the industrial challenge which is exposed in [2].
From a more general point of view, that is not restricted to F4W scope, the final goal of our evaluations is to support the adoption of informed decisions about the next step of a project. After evaluating an intervention, considering the impact achievements, the room for improvement and the cost of changing the solution the next step can be determined. So the Framework supports the decision either to stop or continue the project and, in this case, the definition of features to be implemented in order to improve the software prototype.
The strategy we follow for performing longitudinal evaluations of project developments takes the F4W objectives definition as a starting point. The use cases 3 are defined in [15,16] based on the identification of the industrial partner context of use and on the description of the as-is and the should-be scenarios. The use case definitions include a high level requirements definition and the expected impact of their full implementation. From the high level requirements the more important software building blocks can be identified and prioritized, their main functionalities can be defined and first artifacts can be created and evaluated.
The process described in the previous paragraph is the starting and also the final point of evaluation iterations: because the solution is developed under the perpetual beta philosophy and under the agile project management, each release of the software artifacts must be evaluated. Although first and last evaluation iterations are considered special; all the iterations are performed following a three phase pattern: preparation; execution; and analysis of the result and extraction of conclusions [17].
The maturity of the artifacts to be deployed and the legal frameworks will have an influence on the tools to be used for performing the evaluations. Maturity will also determine if a before-deployment intervention and after-deployment evaluation is required. Finally, the specific evaluation is going to determine how the results are interpreted.
Before-deployment evaluation is required for all the artifacts without considering their maturity. The more relevant results are these obtained from the quality validation. These results determine if the quality of the artifacts is sufficient and, in consequence, will support the decision of continuing with the deployment or stopping the next steps. I.e. for mockups as they provide proof of concept negative results could mean project cancellation. The impact analysis which takes place before the intervention provides a base line to be used as a reference after the solution is deployed and used for a time. Additionally, when the impact analysis is performed during the initial development iterations it provides valuable feedback about the right understanding of the evaluation purpose and the used tools by the workers.
As artifacts mature the after-deployment evaluations increase their value and are required for created prototypes. These prototypes provide real functionalities and their usage is going to support the workers with their daily work. This has an effect on their working practices which makes impact measurement relevant. The impact is measured by comparing after-deployment results with before-deployment evaluation. While this comparison could also be made to a project baseline, we recommend performing it to the before-intervention as it will be more isolated from being influenced by external factors (even in the case their bias can be detected using a control group of workers).
Although the results obtained by quality validation are less relevant than the impact analysis ones for mature artifacts, they still provide high value for supporting the decision of next steps of a project. These results will suggest changes for improvements of the deployed artifacts, new use of the artifacts, new artifacts or changes in work practices. Changes in deployed artifacts, new functionalities and new artifacts can be quoted and also considering the current impact, it can be decided what to do next in the project.

Heuristics for exploring socio-technical systems
A different approach for analyzing systems is the usage of heuristics. While heuristic approaches do not claim to produce perfect "100% solutions", they offer a pragmatic way to identify the most urgent problems sufficiently with a reasonable amount of effort. The most prominent example for this kind of employing of heuristics is provided by Nielsen's usability inspection method for evaluating interactive systems [18], Industry 4.0 scenarios go beyond interactive systems. They feature interdependencies between actors of multiple roles and technology that is characterized by cyber-physical components, autonomy, real-time capabilities and decentralization. The combination of a networked technical infrastructure and complex interactions between people in various roles constitutes a typical socio-technical setting [19]. It is characterized by intertwining technical components with organizational measures for communication, collaboration and coordination. Socio-technical systems can only incompletely be described and documented [20] and are a subject of continuous evolution [21].
To evaluate socio-technical systems, we have developed a set of heuristics based on five different domains: socio-technical design procedures, job re-design, privacy, computer supported cooperative work, human-computer interaction, and process redesign [21]. We suggest that Industry 4.0 systems are an appropriate domain for such a heuristic-based analysis. The project "Heuristics for the Industry 4.0" has developed a preliminary version of a set of heuristics that are derived from the literature and used for practical tests with a specific focus on Industry 4.0 concepts. The set of heuristics includes 13 main items in the following four clusters: (1) Tasks Fig. 3 in section 4.2 shows the heuristic items in detail. Each of the 13 heuristics addresses a significant aspect of sociotechnical system design. Ignoring or violating one or more of these heuristics increases the probability of the occurrence of severe system flaws. While the heuristics as they are shown in Fig. 3 are sufficient to address the corresponding aspects, each heuristic has additional sub-heuristics 4 . The heuristics can be applied either to observations made in concrete industrial plants, to models of Industry 4.0 solutions, to interviews that are run with experts who know the solution, or to a combination of these possibilities.

SMART FACTORY WORKPLACE SOLUTION
In F4W four smart factory industrial challenges prevail in order to demonstrate and evaluate applications of assistive technologies that are developed by perpetual beta principle. The industrial challenge presented in this paper serves requirements for several contexts-of-use of the industrial partners.

Industrial Challenge self-learning manufacturing workplaces
This industrial challenge envisions creating a shop floor prototype solution applied directly to a particular manufacturing line with either a product, resource or process data integration system that will monitor a combination of process or machine parameters. This self-learning manufacturing workplace should provide a proactive, predictive decision support to shop floor workers. This should be established by extracting patterns of successful production processes and linking heterogeneous information sources from worker´s environment and beyond [22].
By implementing advanced IT solutions, Internet of Things (IoT) -technologies and knowledge management procedures serve many possibilities for making the production more successful. A concrete advantage is the creation of self-learning manufacturing workplaces. With the utilization of manufacturing operation data, companies are able to e.g. predictive maintenance and machine assisted decision making for calibrations that allow the reduction of process based or setup based disruptions in order to maintain a smooth workflow. Hidria, an automotive supplier, takes over the role of a forerunner in this industrial challenge where disparate data sources are linked to realize novel decision supporting tools to enable continuous optimization of the manufacturing process [22].

Case vignette Hidria
Hidria is a Slovenian supplier to the automotive industry to which the company delivers critical components. The production and assembly lines are characterized by a fast production rate and consist of many complex operations. Difficult machine setup and many complex fault conditions lead to lengthy solution findings, which are very dependent on the experience of the workers. The information is scattered and difficult to access and maintenance is only event-driven. The F4W project aims to improve knowledge management regarding problem solving and problem prevention. Workers will have fast access to relevant information and more effective collaboration with peers to have a shared approach to arising problems. This should enable them to carry out more maintenance work themselves and prevent machine stops. The production data will be used to analyse and predict upcoming fault conditions in order to prevent them.

Technological approach
The F4W solution provides a wide range of functionalities supporting workers in different processes on the shop floor, therefore different technologies, frameworks and programming languages are used within the project. The whole software architecture shown in Fig. 2 is built with the application build and deployment tool Docker, which allows splitting of the whole system in smaller building blocks. This approach allows the development of each of the building blocks separately and facilitates the reuse and the integration of externally developed building blocks.
At Hidria the mark-up language HTML5 and the framework Angular are used in combination for the frontend building blocks. The backend building blocks are created using various frameworks, depending on the requested functionalities. For communication and exchange of data between the different building blocks, REST APIs are used and an NGINX reverse proxy is implemented. Data of geometrical measurements and the alarms and warnings will be queried from the company's database using a specific adapter. The data will be stored in the F4W database, implemented with PostgreSQL, and will be accessible to all the backend building blocks. The company's document management is linked with the F4W solution by a URL.

The F4W solution at Hidria
Using a tablet directly at the workplace, workers at Hidria can access the F4W solution, which consists of the following core functionalities.

Maintenance Scheduling.
The maintenance leader defines the periodic tasks that must be carried out by the operator to support a preventive maintenance plan. The building block Job Scheduler manages the scheduled events that are stored on the F4W database and can be submitted to workers based on a predefined list. Operations and instructions are available on the tablet of the worker.

Defects and Solutions.
For each alarm, warning and maintenance action the worker can access a database of possible actions (solutions) to cope with the current issue. The Defects and Solutions building block creates a relation between a defect and an already tested solution. It is possible to access all the tested solutions for a specific defect, add new defects and solutions and create a report. The actions will be explained using peer-to-peer comments, videos, photos and audio tracks. These file uploads are handled by an own Multimedia Management building block. The general approach is to share workers' knowledge for an easier and faster problem solving. The user generated content will be rated by the other peers.
In addition, the following functionalities are implemented for the second prototype which was not evaluated yet.

Digital data visualization.
The data regarding the machine setup, operation manuals, description of operation, machine layout, etc. will be available on the tablet of the worker, thanks to the remote access to the repository of the documents.

Trend analysis.
The digital data collected by the machine (measurements, production rate, etc.) will be analyzed and graphically represented. The worker can look at the data when necessary and it will be notified when a "defective" pattern will be found by the system.

RESULTS AND DISCUSSION
The system was tested by the technologist of the line that is a sort of shift leader and the two shift workers. For testing the solution a convertible (add-on keyboard) was selected by Hidria.
The evaluation executed at Hidria considered the prototypes implementing the solution of a use case covering two scenarios "Automated fault prediction and guided checking procedures", and "Shared documents and integrated human-machine information" [16]. To allow a maximal flexibility for the workers, the software is deployed locally and made available by tablets. This way the workers can record the information at any place and time. The intervention was carried out in April and June 2017 and comprised two rounds of data collection.

Evaluation results based on the F4W evaluation framework
The evaluated artifact was the first release of the functional prototype and, in consequence quality validation results are going to be more relevant than impact analysis ones. This release covers the core functionalities of "Maintenance Scheduling" and "Defects and Solutions". In any case, the impact analysis assessment was performed in order to validate the approach and to find possible improvements. Table 1 summarizes the evaluation execution: when it was performed, the tools selected and the object of each evaluation process. Table 1. HID evaluations summary.
The evaluation procedure was set up as followed. Before the pilot test started the solution was briefly presented to workers. Afterwards they were supposed to use a PC and started to work autonomously on the tablet. The process has been really smooth and workers immediately understood the functionality of the tool. After 5 minutes testing without any questions from their side, they highlighted possible improvements, new functionalities, as well as requesting the replication of the solution for other production lines. As expected, because of the maturity of the evaluated artifact, more relevant results correspond to quality validation.
Next are the more relevant findings: the application needs some solutions inside the database to be used by the operators, they will be created by the technologist before releasing the application to operators; Operators suggested including also the timestamp to the solutions used; Readability of the solution is correct, it is easy to access and the used icon is appreciated; The feature to create a new solution has been accessed easily. The tablet is OK for creating a single solution on the spot but they asked us to use the application on a PC for a massive data input (many solutions to be included to populate the database); the keyboard of tablet has been appreciated by the operators; the assignment (just click on a button…) of tasks has been done by the operators without any issues.
They suggest that some events would be assigned automatically by the system to the maintenance leader; a table to select the initial assignment of each event to a different role will be released (2nd product release).

Analyzing the Hidria use case with the help of heuristics
The heuristics shown in Fig. 3 were used to structure a group interview session with designers of the discussed solution. Four persons took part at this session; two interviewees and two interviewers. The two interviewees were researchers responsible for the application of the ICT-system that is described in section 3 and had detailed insights into the software's test run, which is reported at the beginning of section 4. The two interviewers are researchers from the distinct project hi4 and thus, had little prior knowledge about the technical solution of the concrete case and no knowledge about how the system was put to use by workers during the test run.

Fig. 3: Heuristic items for analyzing socio-technical systems
The interviewers used the heuristics that were clustered into four groups (A-D, E-H, I-K, L-M). After the aspects of these groups were sufficiently discussed, the interview was directed towards the topics of the next group. The interview showed that the heuristics help: • to deepen the comprehension of the system and to find out about the features that are being offered to its users, • to understand why the system designers added certain features while others were left out, • to identify blind spots of the system design that demand further clarification or give hints for improvement.
In summary, the interview confirmed the background of the management's decision to roll out the proposed solution on a larger scale: the system seems well-designed as it covers most of the critical aspects of socio-technical system design in a proactive elaborated manner.
The following paragraphs describe some of the interview's insights. We add a 2-tuple to every finding where the first position refers to the corresponding heuristic and the second position indicates whether the system offers sufficient support (+), shows a deficit (-) or further clarification is needed (?). E.g. (E,-) means that there is a flaw in regard to the 'Support of autonomy and flexibility'.
Knowledge management is per se a central contribution to the proper exchange of information (J,+). It requires extra effort for documentation. This additional workload was minimized by making capturing as easy as possible (B,+) with the help of mobile devices that can record videos (D,+); 250 newly entered solutions indicate a successful design choice. Documenting solutions immediately on the shop floor was identified as the ideal task workflow (B,+) and is enabled by the system (D,+), but not enforced (E,+). After a roll out in the large it should be evaluated whether the workforce in general is motivated to contribute to documentation (A,?).
The system relies on user-generated content. Before starting the usage of the systems, 50 solutions for the most common problems were entered. This measure helped to avoid an initial deadlock situation in which workers that need support could not find any content in the system, but were asked to provide content themselves (A,+). While descriptions of solutions can be created and edited by the users (F,+), the set of problems on which the system can react is fixed (F,-).
A major challenge is to offer the appropriate solutions or warnings for the situation the worker has to deal with. It still has to be evaluated how appropriate these solutions / warnings are (C,?) and whether the workers perceive a relevant benefit, e.g. by reducing the stress of complex maintenance work (A,?). Features for letting the workers rate the quality and appropriateness of the proposed solutions allows them to be in control (E,+), makes the quality of these proposals comprehensible for others (L,+), potentially eliminates bad solutions (M,+) and potentially fosters continuous improvement (F,+).
No aggregated data is provided to allow the management to evaluate the workers performance; consequently, privacy is maintained (J,+). However, the workers can identify the authors of the documented solutions (L,+), e.g. to contact them if questions arise (I,+). It is unclear though, if the system offers a direct communication channel with the authors (I,?). The possibility that some workers may be too timid to record a video, which could be bypassed by allowing anonymous postings was not taken into consideration (G,?).
Connecting the knowledge management system with other technological components such as additional channels for humanhuman communication (I,-) or the automated provision of the resources (tools, replacement parts) that are needed to work on a problem (K,-) are open tasks (D,-).
The whole knowledge management system offers fluent transitions between working and learning on the job; editing or authorizing solutions is an opportunity for reflection (H,+). Due to restrictions in time, the topic of autonomy could not be discussed in detail. It has to be understood whether following the proposed solution is mandatory or at least socially solicited (E,?) or how the processes of editing an existing solution and of creating a new description are defined (B,?).
Besides the elaborated design, the high acceptance of the tested system was probably increased by a young workforce that has an affinity towards new technologies. Additionally, a successful information campaign of the management framed the goal of the system as "making work more exciting" instead of emphasizing "increasing efficiency". This framing avoided fear of losing jobs because of technological advancements.   (1), the investigation can try to check whether the details and features of these heuristic are addressed by the system. If not (2) it has to be checked whether this is intentionally the case or not. If the heuristic, such as "support for learning" in Fig. 4, is intentionally neglected (3), the reasons for this omission can be elicited.
If the heuristic was unintentionally ignored (4), it can be checked whether measures for improvement should take place (5) or which reasons stand against such an improvement (6).

CONCLUSIONS
In this paper we present two different approaches to assess and evaluate novel ICT solutions in a shop floor environment. Within the FACTS4WORKERS project we have performed an experimental study. Therefore an evaluation framework has been developed to measure on the one hand the impact of smart factory solutions on workers and organizations (change in practices and ICT solutions). On the other hand it has been developed to gather qualitative feedback from workers for continuous improvements of the workplace solutions. It is a tool in order to support decisions at all stages of the software development which follows a bottom up approach. In contrast to this framework we have also performed a theoretical study that aims to offer a pragmatic way to identify the most urgent problems sufficiently with a reasonable amount of effort. This was realized with the help of heuristics -a top down approach -which help to get a more detailed understanding of critical aspects of the developed socio-technical systems. Using the heuristics to structure an interview helped the process of creating a diverse understanding of the system (for persons that do not know the system) and pointed the creators of the system towards aspects they potentially overlooked when designing it.
While comparing both approaches, the first issue to be highlighted is that even considering different starting points, a relation between the concepts they focus on can easily be established. Also the impact dimension "relatedness" considered by the evaluation framework can be linked to the G and I heuristics proposed by hi4 (see Fig. 3). Both methodologies consider the dimension autonomy.
Moreover a parallelism between the way hi4 heuristics are clustered and the way the evaluation framework groups its tools can also be established. The first cluster can be linked to the frameworks set of tools for validating the quality of the system, while the other three clusters are linked to the individual impact dimensions of the impact analysis tools. However, the evaluation framework also considers the organizational impact dimensionsefficiency and quality -which can be considered similar to heuristics K and M, are not clustered together by hi4.
Out of the F4W project we applied these two methods to one specific context-of-use which is addressed by requirements regarding the industrial challenge "Self-learning manufacturing workplaces". Therefore several software building blocks have been deployed which interact with each other. For this industrial challenge also other industry partners are reusing software building blocks to meet their particular requirements.
Heuristics provide a good way for analyzing qualitative data that can be used for clarifying the definition of the context-of-use and requirements as well as, what has to be measured for each ICT solution. This method can also be used to create system descriptions or project reports in a structured way. The evaluation framework can be used for quantifying the fulfillment of the requirements, continuous improvement of the ICT solution and as a decision support system which is based on an impact analysis in order to decide what to do next in the project. This procedure can be extended by exploring the workplace solutions with the help of heuristics in order to get a holistic view of the human centered design process. This offers a new way of cooperation in future projects.