Post assembly quality inspection using multimodal sensing in aircraft manufacturing

Contactless, non-destructive testing has always been an important pillar in crucial tasks performed in industrial applications, including post-assembly testing oriented toward aircraft manufacturing. This work will examine the topic from the point of view of the quality check for the lining of aircraft interiors, such as sidewalls and hatracks, with the aim of improving safety of operation and comfort during flights, using an automatic approach guided by the usage of computer vision system. In particular, it will present a multimodal approach using a 3d snapshot sensor and a color camera, for identifying defects and anomalies that belong to two distinct categories, such as geometrical and surface defects, where, in the aircraft manufacturing sectors, due to the low-volume production, when compared with the automotive industry, such quality check operation tasks are still done manually, with a low level of automation. The proposed approach is showing potential and has been demonstrated on a proof-of-concept system prototype funded through a European Commission Horizon 2020 research project under the Clean Sky 2 umbrella aimed at the growth of the aviation sector.


INTRODUCTION
Non-destructive testing is a crucial task for checking the quality of an assembled machinery, or any other artefact, in general. In the last 40 years, with the consolidation in development of imaging sensors and the access to sufficient processing power, there has been a constant interest toward automated visual inspection 1 . Computer vision plays a fundamental role in a wide series of application domains, especially when data coming from multiple sources are acquired and processed, due to its capability of emulating functions proper of human vision, also performing complex tasks such as, for example, high-level semantic interpretation of a scene 2 . Bidimensional and tridimensional data are usually exploited to learn and develop complex models using iterative processes, machine learning or deep learning algorithms 3 . The main focus of these algorithms is to extract and effectively use significant features from data, from 2D images 4 , directly from 3D point clouds 5,6 , or even using an hybrid approach 7 . However, automated visual inspection has not gained traction in the same way in every domain. While it has found a receptive market in the automotive sector, also thanks to the mass production (and therefore testing) needed in that case, to the best of our knowledge, other industries, such as aircraft manufacturing, still performs most inspection tasks manually. Nonetheless, in the aeronautics industry, there has been a need to speed-up the inspection process, increasing reliability and accuracy of the quality control at the same time. Yet, the application of new technologies such as laser devices for measuring gaps between parts 8 has been most of the time limited to hand-held devices like HandySCAN 3D 9 , where a human inspector, scans the area using manual instruments until a sufficiently accurate 3d representation can be obtained and interested parts and zones can be measured. Moreover, the inspection process and research interest has been mostly limited to gap & flush management of exterior surfaces, when considering the aircraft industry 10 . Interest for quality checks on assembled parts related to aircraft interiors is not widely represented at the moment. Additionally, while considering aircraft interiors, the identification of quality defects related to gap and flush management, or more in general to geometrical arrangements of panels, covers just half of the necessary case studies because both geometrical and surface defects must be considered in this specific application domain. Besides, the final objective of these checks is not strictly limited to guaranteeing safety of operations, but also addressed at increasing and supporting the passengers' comfort during the flight. Indeed, some of the quality issues investigated by surface defect tests are purely aesthetic.
Quality control in aircraft manufacturing is as important as in any other industry. In particular, quality control of aircraft interiors is usually done manually, after all panels have been assembled. Clean Sky 2, a joint undertaking of public and private partners sponsored by the European Commission for advancements in the European aeronautics industry, is actively exploring and funding research activities in different topics 11 , including the automated assembly, and successive post-assembly quality control of aircraft interiors, which is the focus of this work. As part of this task, a sensor suite composed of a snapshot 3d and a color camera have been experimented as a mean of supporting quality inspection officers during their work, while collecting measures related to geometrical and surface defects in assembled panels. Indeed, their job require a constant level of attention while performing repetitive tasks, with measurements, and consequent decisions, which might be subjected to attention fatigue after some working hours. Moreover, some space of the aircraft interiors, such as the cargo area, can be challenging to inspect, due to the low ceiling height, causing human surveyors to work in nonergonomic conditions. These issues can be addressed by a better tasks' subdivision between robots and inspection officers, with the former dedicated to the automation of measurements tasks and the latter, still in a central role, verifying measures and reported defects and validating them using their extensive know-how. In particular, a system composed of several hardware/software modules have been developed around an acquisition setup able to acquire 3d and color data, with the objectives of collecting measurements for both geometrical and surface defects characterization, using statistical and iterative processes for 3D data and deep learning for 2D data. The solution based around the sensor's suite is described in this paper, which is organized as follows. An overview of the geometrical and surface defects that are considered important in aircraft interiors, and covered by the multi-modal sensing acquisition and processing system described in this work, is given in the next section. Then, after a brief introduction of the defects to be searched and investigated, the software/hardware architecture, as well as the subdivision of tasks and responsibilities is highlighted. The accuracy that can be achieved is highlighted in the experimental results. Conclusions and future work wrap up the paper.

GEOMETRIC AND SURFACE PANEL DEFECTS
Civil aircraft interiors quality control involves the inspection of two distinct environments: cabin (or passengers) area and cargo area. The type, arrangement and number of different panels vary, as well as the relative importance of some checks. However, the type of defects that need to be dealt with are essentially the same.
Geometric defects are all the defects that consider the relative arrangements between different panels. The most important ones for aircraft interiors are: a) steps; b) gaps; c) parallelism; and d) mismatch of tolerances. All these defects are related to measurements of distances between adjacent panels, although the focus changes from case to case on a specific axis. For example, steps (or z steps) are indeed concerned with measuring a "step" distance between adjacent panels, so that one does not look like is higher or lower than the other.
On the other hand, defects that belong to an individual surface are considered surface defects, regardless that they are related to just its color or they affect the shape as well. The most common surface defects are reported in Figure 2. As a look to the figures might point out, while the natural solution for discovering geometric defects is by using sensors capable of capturing 3d information, not all surface defects are best addressed by using color cameras.
Steps occur when two adjacent panels that were meant to stay at the same height, show a step on the z axis that is outside a predefined tolerance interval Gaps are related to the distance between adjacent panels considering their closer sides Parallelism defects are present when two adjacent panels lack proper alignment and do not look parallel Mismatch of tolerances concern relative distances (in terms of gaps) between more than two panels. In the mismatch, while individual gaps between panels are ok, when they are considered together, they are not. In this case, while texture inhomogeneities and color deviations can be identified by the latter imaging solution, the precision and accuracy needed for detecting scratches, along with dents and bumps, favor other types of sensors, especially since the evaluation of depth using color images could prove a suboptimal solution.
Moreover, while finding texture inhomogeneities, replacing the panel if needed, is something that is able of enhancing and magnifying the perceived quality of the aircraft, and the comfort of the passengers, all other defects have implication both on the perceived quality, but most of all can impact the safety of operations of the aircraft as well. Additionally, passengers and cargo areas show slightly different requirements, with the necessity for finding color related anomalies irrelevant for the cargo zone, since passengers comfort requirements do not apply there.
Finally, since the system is designed to provide an advancement on current state of the art, it must provide precision and accuracy matching or exceeding current inspection standards. Regarding defects based on 3d measurements, this means that the maximum tolerance bracket still allowed, before starting reporting a defect, is usually about 1mm, varying from defect type to defect type. In order to achieve it, precision and accuracy during acquisition should be below the mm (at least 0.6mm for gap, steps, parallelism and mismatch of tolerances). Scratches requires even more accurate measurements, since the maximum allowed depth for the scratch is about 0.2mm. On the other hand, a strict approach for considering color deviations, based on differences in a color space, is difficult to achieve in the post-assembly phase, and classification, after considering the higher subjectivity of the topic, is achieved heuristically in this case.

HARDWARE/SOFTWARE ARCHITECTURE OF THE IMAGING SETUP
The variety of defects to be identified does not allow to design an acquisition setup based on a single sensing technology. In particular, the 3d requirements, when considering the usually smooth surfaces, both in appearance and shapes, rules out the practical usage of 3d reconstruction based on passive stereo vision. Indeed, preliminary tests comparing the stereo 3d camera IDS Ensenso N30 against a blue-LED structured light LMI Technologies Gocator 3210 showed the superiority of the latter solution, even though it required to acquire and evaluate smaller patches in order to be able to work in its optimum sensing range. The Gocator 3210 is a snapshot sensor that can provide 3d surface reconstructions and is designed to cover a field of view that goes from 71x98mm at the closest range and 100x154mm at the furthest measurable range, with a clearing distance of about 20cm. The accuracy of the Gocator 3210 is further enhanced by using two cameras, each one aided by the projected pattern. This enables to better handle occlusions (affecting the projected pattern or a clear line of sight from one of the two cameras) and enhance robustness (when the same part can be seen by both cameras with the projected patterns on it). Another advantage of the sensor is provided by its Application Programming Interface (API) enabling access to both the reconstructed 3d surface or intermediate results, which proved helpful, as will be pointed out while explaining the associated processing later.
Unfortunately, the Gocator 3210 is a snapshot sensor able to capture just the depth and intensity of the surface, with no means of providing color information too. Those requirements are satisfied by an additional color camera, mounted on one side of the 3d sensor. The camera is a Teledyne DALSA Genie Nano XL, with a resolution of 5120x5120 pixels, which reconstructs the color by using a Bayer pattern. In this case, the color fidelity that can be achieved by the Bayer pattern has Scratches show both a change in color appearance and changes to its shape Bumps and dents show a local deformation in the material shape Texture inhomogeneities and color deviations show a change in appearance of the panel been experimentally proven to be sufficient, and the developed system does not need to use 3CCD cameras to achieve its objective. The color camera is aided by two custom-made illuminators, on top and bottom of it. They were custom-made for providing diffuse illumination albeit working with different constraints, especially related to their size and weight, which must comply with admissible payload maximum weight of the robotic arm, not detailed here for space limits and paper scope. It is also worth pointing out that the acquisition of the two subsystems cannot be triggered at the same time, since the pattern projected by the 3d sensor would be visible on color images as well, further complicating data processing.
However, color camera could be triggered fewer times, since the imaged area of the panel or panels is larger than the one captured by the 3d sensor, even considering the slight overlapping between adjacent areas for better judging defects at image borders. The way geometric and color data are exploited for investigating the possibility of defects being present is different.
In particular, all the defects that could be detected with 3d data, even if usually considered as part of surface defects (such as scratches, bumps and dents) are handled by a module, while a second one is used for working with color related defects. Geometric processing, as briefly introduced previously, is performed not by using the final 3d point cloud data created by the sensor, but by exploiting an intermediate data structure, which enable to analyze and access the data like a bidimensional domain of 3D points. This means that for each snapshot, it is possible to access both neighbor points on the same line and neighbors between subsequent lines, simplifying the processing involved.
The pipeline requires that the depth map of the point cloud is fed to a 3d flood fill method. It exploits the data structure, complemented by additional, higher-level information provided by 3d cad model, specifying the number of surfaces to expect, for a segmentation of the image in multiple surfaces via a custom 3d flood fill processing. The method is devised around a-priori knowledge, involving the number of different surfaces expected and the fact that interior surfaces are known to be smooth. Their individual surface can therefore be modeled as a 3d smooth surface around a paraboloid model Each contiguous surface computed by the 3D flood fill algorithm is then robustly fit to the paraboloid model using the random sample consensus algorithm 12 (RANSAC). The segmented surfaces are then sent to two different modules, specialized for different tasks. The robust 3d edge detection is used for geometrical measurements between adjacent panels and is based on masking and Sobel filter exploiting the dense point cloud structure that can be handled like a bi-dimensional domain while identifying contour points. Candidates edge points are then fitted on horizontal and vertical lines, again based on the RANSAC algorithm. Once edges are identified, they can be used for computing statistics about gaps, steps, lack of parallelism or tolerances mismatch. Indeed, the identification of many of these cases is very similar. For example, gaps and steps just focus on a different coordinate.
The last module for defect detection based on 3d data deals with surface defects such as scratches, dents and bumps. Their identification can be seen as a byproduct of a robust surface fit, where such anomalies are points considered outliers w.r.t. the locally modeled paraboloid. A fine classification between these cases can then be performed by investigating the arrangement of these outliers. For example, scratches are shown as a concentration of outlier points near a narrow line, while dents and bumps show a more circular arrangement. Additional details regarding defect detection pipeline on 3d data can be found in 13 .

The identification of color anomalies (color and texture inhomogeneities) is performed with a different pipeline, based on learning what is considered a defect and what is not. A convolutional neural network is trained in this case. The input layer
is fed with RGB images of 256x256 pixels. They get processed at different levels of detail through four consecutive convolutional and pooling layers. Each stage reduces the images size but computes more and more high-level features. Two fully connected network layers, with dropout, are present in the final stages, before classifying each 256x256 patch as either containing a defective portion or be acceptable. Choosing a size of 256x256 pixels for the input was done empirically by trying to find a good trade-off between its "trainability" and the ability to discriminate defects, considering patches in the right context. In particular, "trainability" here refers to the possibility to have enough data to conduct experiments that are inherently data-driven, where, for several reasons, including production and logistics, it can be challenging to get access to many panels for covering a range of possibilities. This constraint tends to push patch size to its lowest limit, so that many sample patches can be derived from the same original data. Computational and memory constraints also work against larger patches. At the same time, spatial relationships between pixels matter: a pixel with a dark color can be part of a defect somewhere and be legit elsewhere. This constraint pushes the patch size to the largest possible. In this context, preliminary tests showed that a good tradeoff between these contrasting needs is provided by choosing 256x256 patches.

EXPERIMENTAL RESULTS
In this work geometric and surface defects are handled through two different methodologies. Generally, cases are handled through the 3d and geometric pipeline. Indeed, many surface defects involving a local deformation of a surface are identified more effectively and accurately using depth information coming from the 3d snapshot camera than using colors. The only exception is given by handling of color & texture inhomogeneities where color information can prove beneficial. It is worth noting that it might be theoretically possible to distinguish them into multiple classes (texture inhomogeneities and color deviation). However, a color deviation on part of a surface has also an impact on the perceived pattern, hence practically they are considered as a single class.
3d pipeline handles most of the different cases. Geometric defects, related to the correct alignment of panels, can be verified by the point cloud, or, as explained previously, even from intermediate 3d acquisition stages, such as the depth map or the disparity map, simplifying computation and handling. In particular, identifying discontinuities and hence edges belonging to points "clustered" on different surfaces is sufficient for computing distance statistics. From the statistics, by looking to different axes (for example, z axis for step and x,y for gap), and aggregating data into simpler indicators (like median or minimum and maximum value) it is possible to identify all geometric defects.
An example is provided in Figure 4, where it is possible to appreciate through graphs and boxplots, the kind of accuracy and performance available by using the snapshot sensor in combination with the proposed processing pipeline. Reported measurements were compared with traditional measurement methods, involving the use of a caliper, showing better reliability and accuracy. Indeed, while it is possible to take just a few measurements of a caliper, and, considering the time needed, it is usually done only once, the evaluation of each snapshot with the 3d sensor provides about 1200 points for the identification of edges. As shown in the boxplot, the accuracy is very high. For example, for the gap, the boxplot ranges from 12.024 mm to 12.270 mm with a median of 12.126 mm.
In the same way, by considering the local deformities in patches of points belonging to the same surface, it is possible to use a surface fit for identifying outliers, directly leading to the identification of scratches, bumps and dents. Moreover, for the proper splitting of these outliers in one of these categories, it is sufficient to evaluate the shape formed from the outlier points for initially splitting between scratches and bumps/dents, with scratches showing points arranged in a linear way. Finally, by evaluating the displacements of the outliers from its surface, it is further possible to distinguish between bumps and dents. An idea of the tiny scratches that can be found with this methodology is provided in Figure 5, considering that the area depicted there is about 15cm x 10cm (rotated for page layout needs).  Regarding color and texture inhomogeneities, the experiments were performed as follows. A training set of images was acquired. The images were further manually "labeled" for creating image masks, providing a coarse split of the images in parts that should be considered normal (good) or anomalous (bad). Each pixel of the training images was then labeled for creating a mask. A third category, known as "do not care" was also included and was devised for marking areas of the training images related to background objects, that did not need to be used during training. Starting from these set of labeled images, examples with the proper size were extracted using a sliding window.
An overlapping of 25% was used. Each sample tile was then (automatically) either classified as "good" or "bad" considering the proportion of white pixels in the tile, with the tiles with corresponding parts in the mask with more of 20% white-labeled pixels marked as "bad". Considering the status of the panels at partners facilities, these anomalous areas were usually related to temporary markers or "dirty" areas. Indeed, no permanent defects were added, since panel parts had to be returned. During training, since the number of good patches was considerably higher than the bad ones, it was taken the decision to have just a few good patches for each bad one. In this way, while many "bad" examples were included in the training, with a few left out for test's sake, just a random selection of the "good" ones were used. Forty training "epochs", using a leave one out cross validation strategy, were necessary for getting a training accuracy of about 97%.
Tests, of whom a selection of test patches is reported in Figure 6, have shown that the trained network performed very well with the temporary markers while the "dirty" spots proved more challenging to classify correctly, since the "degree of dirtiness", when considered up close at the pixel level, tends to be very subtle. It should be noted, however, that these "dirty" areas can be easily cleaned and probably they should not be considered anomalous and marked as defect in the first place. The behavior on a bigger panel image is reported in Figure 7.

CONCLUSION
An innovative multimodal sensing setup, and accompanying processing pipelines, have been presented in this work. In particular, a 3d snapshot sensor is used in combination with a color camera, in order for identifying geometric and surface defects in aircraft interiors. Two different approaches are needed, and while the 3d pipelines is able of handling many of the defect cases, a separate methodology is needed for specifically handling defects that can be detected using color exclusively. The proposed approach has been used as part of VISTA, a European research project with the objective of redefining the way post-assembly quality control is performed using an autonomous platform, enhancing accuracy and consistency in the identification of defects in the process. The obtained results show that is indeed possible to use novel ways for automating the factory of the future. As in any research project, there is room for further investigation. This applies particularly to the color and texture inhomogeneities detection, which has been tested on a limited number of panels. Since having access to other panels could be a challenging task, a possible solution might be to synthetize color defects "painting" on available panel images. Moreover, other processing options could be investigated as well, both for 3d and color methodologies.