A RANSAC-based method for detecting post-assembly defects in aircraft interiors

Automated assembly and testing have been key players in the automotive industry in the last few decades, due to the pivotal role of economy of scale possible in that industry. Until now, comparable efforts have not materialized in the civil aerospace industry. A visual inspection methodology for testing of aircraft interiors is presented here. It aims at increasing the level of automation during aircraft testing post-assembly, providing methodologies and technologies for the inspection of both the cabin and the cargo area in single aisle aircrafts. Its expected outcomes are related to a reduction of testing execution time and costs. Additionally, automated testing procedures will increase the level of decisions consistency and reliability, by considering the human factor and addressing the tasks that cause cognitive fatigue after prolonged periods of time. Early experiments validate the effectiveness of the proposed approach.


I. INTRODUCTION
Testing of aircraft interiors is a fundamental quality check procedure that must be carried out after most of assembly tasks have been completed. While in the automotive industry, various factors, such as mass production and economy of scale, have provided manufacturers with the necessary motivation to invest in automated procedures, both for assembly and testing [1], the highly specialized aircraft industry has lagged behind in terms of automation. The consequence is that post-assembly tests are still performed in almost every case by human inspection officers. Their job requires a high level of attention, kept during prolonged periods of time, performing repetitive operations: the very nature of their tasks is error-prone, in a sector where quality and reliability must be ensured for supporting safe operations when the aircraft is flying. Moreover, with cognitive fatigue that might be experienced over the working hours, comes the inability to perform quality control in a consistent way: if an inspector surveyed the same defect in different times, chances are he would likely treat it differently. Last but not least, some areas of the aircraft can be challenging to inspect, such as in the case of the cargo area, where a human surveyor needs to necessarily work in non-ergonomic conditions for a prolonged time, usually adopting a squat or kneeling posture, since the available space is narrow, with an uncomfortably low roof [2].
The visual inspection methodology described in this paper aims to post-assembly testing of aircraft interior installations in both cabin and cargo areas in order to ensure quality and reliability of located components. This goal is pursued through the development of a non-contact solution. The final objective of the system that is being designed starting from this methodology, is to inspect both geometrical and surface characteristics of common aircraft parts, such as sidewalls and hatracks (also known as overhead storage), in an automatic way.
The scope of this paper is to present some early results that concern the acquisition of geometrical information of the panels and their successive processing for detecting anomalies, with a focus on information that can be obtained starting from a 3d point cloud, which is not strictly limited to geometrical defects, but extends to surface ones as well.
The paper is organized as follows. First, related work is introduced. After having briefly introduced the kind of anomalies addressed by this work, the methodology employed in the processing is presented. Test setup and experimental results are then provided. Conclusions and future work wrap up the paper.

II. RELATED WORK
Non-destructive testing and automated inspection have always been crucial tasks in industrial applications often solved by computer vision systems [3]. Recently, costeffective hardware solutions that enable high-throughput 3D data acquisitions are easily available, as shown in a recent review [4]. However, as reported in the literature review, several challenges are still present, such as sensor performance that might prove inadequate in some situations from a measurements point of view. Dealing with real data acquired on field is therefore challenging because state-ofthe-art techniques might still not be accurate enough to address some common and important problems that need to be addressed, if automated post-assembly quality control must be pursued. a) Steps: occurs when two surfaces of contiguous panels should lie on the same theoretical surface and its measure instead falls outside a valid range. b) Gaps: occurs when the distance (in x) between adjacent panels falls outside a valid range. c) Parallelism: occurs when the difference in distance between contigous panels at their extreme goes beyond a predefined threshold. d) Mismatch of tolerances: occurs when there is a discrepancy of gap size. It occurs when two gaps face each other and, while both are within the given tolerances, the different gap size goes beyond a different tolerance appearance. This work demonstrates an acquisition setup and processing pipeline that are able to manage the kind of geometrical and surface defects that must be identified and notified while dealing with aircraft interiors, and in particular with some common panels such as sidewalls and hatracks.

III. A BRIEF INTRODUCTION TO PANEL DEFECTS
Panel defects, post-assembly, can be classified as regarding the geometry or the surface. Some of the most important geometrical defects are called: a) steps; b) gaps; c) parallelism; and d) mismatch of tolerances.
They are reported in Fig. 1. As shown there, geometrical defects are mostly concerned with the physical arrangement of adjacent panels. All geometrical cases concern alignment between panels. In most cases, just two adjacent panels are considered, such as for steps, gaps and parallelism. A more complex case happens when there is a mismatch of tolerance, where four panels are concerned and, while considering individual pairs is with the accepted margins of error, there is a gap size difference that leads to an unfavourable appearance, related to an unmet different threshold condition.
Surface defects are instead related to issues of a single panel at the surface level. They are concerned about surface uniformity, both considering its smoothness when touched and the presence of unwanted colour inhomogeneity, or even alterations due to dents, bumps and scratches. This paper and its proposed methodology consider only the surface defects that could be inferred without using colours.
Hence, only surface defects such as scratches are considered here, since the employed sensor can report 3d data, along with a light intensity value, but no actual colour information. It is a snapshot sensor (capturing a whole area at a time, instead of requiring to scan and merge together multiple lines like for profilometers). It is based on a blue-LED structured light technology. While the standard structured light solution uses a projector for creating pattern fringes and a camera to understand the shape of a model by measuring how the pattern is distorted by the model shape, Gocator 3210 uses a single structured light pattern emitter and a pair of cameras, arranged in such a way that help in solving areas in which the model shape could show occlusion issues.
The sensor is designed to cover a field of view that goes from 71x98mm at the closest range to 100x154mm at the furthest measurable range, with a clearing distance of about 20cm. An entire 3D surface of 100 x 154mm can be acquired in a single snapshot and is made of more than 2 million points.
Since a single panel is usually larger than the field of view covered by a single snapshot, the detection of all geometrical defects requires to aggregate analysis from multiple snapshot. For example, a gap measurement between two hatracks, starting from the gap reported by several partially overlapped snapshots of different parts of the adjacent panels, need to aggregate all these partial measurements, in order to provide a single measure to test against accepted deviations from an expected value.
On the contrary, surface defects related to geometry, such as scratches, can be already identified at the level of a single snapshot, taking into considerations that partially overlapped acquisitions already address the possible issue of surface defects near the border of a particular acquisition.
This work, and the methodology described here, is focused on the analysis and robust 3D measurements of the single snapshot and is summarized in Fig. 3. It is immediate to notice that it is made of the following three building blocks: • a 3D flood fill algorithm for identifying the connected components on the point cloud; • a robust fit of a 3D surface for highlighting regions of interest that are likely to be defects, such as scratches; • three-dimensional robust edge detection for geometrical measurements. This pipeline is valid both for purely geometrical defects and for surface defects that do not require the use of chromatic information, such as scratches.
However, from this point on, a different treatment of these two families is necessary. In particular, common modules are described first. Geometrical defects specific considerations are then reported. Last but not least, scratches evaluation is considered.

A. 3D FLOOD FILL
The objective of this algorithm is to cluster the input point cloud and determine how many contiguous surfaces have been captured by the sensor and where they are localized in a reasonable and optimized execution time. An iterative flood fill algorithm has been customized to perform this task. Each cluster is iteratively constructed starting from a single point called probe (P) and scanning its neighborhood N to find all the points that meet the following formula: where Δ(P,p) is the Euclidean distance between P and p and τ represents a predetermined threshold. All these points are then recursively considered as probe points, repeating the procedure until the criterion is not met anymore. The 3D flood fill stops when the point cloud has been entirely scanned.

Fig. 4. Example of a point cloud, clustered and represented as a bitmap. Each clustered surface is represented with a different color.
The process is facilitated by the fact that the sensor reports the acquired snapshot as a dense point cloud, with a fixed number of rows and columns, enabling to handle it akin a matrix. In this matrix, missing samples, related to depth values outside the operating range of the sensor, are represented with a special value that can be easily handled by the 3d flood fill algorithm.
The information on each cluster is encoded with a different code number directly on the bi-dimensional domain introduced beforehand as shown in Fig. 4 in its matrix representation and in Fig. 5 in its 3d form.

B. SURFACES ROBUST FIT
Each contiguous surface computed by the 3D flood fill algorithm is then robustly fit to the following paraboloid model = , = + + + + + using the random sample consensus algorithm (RANSAC). The analysis of the outliers is suitable to highlight surface defects such as scratches or bumps. An example of an analysis that can be made by using surfaces robust fit is shown in Fig. 6, where red points can be indicative of a scratch. Wherever other kind of defects are concerned, namely all geometrical defects explained before, it is necessary to consider together a group of snapshots. This way, it is possible to determine boundary snapshots that can be used to check spatial arrangements of a panel w.r.t. adjacent ones.
In these cases, identifying edge points is a crucial step to be performed, so that the gap between adjacent panels, just to make an example, can be correctly estimated and checked against accepted deviations from the expected measurement. Fig. 7. 3D edges as identified in the hatrack, in correspondence with the end of the left stow bin and the beginning of the right stow bin. By considering statistically the distance between the two edges point-to-point, it is possible to detect different defect types, such as gaps (constant distance, but not in the acceptable range) and parallelism (edges diverge more than a predefined threshold). Please note that the method is effective even between different hatracks.

C. 3D ROBUST EDGE DETECTION
The bi-dimensional domain of Fig. 4 can be effectively treated as a 2D image to robustly find the 3D edges of each surface, rather than work with unorganized point clouds as in [5]- [7]. To achieve this goal the surfaces are separately processed defining a binary mask M in which the surface points are represented by a 1 and all the other points are set to 0. A Sobel filter is then applied on M to extract the contours of the surface. Finally, two linear models = and = ℎ are estimated with RANSAC. The inliers of g and h define, respectively, the horizontal and vertical edge of the surface.

D. MEASUREMENTS OF GEOMETRICAL DEFECTS
The measurement strategies for geometrical defects are all dependent on the concept of distances between adjacent surfaces (neighbors). Generally, differentiations is limited to three constraints: 1) direction to be considered; 2) sampling point(s) where to evaluate the distance; 3) considered neighbors. In particular: • Gap: given a snapshot made of two surfaces 1 and 2 and their corresponding 3D edges 1 and 2, the gap is defined as the distance between the two edges ( )=| 1− 2| • Parallelism: given two snapshots 1 and 2 of the same object under inspection, the first acquired at the beginning of the assembly and the second at the end (as depicted in Figure 23), the parallelism is computed evaluating the distance between the two gaps =| ( 1)− ( 2)| • Step: given a snapshot made of two surfaces 1 and 2, the paraboloid models 1 and 2 and their corresponding 3D edges 1 and 2, as follows: • Mismatch of tolerance: given two snapshots 1 and 2 of two different objects under inspection that should be mounted aligned (as depicted in Figure 24), the mismatch of tolerance is computed evaluating the distance between the two gaps =| ( 1)− ( 2)|

E. SCRATCHES
Scratches are usually considered surface defects, since they are present on a particular surface and do not require to evaluate distances between adjacent surfaces. Surface defects could require the evaluation of colors, either by specific requirement (for example, checking for texture inhomogeneity on a panel requires color information) or as a mean for distinguishing particular spots that might signal situations such as scratches. In this work scratches are instead identified by means of small changes in the geometry of the acquired surface. A scratch is characterized by its shape (in particular the length) and the depth of it. Not all scratches need to be reported as defects but only the ones going over determined thresholds. This gives an order of magnitude of the accuracy that must be provided by the sensor and the processing algorithm to be considered "fit" for the task. Usually this accuracy is in the order of about 0,4mm.

V. EXPERIMENTS AND RESULTS
The algorithms described in Section IV have been implemented in C++ and Matlab languages. All the experiments have been performed on a workstation equipped with Intel Core i7 6820HQ CPU with 16 GB DDR4 RAM. 3D data are acquired by using the LMI Gocator 3210 sensor.
Test data represents the hatrack of an Airbus A320 aircraft. Experiments reported on this work have focused on testing the accuracy of the methodology. For geometrical defects, this implies verifying that the point clusters created by the 3d flood fill step, followed by the edge detection, are able to find boundaries related to adjacent surfaces and perform a range of measurements, all related to an estimation of distance between these parts.
In order, to test the accuracy of the processing pipeline, different parts of panels have been acquired and processed. Measurements obtained by the processing pipeline have been then compared against a "ground truth", established by using a caliper. Two tests are reported here.
In the first one, a gap measurement is considered for the overhead bins, in order to provide an example on the measurement accuracy that can be obtained while looking for geometrical defects.
In the second test, a snapshot concerning an area in which a scratch is present will be shown and analyzed.

A. GAP MEASUREMENTS BETWEEN OVERHEAD BINS
In the first experiment, the gap between two overhead bins is considered. Since the area covered by the sensor is not large enough to cover the whole height of the drawers, four snapshots have been acquired, as reported in Fig. 8.
As visible in the figure, each point cloud is most of the time sufficiently dense to be mistaken for a grayscale picture, with sparse points usually present near the border of the field of view, where the structured light assisted stereo reconstruction performed internally by the snapshot sensor is not able to recover depth information.
Each snapshot has then been individually processed. The processing results are visible in Fig. 9.
The processing pipeline identifies the boundaries of the clusters based on the measurement to be made. While looking for a gap defect, only the edges between the adjacent parts are needed for comparison. Each snapshot has then been individually processed. The processing results are visible in The processing pipeline identifies the boundaries of the clusters based on the measurement to be made. While looking for a gap defect, only the edges between the adjacent parts are needed for comparison. As highlighted in TABLE I, measurement deviations are very limited, even after considering other factors, such as the limited precision and accuracy in the measurements that can be obtained from a caliper, or the "height" on the hypothetical separation line in which the caliper measurement has been taken. However, it is worth pointing out that in this particular test, the two storage bins are part of the same hatrack, and hence a defect related to parallelism is very unlikely. It should be considered that this result has been achieved without relying on a 3D CAD model of the surface being inspected and therefore performance can be still improved upon.

B. SCRATCHES
Surface estimation obtained through the RANSAC process can also be used for detecting some surface defects, such as scratches. Fig. 10 shows the detail of a single surface as well as its robust 3D model estimated as reported in the methodology. Red points highlight the outliers of the RANSAC algorithm, thus proving that scratches can be effectively extracted with the proposed methodology, once anomaly extension and shape is appropriately filtered (e.g. outlier points are arranged around a narrow line). The scratch is barely visible to naked eyes, especially if the person near the panel does not put close attention looking for scratches. Algorithmic defect characterization reported a length of 5,36mm and a depth of just 0,4mm. It was also extremely challenging to measure with a caliper for establishing a "ground truth" due to the limited width of the defect that did not enable the use of the caliper depth probe.
For this reason, two additional scratches were available, with characteristics compatible with the usage of a depth probe. Results are reported in TABLE II.
The analysis of the differences in length and depth estimation between 3d point cloud measurement and the one taken with a caliper seems to suggest a different behavior, with accuracy negatively affected. In particular, the differences in measurements concerning the left case are negligible, with a "disagreement" of 0,3mm for the length and less than 0,01mm for the depth. On the other hand, the right reference shows a consistent deviation, especially while considering the estimated depth, showing the difference of about 0,5mm.
A possible explanation for this behavior might be related to the characteristics of the scratch and its relative position with respect to the 3d snapshot field of view. The scratch relatively high depth and narrow width, when combined to a likely unfavorable scenario for the triangulation of the cameras on the 3d snapshot sensor, might explain the limited accuracy and contrasting numbers, although further analysis will be required in this case for understanding the real nature of the issue experienced in this particular case.

VI. CONCLUSIONS AND FUTURE WORK
In this paper innovative methods of dense 3D point cloud processing for the automated inspection of aircraft interiors have been presented. The methodology is focused on a clever exploitation of domain knowledge to automatically perform robust surface analysis and 3D measurements. In its current state, the work mainly exploits the relative smoothness of the regularity of the inspected panels. This strategy, from our prior investigations, perform well.
Results on actual data demonstrate the feasibility of the proposed approach and its suitability towards the real time processing of dense point clouds. Future directions of this research will regard the integration of the proposed techniques on a prototype, as well as the development of machine and deep learning models to deal with this kind of 3D data.
Last but not least, while this work has not considered the potential benefits provided by the availability of CAD data models, their potential is difficult to ignore: first, it enables the definition of a specific path for the inspection strategy of whole areas, such as the cabin or the cargo, providing the relative spatial arrangements of different panel types; second, it enables to group together multiple acquisitions related to the same panels, since usually panels are wider than the field of the view covered by the 3d snapshot sensor, providing also a hint of the type of geometrical measurements (gaps, steps, etc.) expected in a determined area; third, at a more local level, it provides a way to better handle particular cases (such as hatrack handles) which might be misclassified in multiple clusters of points if only local smoothness was used as the selective constraint for identifying questionable zones.
Nonetheless, this work has proven the feasibility of the usage of 3d snapshot sensors bundled with the proposed methodology in aircraft assembly quality control, even for situations, such as the detection of scratches, requiring particularly high accuracy.