Machine vision for automatic inspection of pin through hole components assembled on a PCB

Automatic and innovative equipment was developed to inspect the assembly of pin-through-hole components on printed circuit boards. Its performance was tested on an industrial assembly line of motherboards for desktop computers. The test was conducted to inspect, under the binomial criterion Pass/Fail, the process of inserting PTHDIMM components on semi-finished PCBs. Using the concept of Inspection Accuracy in binomial classification analysis it was estimated that the accuracy of this inspection is greater than 98%.


Introduction
Manufacturing systems usually require automated inspection and test methods to guarantee quality. Methods available today, such as Machine Vision-MV can be applied in all manufacturing processes. The application of computer-based image analysis and interpretation is a technology that has demonstrated its contribution. It improves the productivity and quality of manufacturing operations in virtually every industrial segment [1]. The largest MV adopter by far is the electronics industry [2]. In the electronic industry, MV is being used to inspect PCBs for conductor width spacing; populated printed circuit boards for completeness, lead or Pin Through Hole (PTH) components post solder inspection for solder integrity. Inspection is performed not only as a means of sorting reject conditions but also to provide feedback to process performance [3]. Many electronic PCB manufacturing process companies employ MV systems to verify the presence and completeness of surface mount devices (SMD) and PTH components.
The equipment designed, developed, tested and presented in this paper gathers in itself the typical characteristics and methodology described above. The application of this functional MV equipment, endorsed by the acronym LIS (Light Inspection System), enables the identification of critical and highly specific defects of DIMM-PTH components assembly on Printed Circuit Boards -PCB [4]. This identification is achieved by the development of equipment containing a special system of lighting, a CCD camera displaceable on electromechanical actuators and feed backed for proper positioning relative to the device under test (PCB). This equipment allows the acquisition of images in a microcomputer and its subsequent interpretation and decision making. This inspection is performed using an application specifically designed for this purpose.

Description
This MV equipment was assembled in an aluminum metal case ( Figure 1). A video camera and a 45 o optical mirror were installed on a manual vertical actuator, which moves by step-motors on the horizontal X-Y directions. Three linear and horizontal arrays of collimated white Led diodes provide laminated light beans to locate the PTHs. This logical structure is shown schematically in the insert of Figure 1. Our MV equipment includes the following functions: Laminated light beans -to locate the PTHs; Optics -To couple the image to a sensor; Sensor -To convert optical image to analog electronic signal, Analog-to-Digital (AD) Converter -To sample and quantize the analog signal; Image Processor/vision engine -Includes software or hardware to reduce noise and enhance, process, and analyze image; Computer -Decisionmaker and controller; Operator Interface -Terminal; I/O channels to control and to process; and finally a Display -to make visual observations.
The loading/unloading of PCBs is provided by a slider located on the upper case surface (Figure 1). The beans of the LEDs linear arrays are collimated by cylindrical lenses. A micro switch detects the opening/closing slider load PCB window. An adjustable cradle allows loading/unloading PCBs of different dimensions. Sources of power and control of each of its electronics and optoelectronics elements complement the internal components of the equipment. A MS Windows CPU containing an IEEE-1394 interface is controlled by a Lab View application.   (4) and an optical mirror (5). (c) Image taken from two DIMM connector pins inserted into a PCB. The bright spots are the images of the metal pins through the PCB which reflect the light from the LED arrays.

Image acquisition, processing, and procedures for operation
The application of MV technology involves dealing with many visual variables. Even simple applications have to contend with variables that a person performing the same task can easily dismiss. Lighting and optics in many applications can be optimized to enhance the contrast associated with the variable for which a system is purchased or, conversely, can assist in diminishing the influence of the variable on the data required to make a reliable "vision/decision." In a given MV installation, dedicated lighting is recommended. In a MV system, the process of image acquisition is critical to a functional standpoint. The intrinsic variables of the acquisition process such as variation in brightness, signal-to-noise ratio influence the overall system reliability. A good imaging result requires less computational resources image treatment. Thus, the image obtained by the camera varies for the same product, which makes the image processing an important step to address this factor. Much of this variation is mainly due to the scattering of light on the object surface variations (a dirt or dust, for example). Lighting is dictated by the application, specifically, by the features of the object itself and the associated counting task. As the application here is inspection, the specific task determines the best lighting features for flaw detection. The specific lighting technique depends on the object's geometric properties, the background, and the data to extract from the object. Figure 2b illustrates an example of an image obtained with our optical arrangement. In this figure, one can see the sequence of groups of bright pixels generated by light reflected from the metal tip of the 2x128 pins that form part of two DIMM connectors. It is also possible to perceive at the left side of the image the mechanical supports that hold this long PTH component on the PCB. The lighting of the area of interest is performed by linear Led arrays (intensity controlled by current), where the beams are collimated to improve the ratio of the observed objects and the image of the bottom surface of the PCB. As each DIMM connector, consisting of 240 pins, is long (approximately 140 mm in length) and narrow 10 mm wide, in order to obtain images with appropriate resolution was necessary to capture two images (left and right) from each DIMM connector to inspect all of its 240 pins.
The inspection processing here is to count the number of bright blobs as shown in Figure 2b, which correspond to the existence of a region where, in the PCB, have been inserted two DIMM connectors. The total area allocated to these connectors on PCB is approximately twice the area shown in this figure. To locate this area is necessary to determine the position of the centric point or distribution of staining. From this position, is circumscribed a rectangle that delimits the minimum image area to be inspected. This procedure eliminates large number of spurious noises caused by reflections from other regions in the image. Once the pins are located at predetermined distances, it is possible to infer the correct number of pins by counting the number of spots, and no difference is related to a defect, whether the bad positioning of the connector or that there of bent pins.
Our MV system manipulates, via software, images in the spatial domain. In short, image-processing procedures include image enhancement, feature extraction and image analysis/classification. Enhancement is generally performed to remove unwanted clutter or noise. Feature extraction involves histograms, segmentation, masking (structuring an element), line thinning, and others. Image analysis/classification involves region labeling, line segment labeling, histogram, pixel counts, among others.
The procedures used to extract information from images (Figure 2b) are well-established procedures of MV operations [5]. They were used in the following sequence: binary intensity threshold, dilation, erosion, location of the block centroid, rotation, location of inspection areas and finally pixel cluster counting. Images obtained by CCD or CMOS digital cameras are generally colored (as is our equipment). In order to use basic inspection algorithms in the acquired image, binary processing was employed as the first image operation. Since the PCBs are manually loaded, their inspection location relative to the CCD camera are not preserved from load to load which make the complex image operations described above necessary.
Prior to the process of pin counting, an image processing filter is employed to restrict the inspection area. This filter prevents that noise outside the area of interest be counted as valid data. Thus, counting the number of bright spots is restricted to an area (image mask) whose center coincides with the center of the light distribution of spots. This restricted area, is repositioned for each image in order to minimize the effect of displacement generated by board loading. Figure 6 shows an inspection masks (gray rectangle). The existence of 112 pins is expected the mask so that any different number is considered a failure.

Signal to noise optimization
Groups of white pixels in the acquired image ( Figure 3) by LIS, which have no corresponding to through-hole pins expected at the predefined positions on the PCB, represent errors in the final inspection result. Therefore, treatment and exclusion of these unwanted pixels (noise) increases the accuracy of the MV inspection. The most critical pixel noise originates from reflection of the collimated and skimming light at points adjacent to the metallic connector pin holes. In some cases, the white areas generated by these spurious reflections are comparable to the white area of real pins which makes the operation of filtering useless. Image dilatation was selected to deal with this situation [6]. A binary processing the neighborhood rules established that a given pixel in the output image is white only if any of its eight closest neighbors are also white. This has the effect of growing white regions, and closing interior holes. For each new batch of PCBs LIS needs to be configured and calibrated [7]. Therefore, it is required to estimate the signal to noise ratio that stems from taking many pictures (multiple PCBs) over time. An image acquired by the camera can be treated as an array of intensities I (i, j) where i varies from 1 to n, and j from 1 to m being either integers representing the image size in pixels. In addition to the indices which determine the position of (i, j) pixel in the image, we can associate a different index k representing the image time course (time discretization). Hence, I(i, j, k) represents a set or sequence of matrices that display intensity fluctuations in the corresponding 8-bit monochrome image of an object at a certain instant of time (commensurate with the exposure time). In this case, these fluctuations are due to internal and external factors. Among the internal factors are fluctuations in the signal level in the CCD array and fluctuations in the illumination intensity. Among the external factors, small displacements or translations in the position of each plate, measured from an arbitrary origin, should be considered. Thus, we can define, for every instant k: Where <I(k)> is the mean intensity on that area and (k) 2 is the variance associated to the discrete instant k. On each region delimiting a 'pin', one can define a sub-region to sample the signal-to-noise ratio generated by the combination of the camera and lighting system (Figure 3). In a sub-region N.M (N <M and n <m) around each white pixel region, a square area is defined with, for example, 10x10 pixels for feature sampling. Those square areas are sampled from two regions, a 'background' Ibackgroud(i,j,k) and a 'pin' area Ip(i,j,k) ( Figure 3). Each "pin" or white pixel region can be represented, for example, by arrays of 10X10 pixels at the center of a white pixel region by a set <Ipino(k)r>, where r varies from 1 to N, where N is a fixed number of pins expected in the image. <Ipino(k)r> is the mean value of the intensity within each subset as previously explained. We then calculate a grand-average and variances for the set of pins over a period of N.k times samples as: Which are average intensities over time for each of the N pins. The same procedure is applied to the background to obtain average values <<Ibackground>> pins, one may use the following procedure: Extract the background level for each adjacent pixel region from a 10x10 matrix as explained above; extract the intensity level for, e. g., a 10x10 matrix centered on white region corresponding to a pin; calculate average levels <Ipin(k)r> , I background (k) r  and associated variances (averages for all pixels in the region); determine the 'grand-average' and variances for all pixel regions; Calculate S/N as defined by Eqn. 3 for each pixel region and background.
The function F (Eqn. 4) depends on various geometrical and optical parameters (defect size, relative positioning, alignment, contrast, hue and image intensity etc.). F should be maximized in order to obtain optimum values for these parameters. This function embeds in itself temporal variations generated by all expected changes in the illumination system, PCB loading and provides calibration of the MV system. This methodology is applied to LIS during the configuration phase for each new batch of PCBs to be inspected. This procedure is done in a non-automated way.

Results
The complete inspection test is performed in two stages. First, right side of the DIMM connector is inspected, and then it left side. At each step, the application indicates on the screen partial inspection results. If the unit passes or fails, the result is shown on screen (Figure 4a). This SW application also allows enlarging the image to diagnostic as shown in Figure 4b. If the result is compliant, the test is terminated and the PCB approved. Partial results of a particular batch of PCBs are also displayed as shown in Figure 5. In the case of a non-compliant result, the test is interrupted and the region of potential defect is displayed on the screen as shown in Figure 4a. The developed equipment was tested in an assembly line with a group of 469 sub mounted PCBs. The total number of pins to be tested was 225120. The average loading-unloading time plus optical inspection itself of two DIMM connectors (per board) was 20±3 seconds. The number of insertion failures found with LIS in the 469 PCBs was 18, representing a total failure rate of 3.8%. This rate was later confirmed by the company division of quality sector. The failures were divided into two types: partial (miss) insertion (89%) and bent pins (11%). According to the well-established concept of rating binomial [8], the percentage of false failure (FF) or false negatives is the percentage of erroneously identified PCBs without any real failures.

Figure 5
Short report of results obtained after the inspection of 56 PCBs.
The number of FF mounting DIMM connectors identified by LIS in the 469 PCBs was 23. This means a 5% of FF. Three factors caused this: 1) Presence of dirt in the inspection area, 2) Error in loading the PCBs, 3) Characteristic of PTH own machine which has an intrinsic signal/noise ratio for this application .As can be seen later through a visual inspection on non-compliant PCBs, only 17% of false-negative results are related to intrinsic problems of LIS, which represents 0.9% of FF observed

Conclusion
This article describes the development, and a pilot test of innovative automatic equipment designed to inspect the assembly of Pin-Through-Hole components mounted on PCBs. The application aims to identify critical defects occurring during the assembling of PTH components on PCBs. The equipment allows the acquisition of images, and its subsequent interpretation and decision making. The performance of this equipment was tested on an industrial assembly line of motherboards for desktop computers. A pilot test was conducted on-line in an industrial assembly line of PC mother boards. In this pilot test, two DIMM connectors, containing 2x240 pins, were manually assembled on each of 469 PCBs. The number of PTH pins inspected for its correct insertion into all PCBs was 225,120. Using the well-established concept of inspection accuracy in binomial classification analysis to this pilot test, it is estimated that the accuracy of this equipment is greater than 98 %.