Comprehensive Pineapple Segmentation Techniques with Intelligent Convolutional Neural Network

ABSTRACT


INTRODUCTION
Pineapple is a tropical plant from Bromeliaceae family and one of the most consumption fruits around the world after banana and citrus.The government under East Coast Economics Region (ECER) program has allocated 7400 hectares land for pineapple cultivation in order to increase the production of pineapple [1].Pineapple grows under swarm climate, optimum water and no infestation.However, it takes about 24 to 39 months to yield fruit before it can be harvested.Moreover, to identify the harvesting time become a quite challenging.This is because the quality and sweetness of the pineapple are dependent on the time of harvesting, if too early will cause pineapple to become less sweet, meanwhile, harvest it too late will cause pineapple to become too juicy [2].Therefore, pineapple must be harvested before it become overmatured.
Currently, machine vision system can provide a huge advantage in helping farmers to identify the fruits' maturity and the optimal time for harvesting.However, the main problem that limits the capability of machine vision system is image recognition process and object detection especially in detecting the pineapple image.There are many researches have been conducted to address this problem.Mohammad et al. [2] used an image processing technique to categorize N36 pineapple.By using binary ellipse mask together with morphological normalized RGB to filter out the background, the research showed a promosing result.However, the used of binary ellipse mask has its own drawback where the segmented region did not include the entire region of interest.The minimum symmetrical distance method has been proposed by [3] to overcome the problem mentioned in above.The author managed to segment the entire region of interest from the background.However, the proposed method only works using symmetrical shape pineapple only.[4] have chosen Otsu's Method as their image segmentation technique before applying morphological process.Otsu's Method assuming the image contains two classes of pixels following bi-modal histogram (foreground and background) and it calculates the optimum threshold separating the two classes.Prabha and Kumar [5] also used threshold method to segment the banana from the black background.However, threshold method only works efficiently if the image contains two colors only.So, the accuracy will gradually decrease if the image contains more than two colors.Then the used of g-r gray image was proposed by [6] to remove the background from the image.
Based on the literature review, it was found that to extract fruit from background at the plantation level is very difficult and a few factors need to be considered.Firstly, the algorithm must be robust for both immature and mature fruits.Thus, most of the researchers only focusing on mature fruit because the intensity between mature fruit and background are easily to distinguished.On the other hand, the intensity between immature fruit and background is almost the same.Therefore, to apply the algorithm on immature fruit cannot be used without modifying the parameters.Secondly, the algorithm also must be robust against natural illumination.Thirdly, the fruit in plantation mostly cluster together or occlusion with other fruits, leaves or branches.Therefore, to improve the robustness of the algorithm, researchers have combined the process of color segmentation with other techniques such as feature extraction, edges detection, blob-based segmentation and others [7]- [10].

RESEARCH METHOD
Image segmentation is a very important process to remove the unwanted background from the main object in one image.In this case is pineapple fruit, in order to ensure that the output image has high accuracy.The process starts by the data acquisition, all the images going to Cascade Object Detector (COD) to confirm the fruit's location before going to Hue Value Segmentation (HVS) process.The overall procedure for this process is shown in Figure 1.Therefore, to make sure the unwanted background is successfully removed from the image, this work has developed a Convolutional Neural Network (CNN) as a searching engine to evaluate the results from Adaptive Red and Blue Chromatic Map (ARB) and Normalized Difference Index (NDI).

Data Acquisition
Data aquisition for image has been captured during day time with various illumination conditions at pineapple plantation at Pekan Nanas, Pontian, Johor.The captured images have been processed and finally all these datas are used to develop an algorithm for pineapple image segmentation technique.In this work, MD2 type pineapple is selected based on the suggestion from Malaysia Pineapple Industry Board (MPIB), which one of the most popular and highest demand in Malaysia.Figure 2 shows the sample images used in this project.

Algorithm
Figure 3 shows the flowchart of the proposed algorithm for segmentation process and detecting the pineapple object.Since the image used in this project consists of pineapple and background, image segmentation process is implemented to remove any unwanted background from the focus image.The proposed algorithm starts with data acquisition, by acquiring image from the camera.Since the size of the captured image is too large, this project has conducted a few tests to find out suitable scale to resize the dimension of the image.By reducing the dimension of the image, the computational time has been reduced.After that, the proposed algorithm begins to remove the background in the image by doing image segmentation.Firstly, Cascade Object Detector (COD) is implemented to detect the location of the pineapple in the image.By returning the bounding box around the detecting pineapple, the proposed algorithm will remove all image (pixel) outside this bounding box.By doing this, the algorithm can further reduce the computational time and increasing the accuracy for the next process.
Secondly, the Hue in Hue, Saturation and Value (HSV) color space is extracted and by setting threshold value obtained from the experimental test to detect ground and sky.Since the further image segmentation is quite difficult, it is necessary to remove any background that is easily to distinguished from the fruit.The Hue value for ground (orange) and sky (white) are easily differentiated from other colors.The threshold value has been selected based on these properties.Any value less than threshold value is removed (pixel intensity becomes 0), meanwhile, the value greater or equal than threshold value is maintained.In image segmentation, there are often problem where some images only work well using certain method and other images using different method.Therefore, if the same image is used from different method, the interest object cannot be segmented completely from the background and vice verse.Thus, this will be resulting in low accuracy algorithm if the multiple images are being used.Therefore, to overcome this problem, Adaptive Red and Blue Chromatic Map (ARB) and Normalized Difference Index (NDI) at the same time have been implemented.
The threshold value obtained from the experimental test is used in ARB to segment the potential pineapple fruit from the leaves.Similar to the HVS, any ARB value which less than threshold value is removed (pixel intensity becomes 0) from the image.This value is considered as leaves.On the other hand, ARB value for fruit is greater or equal than threshold value.After implement ARB, the result still shows the misclassified pixels and the edge of the detecting pineapple is not smooth.This is because ARB value for background especially for darker leaves having almost the same value with the fruit.Therefore, to remove misclassified error and smooth the edge of the detecting pineapple, this project has proposed the use of Template Matching Method (TMM) for further segmentation process.The major axis and minor axis are calculated based on the resulting result from the ARB to find the right template, which then is used to remove any misclassified pixels and smooth the edge of the detecting pineapple [7].
The implementation of ARB and NDI will produce two segmentations image, therefore, the Convolutional Neural Network (CNN) is implemented in this algorithm as decision maker.The purpose of implementation of CNN is to select the best segmentation image from ARB and NDI.Firstly, the CNN will determine the segmentation image produced from ARB and NDI whether the object is a pineapple or not.If the segmentation image is a pineapple, the algorithm will accept the segmentation image.Meanwhile, if the segmentation image is not a pineapple, the algorithm will reject it.However, if both segmentation images are classified as a pineapple, the CNN will go to the second step by using the weight produced when classify the image to determine the highest weight between ARB and NDI.Finally, the system will accept only the output segmentation image with the highest weight.

RESULTS AND ANALYSIS
This section presents the result and the analysis of the image accuracy from various segmentation stages such as Cascade Object Detector (COD) to detect location of the pineapple, Hue Value Segmentation (HVS) to remove ground and sky, Adaptive Red and Blue Chromatic Map (ARB) and Normalized Difference Index (NDI) are for leaves removal, Template Matching Method (TMM) for segmentation enhancement and edge smoothing and Convolutional Neural Network (CNN) have been used as decision maker to select the best segmentation image.

Cascade Object Detector (COD)
Firstly, a Cascade Object Detector (COD) has been selected to detect the location of the pineapple from an captured image.Refer to Figure 4, the COD returns the bounding box around the detecting pineapple and remove all pixels outside this bounding box.By implemented COD, pineapple can be located anywhere in the image and this method has increased the accuracy as shown in Figures 4(a

Hue Value Segmentation (HVS)
The second step is the selection of Hue parameter.The accuracy will increased due to the background in the image has been decreased.Refer to Figure 5, the ground has been successfully removed from the image and other objects are maintained.Finally, it has been found that the output image still can be considered as acceptable, even though a few interest pixels have been removed.

Adaptive Red and Blue Chromatic Map (ARB) and Normalized Difference Index (NDI)
The third step is to remove leaves from the image as shown in Figure 6 using ARB and NDI methods.It can be seen that, the ARB and NDI have successfully removed the leaves from the image as shown in Figures 6(b) and 6(d) respectively, while maintained the fruit area and shape.In addition, it has been found that the output images in Figures 6(b) and 6(d) still can be considered as acceptable, even though a few interest pixels have been removed.However, when using image in Figure 6(a), the ARB has considered all pixels in the image as background.This is because the ARB's value for the all pixels in the image is less than threshold values.However from NDI method, the result is much better as shown in Figure 6(c

Template Matching Method (TMM)
The result from ARB and NDI segmentation methods have a few misclassified pixels and the edge of the detecting pineapple is not smooth.Therefore, this project applies additinal image enhancement process for further segmentation using Template Matching Method.The different in shape and size of template have been constructed in advanced and store in the database.The right template is selected based on the major axis and minor axis.Then, the pineapple shape is constructed to produce a smooth edge and all the pixels outside this shape are eliminated as shown in Figure 7.

CONCLUSION
The image segmentation and detection algorithms have been presented to detect pineapple fruit based on color maturity indices.The results found show the used of Cascade Object Detector (COD) was very effective and has good potential in detecting the location of the pineapple.The use of Hue value with Adaptive Red and Blue Chromatic Map (ARB) and Normalized Difference Index (NDI) were very effective for background removal process from the focus image.The implementation of Template Matching Method (TMM) has improved the accuracy of the algorithm by detecting the shape of the pineapple and then eliminated other pixels with a smooth edge.Furthermore, the implementation of Convolutional Neural Network (CNN) as decision maker has increased the efficiency of the algorithm and at the same time increased the accuracy in segmenting the fruit from the background.Further research needs to be done using intelligent optimization technique for optimal maturity identification.

Figure 1 .
Figure 1.The overall process

Figure 2 .
Figure 2. Sample images with different stage of maturity and background

Figure 3 .
Figure 3. Flowchart for the proposed algorithm ) and 4(b) because the background of the selected image has been decreased.

Figure 7 . 3 . 5 .Figure 8 .
Result after applying template matching (a) Immature pineapple (b) Mature pineapple Convolutional Neural Network (CNN) The implementations of ARB and NDI have produced two segmentation images.Refer to Figure 8, Convolutional Neural Network (CNN) has successfully selected the best segmentation image between ARB and NDI.The implementation of CNN in the algorithm has increased the efficiency and accuracy by allowing the algorithm to have many methods to detect pineapple. ISSN: 2502-4752 Indonesian J Elec Eng & Comp Sci, Vol. 10, No. 3, June 2018 : 1089 Result after applying CNN (a) Result from ARB (b) Result from NDI (c) Result from CNN