Machine vision for low-cost remote control of mosquitoes by power laser

In this paper, we present an innovative and effective method for remote monitoring of mosquitoes and their neutralization. We explain in detail how we leverage modern advances in neural networks to use a powerful laser to neutralize mosquitoes. The paper presented the experimental low-cost prototype for mosquito control, which uses a powerful laser to thermally neutralize the mosquitoes. The developed device is controlled by a single-board computer based on the neural network. The paper demonstrated experimental research for mosquito neutralization during which, to maximize approximation to natural conditions, simulation of various working conditions was conducted. The manuscript showed that a low-cost device can be used to kill mosquitoes with a powerful laser.


Introduction
The purpose of this manuscript is to demonstrate the possibility of using a computer vision system for remote monitoring of the position of mosquitoes and their subsequent neutralization. The relevance of this work is confirmed by the presence of problems associated with diseases that are transmitted by mosquito bites [1][2][3][4]. Due to the fact that the problem with mosquitoes is becoming more relevant every year, it is advisable to use technological progress for solving this problem, in particular: neural networks.
For remote object detection, the characteristic of a camera is very important. A typical camera can correctly track small moving objects at several centimeters. This is a limiting factor when using machine vision to monitor mosquitoes. The speed of a mosquito is no more than 5 m per second. For a linear scan camera, the color of a mosquito is a moving few pixel. Since the surrounding color scheme plays a role, it is necessary to use color cameras. Monochrome cameras are faster, but they are suitable for laboratory tests, where artificial conditions of an experiment are made: the background is white, the mosquito is dark. The temperature of the mosquito is like the environment, which is why infrared cameras cannot be used. The size of a mosquito can vary from 1 to 5 mm, this is the main criterion for choosing the size of the sensor matrix along with the number of pixels, and for choosing a camera lens.
The following articles have a similar purpose-mosquitoes control. In the article [5], the accurate determination of the presence of a mosquito by its acoustic signature by the neural network was achieved. However, this research indicates only the presence of the mosquito and not its exact location. Mosquito image processing in the work [6] allowed researchers to detect mosquitoes and their location. In the paper, the photographs are in high resolution were used. The proposed neural network model was intended for classification and not for monitoring. In the manuscript [7], automatic conveyor of acoustic data collection is introduced. The machine-learning algorithm allows you to detect several types of mosquitoes offline by sound recording. The proposed method is intended for mosquito's classification. The neural network for the classification of mosquitoes was presented in the manuscript [8]. Mosquito monitoring was not considered. The next manuscript is the close topic of our research [9], which presents the results of mosquito detection using a camera with high resolution. In the paper, 1 3 monitoring for a close distance was considered. The results for determining the position of z are not presented.
Mosquitoes are several millimeters in size, but in the research, as a rule, for detection, larger objects are usedpedestrians, ships, cars, and crops [10][11][12]. Among insects, researches on tracking bees are popular. In these studies, a swarm is tracked and not a single bee [13].
It is worth adding that the Bill and Melinda Gates Foundation research on the introduction of laser mosquito neutralization technology as a promising method of combating malaria was sponsored. As a result, the company Photonic Sentry was created (https ://photo nicse ntry.com/). This company demonstrated a video on mosquito neutralization by laser but did not provide any technical characteristics of the device on the website.
In the paper [14], a system that uses a combination of optical sources, detectors, and sophisticated software to locate and destroy mosquitoes using a laser is presented. This research was implemented at the junction of the emergence of deep neural networks, and therefore, their capabilities have not yet been considered in this article. The authors studied the effect of mosquito speed on tracking accuracy. At the same time, sufficient information about the equipment was not provided. We used a similar method to control the position of the laser with mirrors. Auxiliary methods for determining the position of mosquitoes, due to the combination of optical sources and detectors that were considered in the paper, are very valuable and can be used by us in the future to improve the tracking accuracy.
One of the latest and most relevant works in this area was presented by Intellectual Ventures Laboratory, Bellevue, WA, USA [15]. The authors have achieved impressive results in the use of the laser and identified several key parameters required for accurate tracking and targeting of objects during their flight. Remarkably, several species of mosquitoes have been considered. It was not possible to take full advantage of the results, since the presented installation is difficult to implement, but we plan to introduce the ideas laid down in their implementation into our research in the future. The main goal of this research is to eradicate mosquitoes. To do this, you can choose two ways-to increase the tracking accuracy-which requires more processing time and more expensive equipment, or, as we did, choose low tracking accuracy. We used more laser shots and cheap equipment, which ultimately gives the same positive result. Since with accuracy (10%) of 10 shots per second, we can destroy one mosquito. At the same time, in the future, the accuracy of the device will be increased when using STM32 and deep neural networks. This improvement will reduce processing time and installation cost.
It is worth noting that the problem with vectors of diseases with the help of mosquitoes is primarily pronounced in countries with weak economies, so the solution to this gap should be in a low-price range.

Materials and methods
A mosquito is comparable in size with a few pixels for the low-cost camera (10 MP), an increase in the mosquito size of the camera leads to an obvious improvement in the result, but significantly reduces the area of online monitoring. According to flight dynamics, it is possible to predict what type of insect flies we are monitoring, for example, when flying, Drosophila melanogaster has several atypical movements [16]. Since the position of the camera can be arbitrary, the camera was calibrated by a coordinate system. The process of camera calibration is described in detail in the manuscript [17].
To use the pre-trained deep-learning network on the Raspberry Pi to monitor mosquitoes, it was originally intended to use deep-learning methods. But due to the limited amount of RAM on the Raspberry Pi 4 (1-4 GB) and because of the low processor speed of 1.5 GHz, the use of deep neural networks is almost impossible (ResNet > 100 MB, VGG-Net > 550 MB, AlexNet > 200 MB, GoogLeNet > 30 MB). In an attempt to use SqueezeNet, we managed to get a model with a weight of 5 MB, but even in this case, the result of processing the image for the presence of the desired object was about 1 s. Real-time detection with R-CNN, Fast R-CNN, Faster R-CNN, Yolo, and RetinaNet has the same recognition speed problem. The solution is to use NVIDIA Jetson TX1 and TX2-a special platform for computing neural networks. The main disadvantage is the high cost. Therefore, we focused on the methods that can be implemented on the Raspberry Pi, which allowed us to create a low-cost device.
One of the most common methods for detecting moving objects is frame difference, background subtraction, and analysis of the optical flux field. However, this method showed the low result for tracking.
The type is common mosquito (lat. Culex pipiens). The size of mosquitoes did not exceed 5 mm. Gender and age of the mosquito was not considered. Photos of mosquitoes were taken by camera from distances of 300 mm. Resolution of mosquito's image composed of − 250 × 250 pixels. In this work, the cascade of Haar was used. In the research, 1460 photos with mosquitoes were used as positive examples and 1500 photos as negative examples.
Power laser (15 W) is dangerous for people. Therefore, the command-"use the laser" should be given with maximum confidence, when the target is identified correctly. It is also necessary to know the mathematics of mosquito flight. Descriptions of the flight and the compilation of a mathematical model of the mosquito was described in the papers [18][19][20]. Similar research is presented in the paper [21], where the speed of a mosquito is determined by the expression, which was used in this work: where S max and S min are the maximum and minimum speed of a mosquito, F is a ramp function that takes values between 0 and 1.
In operation, the flight direction and speed of each mosquito are updated every time with units. The updated position of mosquitoes is calculated by where (x n , y n ) is the position of the mosquito at time step n. Number d is a direction vector that varies between tracking and tracing. V is the wind speed.
There are two main types of distortion: radial distortion and tangential distortion. In our case, radial distortion does not bring an error. However, there is tangential distortionbecause of which we have image distortions caused by instability in the camera position parallel to the image plane, due to the mobility of the developed prototype.
As a result, the coordinates of the pixels were recalculated by the following formula: where ( u corrected , v corrected ) is the pixel location after removing geometric distortions; k 1 , k 2 , and k 3 are the radial distortion coefficients; p 1 and p 2 are the coefficients of tangential distortion.
The next tools in the research were used: -Raspberry Pi 3 Model B +, Broadcom BCM2837B0 with a 64-bit quad-core processor (ARM Cortex-A53); -Programming language-python 3.6, library of machine vision (1) All experiments were carried out during the daytime. Therefore, the monitoring was carried out with a widely closed diaphragm. Vignetting and other aberrations are reduced to the minimum, and the depth of field was increased. The main problem for identifying a mosquito is the background on which the mosquito is located. The problem with the environments can be solved by correctly pre-processing the image, which allows us to identify useful features from an image.
In this research, the task was to show the prospects of using this method to fight mosquitoes. To simplify the task, the background was taken monophonic, which made it possible to determine the position of the mosquito in the image accurately, Fig. 1 (up to row). We considered with a diverse background, when the mosquito is next to the plant, Fig. 1 (low row). In this case, the standard means of computer vision are not able to extract useful features. As can be seen from Fig. 1 (up to row), if one tone behind the mosquito is obtained, it is quite easy to identify its signs, because of which it is possible to calculate mosquito coordinates without using deep neural networks. However, at the same time, the use of these filters with a diverse background (low row) does not give positive results. In the natural environment, the use of various filters and cascades of Haar will not allow to correctly track the object. In this case, the use of deep neural networks is feasible, for example-Mask R-CNN.

Technical description of the project
For the implementation of machine vision, we used OpenCV-4.1.1 library. The next researches have similar methods for insect identification [22][23][24].
To facilitate navigation, we developed an electronic board with determining the distance to the source of the object. The electronic has an audio sensor for mosquito sound control and an analog amplifier for an increased signal from the audio sensor, Fig. 2.
The buzzing of a mosquito is accompanied by a subtle sound since the oscillation frequency of its wings is from 1000 Hz and with a sound from 36 dB-depending on the species this is most likely to be the second harmonic, mosquito fundamental tones can range approximately between 300 and 700 Hz [25].
A board with a sound sensor can detect a mosquito sound of 30-50 dB in a radius of 70 cm with an accuracy of up to 8 cm. For research purposes, this is enough, since in practice you can use a sensor with a higher resolution. Initially, we wanted to add additional noise or record in real conditions. However, these technical issues have already been solved in the works [26][27][28]. The buzzing of a mosquito is accompanied by a subtle sound since the oscillation frequency of its wings is from 1000 Hz and with a sound from 36 dB. We measure the sound of the commander with Audio Sensor, MAX9814. With an external audio noise generator, we set the setting about the presence of a mosquito at 30 dB, 800 Hz. For checking, we measured the noise with Noise Level Meter PCE-MSL with frequency range 31.5 Hz and accuracy 2 dB (Manufacturer: PCE Instruments, USA).
In our case, the focus of the camera is configured in such a way that after focusing the window for monitoring is 5 cm by 5 cm in size, which, given the speed of the mosquito, will give the program in the Raspberry Pi3 time to identify it. When the sensor is triggered, the program usually has 0.1 s to detect the mosquito before it has time to leave the camera's viewing area. To implement the protection of a specific area against mosquitoes, Fig. 3b shows a diagram. The analog signal from the audio sensor is used to control the focus of the camera. It should be noted that today there are cameras capable of reading a car number for 1 km, so the distance for control largely depends on the resolution of the camera. The experimental design is shown in Fig. 3.
In the experiment, the system at the expense of a camera with the servomotor monitors mosquitoes in three boxes. When the audio sensor is triggered, the camera focuses on a predetermined area due to the telephoto lens. The audio sensor, when triggered, transmits an analog signal to the esp8266 which via Wi-Fi transmits a signal to a Raspberry Fig. 2 A device for converting an analog signal: 1-analog devices for converting and filtering an audio signal into voltage from 1 to 4.5 V, 2-an audio sensor, 3-terminals for connecting the board to esp8266 microcontroller Fig. 3 a Scheme of sensor action: 1-three boxes with mosquitoes, 2-laser machine with the telephoto lens, 3-servomotor for changing the position on the x-axis, 4-the distance between the camera and the boxes changed from 0.5 to 1.5 m, and 5-laser, 15 W, 450nm wavelength; b schematic representation of the control area, where radius it is the length of a control area Pi3. In the next step, Raspberry Pi3 sends the coordinates of the mosquito to the galvanometer and sends a signal for turning on the laser. The servomotor changes the position along the Z-axis-360°. This can allow one device to control a fully specified radius, Fig. 3b.
For the experimental scheme, a mathematical expression is derived that determines the dependence of the number of sensors on the area control: Average approximation error for this formula is 13.4%. We measured 115 times for each box and calculated the mean error. This dependence is shown in Fig. 4.
Due to the small size of the mosquito and the slight expansion of the image, the Haar cascades cannot identify the mosquito by exact features, such as wings and legs, therefore, the cascades determine only the general contours of the mosquito and the background. Due to this reason, we used OpenCV to make a bright background with filter-segmentation by cv2.kmeans. The algorithm of the developed system present in Fig. 5 Figure 6 explains the principle of operation of the mirror galvanometer.
The speed of the galvanometer is 20 kpps, that is why this element is the fastest in the device, it has enough milliseconds to hit the target. To control the galvanometer, we use Raspberry Pi3. Raspberry Pi3 receives the image from the camera and after that calculates the position of the mosquito. After that, the protocol computer sends 22 bits signal to the MCP3553 by SPI where the electronic signal processing board (Fig. 7, position 8) converts this signal in voltage from − 12 to + 12 V. The DC motor uses this signal to control the position of the mirrors, and therefore, the direction of (6) y = e 0.3906+0.1707x .
the laser. Figure 7 shows the developed prototype device, principle of operation and his dimension. Figure 8 demonstrates mosquito detection process. Figure 9 shows the complete installation of the equipment. It should be noted that the number of boxes with mosquitoes does not affect the project's efficiency. Significant

Number of audio sensors
The radius of the object to be monitored, m Dependence of the number of audio sensors on the radius of the object to be monitored Fig. 4 Dependence of the number of audio sensors on the radius of the object to be monitored

Experimental results
We did 150 attempts for each experiment (0.5 m, 1 m, 1.5 m), and Tables 1, 2, and 3 present the arithmetic mean for each case. Tracking is successful is positive when the camera detects the mosquito and tracks it with success. Detected by the camera-when the mosquito was in sight of the camera and was detected. Detected by the audio sensor-when the mosquito was closer than 10 cm away from the audio sensor and was detected. Neutralized by laserwhen mosquito was neutralized. The state of the mosquito was determined visually. The delay between attempts was 5 s. The box contained one mosquito each-to exclude the option with the accidental destruction of the mosquito. Each time when mosquito was neutralized, the experiment was stopped and continued after the mosquito was placed in the box. The experiment took place in the daylight, lux-150 lx. External light on the camera was excluded and external noise sources were not simulated. Mosquitoes are not classified by gender. Type-Common mosquito (Latin Culex pipiens). The mosquitoes did not exceed 5 mm in size. The readings are averaged; the average is taken from 150 readings with different samples from the array, when the distance between the camera and the box with a mosquito was 0.5 m, 1 m, and 1.5 m.
The relationship between the increase in distance between the tracking system and the mosquito was not noticed, which allows us to conclude that neural networks can be used for remote neutralization of mosquitoes. The camera identification of mosquitoes has high rates, which proves our suggestion about the possibility of using this method to neutralize mosquitoes. At the same time, the work has not yet succeeded in attaining high indices when precisely operating a laser, which relates to the technical execution of the project. There are difficulties with focusing the beam of the laser, because of which, even with the correct guidance to the mosquito of the laser, about 50% of the energy of laser passes by due to the small size of the mosquito.

Deep learning for mosquito detect
It is needed to improve concentration on the size of the laser beam on the mosquito. To test the feasibility of using deep neural networks on a stationary computer, the Mask R-CNN network was used. The Mask R-CNN network mask is one of the most advanced and highly efficient networks in the classification and tracking tasks. The model was trained following the recommendations of the author's Mask R-CNN [29], the initial weights were used from the developers, who are publicly available at https ://githu b.com/faceb ookre searc h/Detec tron. Accurate determination of the contour of the Fig. 7 The developed device: a photo of the device, b structural scheme, c device dimensions in mm: 1-PI cameras, 2-galvanometer, 3-Raspberry Pi, 4-laser rangefinder for checking the distance, 5-laser device, 6-power supply, 7-galvanometer driver boards, 8-analog conversion boards, 9-object detection, 10-laser beam  Installation during the experiment: 1-telephoto lens, 2laser setup, 3-microcontroller esp8266, 4-servo motor to change the position along the axis-z, 5-three sections with mosquitoes and audio sensors mosquito will allow us to aim the laser more accurately-to hit exactly at the center of the mosquito. In addition, we will be able to shoot at the wings and make it impossible for mosquitoes to fly, and this will allow using lower laser power.
For training, 400 positive images of mosquitoes and 1000 negative examples were used (for negative examples, other insects and plants were used). Positive results were obtained in tracking problems when expanding from 50,000 pixels on an area of not more than 25 cm 2 . In view of the small area, it is extremely inefficient in terms of mosquito control. The model was trained following the recommendations of the author's Musk R-CNN [30], the initial weights were used from the developers, who are publicly available at https :// githu b.com/faceb ookre searc h/Detec tron. Accurate determination of the contour of the mosquito will allow us to aim the laser more accurately-to hit exactly at the center of the mosquito. In addition, we will be able to shoot at the wings and make it impossible for mosquitoes to fly, and this will allow using lower laser power. For example, if we take the mosquito position above the center. The center of an area with a mosquito can be determined in real-time using an OpenCV-search for color contrast. The model calculation error is calculated using pixels. The model subtracts pixels from the image that came to the input of the model from the pixels in the image that came out at the output (as on cycle GAN), as a result, after that, it counts how many pixels are classified incorrectly. Mask R-CNN develops the Faster R-CNN architecture by adding one more branch that predicts the position of the mask covering the found object, and thus solves the instance segmentation problem The mask is just a rectangular matrix, in which 1 at some position means that the corresponding pixel to an object of the specified class, 0-that the pixel does not belong to the object. In our case, for Mask R-CNN, only one object was used for tracking-a mosquito, so the neural network quite simply found solutions for determining the contours of an object, focusing on the color edge contrast this reason in spite of so small amount image for training we received results with high accuracy. Finally, Mask R-CNN showed an accuracy of recognition of 90%, Fig. 10.   In tracking tasks in the recorded video with a mosquito with the same background, the accuracy was comparable with the results obtained on the Haar cascades, Fig. 11.
The results of using deep neural networks can be significantly improved using more images to train the model.

Discussion
The accuracy of laser effectivity is not more than 10%. However, the developed installation within 1 s can make 10 shots. As a result, this installation can destroy one mosquito per second with a probability close to 100%. The improvement of this result is possible due to the use of more accurate equipment.
We proposed an innovative method for remote monitoring of mosquitoes using a camera, where sound sensors were used to determine the initial position of mosquitoes.
The article conducted theoretical studies aimed at monitoring mosquitoes at a remote distance. As a result, mathematical expressions were obtained that determine the position of the mosquito when it is located at a remote distance from the camera. A formula is derived by which the necessary number of sound sensors are calculated to protect certain territories from mosquitoes. The main problem with mosquito monitoring is its small size. At a meter, a mosquito for the camera appears as a few pixels, which makes it difficult to extract any useful characteristics from the image. In this study, we concluded that to identify a mosquito in an image with an area of 100 cm 2 , it is enough to use an image with 24,025 pixels. These settings will allow you to process images at a speed of 20 frames per second.
We presented mathematical modeling of mosquito flight that allows us to predict the flight mathematically. From the moment, a mosquito's position is detected by the camera and the computer calculates its position, the mosquito's position on the image changes within 1 cm. This change is critical for the galvanometer. However, the mathematical modeling of mosquito flight allows the galvanometer to direct the laser beam in advance to the point where a mosquito will be.
We solved problems with remote monitoring, for this, we adjusted the focus of the lens, and its position in the x, y, and z-axes. Which allowed us to monitor the mosquito's position, regardless of the distance between the object and mosquitoes.
This algorithm managed to unite such components as mechanics, optics, mathematical modeling, and machine vision. From the point of view of autonomous systems, this study can be used in various fields, such as monitoring different remote objects.
The article conducted theoretical studies aimed at monitoring a mosquito at a remote distance. As a result, mathematical expressions were obtained determining the position of the mosquito when it is located at a remote distance from the camera. A formula is derived by which the necessary number of sound sensors are calculated to protect certain territories from mosquitoes.
As an alternative to using sound sensors to identify the approximate location of mosquitoes, continuous monitoring of a given area can be used, Fig. 12a. The room is divided into sectors, where the camera monitors the box with x-and y-axes, then the focus of the camera goes deep into the room along the z-axis (1,2,3 panels, Fig.12a) and continues scanning along the x-and y-axes. At the same time, to attest to the depth of the mosquito's location along the z-axis of the mosquito, we can use the Gauss filter, in Fig. 12b, c. Room-monitoring scheme, b the mosquito is focused, and c mosquito is out of focus A promising direction in the development of the developed installation is the use of STMicroelectronics STM32 microcontrollers. It is given that X-CUBE-AI has an expansion pack for STM32CubeMX. This extension can work with various deep-learning environments such as Keras, TensorFlow, Caffe, and ConvNetJs. Thanks to this, the neural network can be trained on a desktop computer with the possibility of computing on the GPU. After integration, the optimized library for the 32-bit STM32 microcontroller is used. Machine learning is planned to be implemented on the edge computing of the STM32 microcontroller. This will reduce the cost of the device and reduce the size of the device. At the same time, peripheral calculations will reduce the speed of image processing, and in our case, this is the most time-consuming procedure. Therefore, by reducing the image processing time and increasing the processing power of the processor for working with deep neural networks, we can achieve results in which dozens of mosquitoes will be destroyed within 1 s.
All lasers from 1 mW start to pose a hazard to the eyes. In the future, we will consider the possibility of using a device with tracking a person in the laser field of view for safety tasks.

Conclusions
The article proved that modern advances in machine vision and machine learning are enough to use a powerful laser to neutralize mosquitoes, which, given the scales of poverty, caused by transmission diseases, mosquito bites, is a promising direction for the protection of certain areas from the presence of mosquitoes. The results of this work can be significantly improved when using another type of equipment since the accuracy depends on the technical characteristics of the camera, servo motor, and sound sensors.