Novel Noise-Robust Optoacoustic Sensors to Identify Insects Through Wingbeats

For certain countries, the production of olive oil and fruits is a significant percentage of the total gross income. Fruit flies affect a yearly crop loss calculated in billions of dollars worldwide. At the top of the hazard scale, certain mosquitoes and flies transmit harmful organisms that cause serious illnesses and even death to humans, pets, and livestock. In this paper, we develop a novel, noise-robust optoacoustic sensors to record insects' wingbeats. Our experiments with various insects of economic and social importance demonstrate that the sensors deliver high SNR recordings of the wing-flap. This paper emphasizes the development of the sensors and successfully analyzes the wingbeats of many insects of economic importance. Accurate analysis of insects' wing-flaps will allow the embedding of the sensors in common traps. The goal is to achieve species classification and remote monitoring of crops, which is currently carried out manually, and to develop reliable decision making regarding the initiation of large-scale spraying.


I. INTRODUCTION
I N THIS work, we describe novel high-precision and noiserobust optoacoustic sensors that can record the wingbeats of insects. Certain insects, e.g., fruit flies of the Tephritidae family, have a great impact on international marketing and world trade of agricultural products because they can infest a wide range of commercial and native fruits and vegetables [1].
We offer a small sample of referenced figures, mainly to present the serious impact of these pests on economies, which may not be well known in other research areas: a) The olive fruit fly Bactrocera oleae (Gmelin) (Diptera: Tephritidae) is the world's most serious insect pest of olive fruit. It inhabits the Mediterranean area, Africa, the Middle East and North America. If left untreated it is held responsible for losses of 80% of oil value and up to 100% of table cultivars and actually effects economic damage of hundreds of millions dollars per year [2], [3]. b) Ceratitis capitata (Wiedemann) (Diptera: Tephritidae) causes extensive damage to a wide range of fruit crops. It is native to the Mediterranean area but has spread invasively to many parts of the world, including Australia, Asia and America [4]. c) West Nile virus, malaria, certain forms of encephalitis and other infectious diseases that annually prove lethal to humans are transmitted by a number of mosquito species, such as Aedes albopictus, Aedes aegypti, Culex pipiens, Anopheles gambiae and many others.
Insect traps of various shapes are commonly used to determine the nature, density and location of such pests; they are currently inspected manually [5]. Manual inspection is tedious and unreliable and prevents large-scale spatial and temporal deployment. A sensor that could reliably analyze the sound spectrum of a flying insect entering a trap would allow the recognition of the species and thus radically change the way monitoring is carried out.
The acoustical properties of the flight activities of economically and socially important insects have drawn much attention even prior to the wide acceptance of microphones. The research literature is rich in studies of the acoustical properties of various insects, their relationships to temperature, their flight habits and their communication [6]. However, a device based on microphones functioning in the field is exposed to an uncontrolled number of audio sources that will be recorded by the microphones regardless of the trap configuration [7]. Because the number of audio sources, and possibly also their spectral content, is unknown and timevarying, their practical use in the field for reliable counting and recognition of flying insect identity is limited. The present work reviews and builds on optoelectronics applied to this specific task; it does not address camera vision solutions.
Efforts to study insect wing motion can be traced back to 1939 [8]. Chadwick refers to acoustic and optical methods applied to the study of insect wing-flaps that are based on techniques that date back to 1827. He describes a stroboscope approach in which the cyclic motion of the wings appears still when a light source is synchronous with the motion to be observed, and therefore the frequency of the wing-beat can be derived. The use of the wing-beat frequency to classify insect species was explicitly stated for the case of Drosphila pseudoobscura in [9]. In the early works of Sotovalta in [10] and his contemporaries referenced in this paper, one can also see attempts to establish a clear connection between wing-beat frequency and perceived frequency using all technical means of the time and even subjective acoustical observations.
In 1955, Richards discovered by chance that insects flying between the sun and photoelectric cells produce fluctuations in the light intensity received that can be converted to audio [11]. Repeating the experiment with insects in a transparent box, he reported that wing fluttering is imprinted in electrical fluctuations due to the wings' partial occlusion of the light path between an emitter and a receiver. Both Chadwick and Richards make an implicit claim that different species have different wing-beat characteristics and that the optical sensing of their wing-beat can possibly lead to insect classification. Subsequently, the notion that the perceived wingbeat tone could potentially be used as a taxonomic characteristic appears in many publications, e.g., Sawedal [12]. The work of Richards inspired a generation of researchers to use different photosensors to detect fluctuations in light intensity caused by reflections of flying insects [13]. By that time, it was clearly suggested that automatic instrumentation and various types of photosensors could be used to discriminate among flying insects by analyzing not only the fundamental frequency associated with the wing-flap but also the harmonics produced by this action [14], [15]. Various configurations of opto-electronic systems have been developed to study different insects [16]- [19]. An unseen connection between lasers, bees and explosives can be found in [20]. Despite these efforts, until 2006, all publications were constrained to laboratory experiments, probably because the electronics of the time and the diversity of disciplines involved made it difficult to develop a stand-alone prototype that could be employed in the field to monitor insects of economic importance. Although not a widely known contribution, Kirsch et al. [21] introduced a beta-system approaching a device that could recognize flying insects in the field. In [22], the authors presented a stand-alone device that would count and transmit counts of Bactrocera Dorsalis from the field. This system, however, used an optical system that had the basic functionality of on-off, i.e., photo-interruptions due to the passages of insects that were counted. It did not involve spectral analysis of recordings and therefore could not discern insect species. Keogh E. and his team revived the idea of optically sensing insects and gathered recordings from optical sensors [23]. The same team made a part of the database available and organized an open competition for classification algorithms to crowdsource the answer to an important question: the extent to which recordings of optically sensed wing-beats generally lasting between 50 and 100 ms allow separation of species. This step is important because past results indicated medium scores, probably because the electronics of the time were not sufficiently advanced and the authors performing the classification experiments were not dedicated data scientists but wanted to show a proof of concept. The outcome of this large experiment [24], [25] was that optoelectronic recordings combined with advanced signal-classification techniques could indeed return highly accurate recognition results for several species in flight. Despite these efforts, at the time of writing, to the best knowledge of the authors, there was no definitive demonstration of a device that classifies insects and operated in the field using optoelectronic principles. The first actual prototype trap that integrated all these components and was functional was presented in [26], where the widely adopted McPhail type trap [5] was modified to be electronic. In the present work, we describe a more advanced optoacoustic device that can monitor the wingbeats of any insect species. Certain butterflies and Lepidoptera flap their wings with frequencies of a few hertz, but the vast majority of insects flap their wings at frequencies between 100 and 1000 Hz. Our sensors are based on photodiodes that can operate at much higher frequencies and therefore can perfectly resolve the fundamental and overtones of wing-flapping.
In this work we carry out controlled experiments on insects of great economic and social importance, namely, the devastating pest for olive trees, B. oleae, a mosquito vector of malaria, A. gambiae, and Apis mellifera (the western honey bee).
The sensors described here are noise immune to the interference commonly found both in the lab and in the field because they perform modulation/demodulation of light at high frequencies. They can work unobtrusively either in the dark or in sunlight, and we will demonstrate that their accuracy in analyzing the spectrum of a wingbeat recording matches that of the microphone. The simultaneous recording of insect sounds using optical devices and microphones and their comparison is currently absent from the research literature.

II. MATERIALS AND METHODS
We describe in detail the construction of an optoelectronic sensor that is able to record the elusive motion of a wingbeat. The sensor must be able to operate in illumination conditions ranging from bright light to total darkness. The energy of the wingbeat signal is very small; therefore, the smallest interference can mask the useful signal. The construction of our sensor is virtually immune to interference from electric devices, power supplies and lamps typically found in laboratories where insectaries typically exist. This immunity is achieved by employing appropriate modulation to the emitting light while the demodulation at the receiver end is performed at high frequencies. We also study different light sources (infrared and laser), and we demonstrate that they perform almost equally with a slight objective advantage to laser sources. We also experimentally demonstrate that the analytic accuracy of the spectrum of the optoelectronic sensors matches that of microphones; this is the first such report in the literature.

A. Recording Devices
In view of practical applications in the field, there are clear advantages to using optoelectronics instead of microphones for recording insects' wing-flaps: a) the optoelectronic device records an event only when the path from the emitter of some form of light (e.g., infrared, laser) to the receiver (e.g., phototransistor, photodiode, photoresistor) is interrupted, whereas a microphone picks up sound from all-directions, b) optoelectronic devices return a very high signal-to-noise ratio, whereas microphones record all sound sources (e.g., mechanical sounds, birds, cicadas, weather effects), and therefore, the recordings can become very noisy,  c) microphones, although they can be protected in several ways against weather conditions, are more vulnerable to open field conditions.
A block diagram of our optoelectronic device designed to record the very low-level signals of a wing-flap is shown in Fig. 1. Amplitude modulation is intended to shift the low frequencies of the wingbeat to high frequencies and suppress interference due to electronic light appliances in the lab. These appliances produce strong optical interference at multiples of the main power supply frequency; other sources of electrical/ electronic interference (mainly LED lighting) are possible as well. Wing-beating motion modulates the amplitude of light scattered from an insect and the frequency of the modulation provides information that can be used to categorize its species. We have developed two versions of the optoelectronic device; the emitter is either a laser or an infrared light source.
The receiver (Fig. 2) is an array of photodiodes connected to a low-noise operational amplifier (OPAMP). The OPAMP is selected to have large bandwidth (54 MHz), a quick response time (640 V/µs), low noise levels (6.6 nV/ √ Hz) and a small input bias current of 0.6 pA. We did not use phototransistors because their response times of 10 µs are slow compared to those of photodiodes (response time 40 ns). The output of the OPAMP drives a switched capacitor band-pass filter (Fig. 3). This 8 th -order filter (based on IC1 in Fig. 3) has a 60 kHz center frequency and a 10 kHz bandwidth and therefore rejects the signals from conventional light sources by at least 60 dB, allowing only the signal from the emitter after amplitude modulation by insect wingbeats. The insect wingbeat causes a modulation in amplitude of the light beam from the emitter to the receiver; the signal is demodulated through a precision rectifier (IC2 in Fig. 3).
The output of IC2 drives a low-pass switched capacitor filter (IC4 in Fig. 3) with a 3 kHz cut-off frequency that cuts off the carrier at 60 kHz; IC5 is a filter-buffer to eliminate any clock feed from the switched capacitor filters. The power supply ( Figure 4) has input voltage varying from 3.3 VDC to 5 VDC and three output voltages: +8 V, −8 V and 3.3 V. The input voltage can be a 3.7 V lithium battery, the USB voltage of any PC or a single 220 VAC to 5 VDC power supply. Figure 5 shows the microcontroller circuit; it emits light pulses at 60 kHz frequency instead of continuously and the receiver, composed of an array of photodiodes and filters, demodulates the signal back to acoustic frequencies. This circuit produces the 3 MHz clock to the band-pass switched capacitor filter, the 150 kHz clock to the low-pass switched capacitor filter and the pulses for the laser and led infrared emitters. The laser emitter has a beam splitter that opens the laser beam as a thin sheet, directing beams to terminate on the array of photodiodes. The infrared light is diffused and no splitter is used. The details of all elements are included in Appendix.
Figures 2 through 5 and the Appendix provide all necessary details needed to build the optoacoustic sensor. With the exception of [13] from 1979, to the best of our knowledge, there is no publicly available description of an optical sensor that can record movements of insects at the scale of a wing-beat.

B. The Equivalence of Optical and Microphone Acoustics
The waveforms generated by the flight sounds of Musca domestica (the common fly) were recorded and  analyzed simultaneously using the optoelectronic devices and a small-aperture microphone (UMIK-1 omnidirectional measurement microphone), and we compared these two different modalities. To avoid subjective opinions of which produces the best results, we took measurements of each light source (laser and infrared) while simultaneously recording the wing-flap with a microphone. Therefore, we always had the two different modalities recording the same event. We kept the microphone recording as a reference, and we measured the difference between the optoelectronic recording and the microphone signal. The distances between the receiver and the emitter and the insect were fixed for both experiments. The optoelectronic apparatus and the microphone both sensed the wing movements but are actually two modalities functioning on different principles. The optical one is based on interrupting the light between emitter and receiver. The wings are semi-transparent membranes that are usually strengthened by a number of longitudinal veins. Cross-connections between veins form closed 'cells' in the membrane. The wing shape, the patterns on it and the complex movement affect the waveform of light intensity recorded by the photodiode. The microphone is indifferent to transparency issues and body movements. The shape of the wing and the rhythm that is vibrated play the key role in setting the air molecules in vibration during the complete wing stroke. The optical and sound recording modalities are based on completely different principles. Yet, the resulting waveforms are almost identical, as shown in Fig. 6 and Fig. 7. In Fig. 6, we have recorded the same adult fly using an infrared optoelectronic device simultaneously with a microphone. We present all phases recorded side by side using both modalities, and we see the striking resemblance in the spectrograms. Both recordings are rich in harmonics; the fundamental frequency often drifts slightly. Both the fundamental and the harmonics up to 2 kHz are clearly resolved. The microphone was able to resolve harmonics above 2 kHz, but this also depends on the relative position of the microphone with respect to the insect. In this case, the microphone was placed 1-2 cm away from the insect, whereas the distance from the emitter to the receiver was 11 cm, and the insect was placed approximately in the middle.  microphone recordings. All recordings are almost identical, with the laser having a slightly sharper analysis of the lower part of the spectrum, but the objective distance measures (see following paragraph) show that the infrared is slightly closer to the microphone. All insect measurements in this work were carried out in a controlled laboratory environment of 25°Celsius and 55% humidity.

1) Distance Measures:
Listening to the recording of either the laser or infrared version of the optoacoustic device sounds quite similar to the microphone recording to the point of being difficult to discriminate between them by ear. The microphone sounds slightly more 'natural' to the ear. Because most reported work is based on audio recordings (see [27], [28] and the references therein) and much less on optoacoustic recordings and the audio reflects our everyday experience, we take the microphone as a measure of naturalness and we compare the optoacoustic devices to the microphone. To determine objectively which emitter is closer to the microphone recordings, we calculated several distance measures [29]. The corpus we used to compare is composed of 32 wing-flapping events totaling approximately 3 minutes of recordings with the silences between events removed. These events were recorded with the laser, the infrared and the microphone.
We applied a normalization that we found to work better than normalizing with the maximum value: Normalization is applied to match the level of each optical modality with the microphone level. After normalization, both signals to be compared possess the same energy (i.e., the laser and microphone pair first, and then the infrared and microphone pair).
We derived the Log spectral distance defined in (2) between two data chunks of 256 samples each, and we averaged over 10382 frames that corresponded to the recorded data.
where P o and P a are the power spectra of the optical and audio data and M is the number of spectral bins.  The Itakura distance between autoregressive parameters is defined in (3), where x is the observed linear prediction coefficient (LPC) [−1 a 1 a 2 . . . a p ] and y is the corresponding LPC for the reference signal (the microphone recording): The Itakura-Saito distance is defined as in (4): Note that the smaller the distance is, the closer is an observed recording to a reference recording, and a perfect match results in 0 distance. All distance measures agree, giving a small benefit to the infrared emitter compared (see Table 1). The small differences between the optical modalities and the microphone are consistent with Fig. 7. Informal listening experiments do not demonstrate a perceptible difference between the laser and the infrared emitter. We use as a result that the laser emitter is comparable in quality to the infra-red emitter if we use the microphone signal as a reference.
One should note that the goal of the optoacoustic devices in the context of our future work on electronic insect traps is not listening naturalness but classification accuracy.
2) A Case Study of Very Small Insects: In Fig. 8, we attempt to study the practical limits of the device in analyzing  the wing-flap of insects. Therefore, we have chosen a Drosophila melanogaster insect. D. melanogaster is a very small insect; the specimen we used was 2.5 mm long, and the major axis of its wings was 2 mm. One can see in Fig. 8 (top) that the signal-to-noise ratio in this case was not as high as in other recordings. However, wing-flap events are resolved even for this very small insect and the recording is clearly audible.
3) Analyzing Insects of Social and Economic Importance: The objective of the spectrograms in Figs. 8-10 is to study the waveforms generated by flight sounds of different insects and see how they develop a frequency signature of the wing-beat so that we can determine whether different insect species can be identified by their opto-acoustical properties. The spectrogram of a potentially dangerous mosquito species is shown in Fig. 9, and an irritated bee is wing-flapping constantly for 37 seconds in Fig. 10. The sounds of insects in general are considered Fig. 11.
In-flight recording from the optoacoustic device. a) The laser emitter and the beam splitter attached to the emitter, b) small aperture gun microphone, and c) array of photodiodes. The cross-lock tweezer in the middle holds insects by their legs.
to be fingerprints of the species and can be used for insect recognition [30], [31]. A. gambiae is a mosquito species in the genus Anopheles and is one of the most important vectors of malaria. Although many insect traps are available in the market, there is no device known to the authors that can recognize the species of the insects. We envision that the sensors described here can be embedded in normal traps and can subsequently count and transmit data regarding the species captured. These counts are invaluable as inputs to mosquitoborne epidemic models that currently receive their input data from manual counting of dispersed traps. In Fig. 9, we show the recording of a male A. gambiae. One notes the high frequency of wing flapping compared to the much larger bees in Fig. 10. One should also note how the different fundamental frequencies of the wing flap, the difference in the shape of the wings and the different beating habits result in a unique frequency signature for each species. Fig. 11 illustrates a different experiment in which insects were enclosed in a plexiglass box containing the powered-on sensor (see Fig. 12 for a close view of the electronics board) to record in-flight wing-flapping (in contrast to a specimen held manually in front of the sensor). In-flight wing-flapping is a procedure in which an insect flies freely as in nature and crosses the beam of the emitter. If we observe the insects in slow motion, the flying insects gradually partially overlap the light beam. As an insect moves totally inside the beam the light variation measured by the sensor starts from the noise level, achieves a peak and then returns to the noise level, producing the quasi-periodic recordings of Figs. 13 and 14.

C. In Flight Recordings
In Fig. 13, 150 ms of flight is recorded for Apis mellifera (a large insect compared to a fruit fly), and in Fig. 14, 50-100 ms of flight is recorded for B. oleae fruit flies. In [24] and [25], it is demonstrated in other datasets  that this duration provides data sufficient for accurate discrimination of species. Moreover, we spent time observing the entrance angles of the arrivals of various insects crossing the emitter-receiver path and their corresponding spectrograms.
A small sample of our observations is gathered in Fig. 14, in which six different freely flying B. oleae insects are recorded. Their spectra and spectrograms provide similar results, with small variations expected from biological organisms. This implies a significant attribute of the sensors: the spectra they provide are invariant to the entrance angle of the flying insect.

D. Discussion
We have constructed an opto-electronic sensor that can accurately analyze any insect's wing flapping. Note that most insects beat their wings with frequencies <1 kHz (see [32] for a large collection of wing-beat frequencies). Clearly, the operating frequency of photodiodes that responded perfectly at 60 kHz is far higher than any biological organism can reach with its wings. Therefore, we conclude that optoelectronic devices cannot fail to respond in tracking the wing-beat of any insect. The constructed sensor is indifferent to illumination variations and operates from bright light to total darkness. It is robust against electronic interference because it modulates emitted light to high frequencies and therefore is immune to interference from devices typically encountered in laboratories; therefore it can be useful for in-lab research and in the field. All experiments presented in this paper were performed in the laboratory under strong electric lamps that had no effect on the recorded spectrogram.
We did not observe any difference in the spectrogram when we rotated the insect specimens held with a short pointed, self-closing cross lock tweezer. At this point, we note that we also used the simpler sensors presented in [26]. These sensors differ mainly in that they do not modulate-demodulate at high frequencies. We report that they failed in the context of these specific artificially illuminated conditions because these sensors registered strong interference at the frequency of the power supply as well as its harmonics, mainly from the electric lights of the laboratory and other electronic appliances. This interference totally masked the very low-level signals of a wing-flap.
We have demonstrated that the recordings are very close to the quality of a microphone, yet they do not face the problems of a microphone that picks up sounds from every direction. We did not find significant differences in quality between an optoacoustic sensor that employs a laser emitter and a simpler infra-red emitter. We have operated our sensors on insects of economic and social importance, such as fruit flies, mosquitoes, and bees, and found that the sensors clearly resolve their spectra by finding the fundamental frequencies and the manner in which wing-flap energy is distributed in the harmonics.
We believe our sensors are an important contribution to the construction of a reference database of insects' wingbeats because these data can now be collected easily anywhere, even in the insects' own habitats, without interference from competing bird and insect species. Reference collections are constructed because they serve as an invaluable tool for educating new scientists and for saving information about a species before it becomes extinct. Currently, there are collections of sounds but are constrained in the use of microphones. This involves a difficult procedure because the recordings are prone to acoustic interference and an anechoic chamber is required for reliable recordings. We propose that optoacoustic devices can replace microphones for the specific task of recording insects that produce sound by wing-flapping.
We project that our robust sensors pave the way for many innovative applications not yet in the market, such as the following: a) Electronic beehives [33] that would count the number of incoming and outgoing bees, guard against invasive species and even assess the health of the entering bee, as the health of an insect has an impact on its flying patterns. b) Mosquito traps that will transmit counts per species to alarm for species that are possible carriers of serious diseases and to provide input to statistical vector models that predict the spread of mosquito-borne diseases based on counts of traps at dispersed locations. c) Electronic traps that are hung from trees and report the density of captured insects and their species composition to initiate spraying procedures at large scales for a number of fruits.

III. CONCLUSION
Precision agriculture relies heavily on sensors to accurately measure parameters of interest from the field. Detection and localization of insects in the field can become a valuable component, as it will allow remote monitoring of targeted insects of economic and social importance. We have constructed a hardware-based board that demonstrates high precision in analyzing insects' wingbeats. It is immune to external light interference and, therefore, is capable of both in-lab and field experimentation. Moreover, we anticipate that once embedded in typical insect traps, the electronic traps will change the manual way insect monitoring is currently performed. The same technique in different configurations can be embedded in traps for different insects, such as mosquitoes (e.g., alerting for species that are possible carriers of the west Nile virus), bees and fruit flies. In the near future, we will report on devices for these separate cases.

ACKNOWLEDGMENT
The authors acknowledge A. Kambouraki from the Department of Biology, University of Crete, Heraklion, Greece, for preparing insect specimens for the recording experiments and Prof. J. Vontas, Head of the Laboratory of Molecular Entomology in the same department, for allowing us access to all insectaries and associated equipment.

APPENDIX
The following is a list of electronic elements and integrated circuits that are used in the production of the board, as described in Figs. 2-5 and seen in Figs. 11-12. They are categorized based on the subunit to which they belong.  II  LIST OF ELECTRONIC ELEMENTS AND INTEGRATED CIRCUITS THAT   ARE USED IN THE PRODUCTION OF THE BOARD  TABLE II  (Continued.) LIST OF ELECTRONIC ELEMENTS AND INTEGRATED   CIRCUITS THAT ARE USED IN THE PRODUCTION OF THE BOARD