Neural Network with Optical Frequency-Coded ReLU

We demonstrate a photonic rectified linear unit (ReLU) function accomplished through frequency-coded neural signals. We show operation of an optical neuron with weighted sum and ReLU activation to perform with a 1% penalty in accuracy.


Introduction
Brain-inspired computing has become a focus of interest as a new paradigm in information processing.It is triggered by the bottleneck of the von-Neumann computing architecture in latency and bandwidth-distance trade-off [1].Photonics is considered as a foreground to implement the parallelism of neural networks (NN) while accomplishing high speed at low power.This positions neuromorphic photonics as a next milestone to beat the barrier inherent to digital electronic computing.The corresponding NN model includes two main building blocks: a weighting scheme based on a multiply-accumulate operation, followed by an activation function.The need of optical synaptic receptor that mimics the linear operation in NNs has been addressed via Mach-Zehnder interferometers (MZI) [2], broadcastand-weight network using micro-ring resonators (MRR) [3] and frequency coded synapses in combination with colorless demodulation and detection [4].The linearity of optics renders the practical realization of an analogue weighted sum as a feasible task.On the contrary, it is not trivial to realize a nonlinear activation function in the optical domain.Rectified linear unit (ReLU), sigmoid, hyperbolic tangent and exponential linear unit are examples of activation functions; each of them having advantages for distinct applications.Some of these functions have been accomplished optically via SOA-MZI [5] and EAM [6].
In this work, we experimentally demonstrate photonic ReLU activation via frequency coding of synaptic signals, as proposed conceptually recently [7], using a combination of chirped directly modulated laser (CML) and an optical interleaver (IL).We show that the inclusion of the proposed ReLU function in an optical neuron allows to set the polarity of its synaptic weights, thus contributing to simplified optical neural network (ONN) implementations.Our experiment proves that even without adaptation of the NN training to the specifics of the optical layer, a small penalty of only 1-3% applies in terms of NN accuracy when integrating the proposed ReLU and the entire optical neuron.This relates to an all-digital NN (DNN) performing Iris flower classification at an inherent accuracy of 93%.

Photonic ReLU through Synaptic FM Coding
As a first step we investigate the implementation of a photonic ReLU function for two schemes (Fig. 1b/c): (i) an IL as optical frequency demodulator with a linear transmission slope, which, together with the natural noise floor n of the receiver, resembles the ReLU function, and (ii) a DML that offers its linear light-current (L-I) characteristic as electro-optic ReLU.The lower part (Ψ) of Fig. 1d shows the experimental setup dedicated to the evaluation of these photonic ReLU functions.The accuracy of the stand-alone ReLU embedded in the DNN has been assessed in the context of the Iris flower classification challenge, for which a set of 150 flowers is to be attributed to three classes of   z i ) is applied at the output layer to generate the prediction probabilities.The MLP network has been trained offline and the "ideal" DNN will be our benchmark to evaluate the photonic ReLU and the optical NN performance.
For this purpose, we send the weighted sum data Σw i s i (t) from the node 0, layer 2 of the DNN (without being processed by activation function) to the optical activation unit using an arbitrary waveform generator (AWG) at 1 Gb/s rate.In the proposed ReLU implementation, the AWG drives a 10-GHz butterfly CML as a synaptic emitter operating at 1600.6 nm.The integrated chirp-conversion filter of the CML has been tuned to a neutral spectral position, resulting in a purely frequency modulated (FM) weighted sum signal Σ FM (t) with an intensity extinction ratio of only 0.3 dB (Λ).This FM signal is subsequently rectified by the activation function via an optical 25/50 GHz IL. Figure 2a depicts the optical spectra of CML before (α) and after tuning the wavelength of CML with respect to the IL slope (β), together with the transmission of the even IL channel (γ).The output of the IL-based ReLU, which has been demodulated from a frequency-to an intensity-coded signal, is then received by a PIN receiver and digitized by a real-time oscilloscope.For performance comparison, we replace the combination of CML+IL with a 10-GHz butterfly DML operating at 1554 nm and having a threshold current I th of 6.5 mA.
Figure 2c describes the normalized optical signal after ReLU function with CML+IL (•) for all 150 Iris data samples, which stand in good agreement with the all-digital NN (○).This is confirmed by the comparable eye diagrams after the ReLU (i,ii).Integrating the photonic ReLU function in the NN yields an accuracy of 90%, which is 3% lower than that of the all-digital NN.This proves the correct operation of the proposed neural activation unit.
In contrast, ReLU operation with the DML (▲) introduces a larger error (Fig. 2d) when applying a sufficiently low bias current to rectify the RF signal.The source of this error can be understood from the eye diagram (iii) where gain switching is introduced when the sub-threshold bias of 5.5 mA < I th is applied (Fig. 2b, δ).However, increasing the bias to 11.2 mA > I th to suppress this effect (Fig. 2b, ζ) operates the ReLU in the linear L-I region and does not lead to the required rectification (iv).These results are further summarized in an error histogram (Fig. 2e), where the error ɛ is defined as an absolute difference between normalized optical signal and the all-digital NN.The error with CML+IL (κ) is clearly lower (average error ε̅ = 0.14, 3σ = 0.36) compared to DML (ς).Therefore, we will rely on the CML+IL based ReLU scheme for the further investigation of a functionally-complete optical neuron.

Optical Neuron with Weighting and Activation
In a second step, the upper experimental setup (Ω) in Fig. 1d implements an optical neuron with four synaptic inputs, realizing an optically weighted sum and the ReLU activation.We used 4 CMLs of which two operate in the C-band (1535 nm) and two in the L-band (1600 nm).All of the CMLs share the same IL to ensure a cost-optimized layout of multiple neurons within a NN layer.In this experiment, the C-band CMLs represent the emitters of two nodes in layer 2 of the NN (A and B in Fig. 1a) and the respective PIN receiver in the C-band yields the output of the neuron of node 2, layer 3 (E).In the same manner, the L-band CMLs are used for the neuron at node 0, layer 3 (D).
We send the neural data s i (t) of the corresponding neurons in layer 2 at 1 Gb/s to the CMLs.Based on the NN training, the two neurons (A and B) have alternating signs (w + ,w -).The FM signals are aligned to the respective positive/negative IL slopes, generating sgn(w i ) accordingly.The required synaptic interconnect towards the even (+) The spectral processing introduced by the ReLU on a C-band FM signal, generated through a CML with a chirp parameter of 2.9, is shown in Figs.3b/c for negative/positive polarity.The RF spectra are reported for heterodyning the IL inputs (σ, χ) and outputs (ρ, φ) to an intermediate frequency of ~6.6 GHz, as indicated through the CW beat notes (π, τ) of CML and local oscillator.In Fig. 3b, the higher frequencies of the upper FM sideband (ρ) are suppressed by the falling slope of the IL, introducing the desired rectification effect.Similarly, higher frequencies of the lower FM sideband (φ) are suppressed by the rising IL slope for the sign-inverted ReLU in Fig. 3c.
Figure 3d reports the error histogram when incorporating the functionally-complete optical neuron in the NN while sharing the ReLU with a second neuron.Both, node 0 (μ, L-band) and node 2 (υ, C-band) in layer 3, have an average error ε̅ as low as 0.1.We further compute the accuracy for Iris flower classification by propagating forwardly the optical measurement data in the NN. Figure 3e reports the NN accuracy as a function of ε̅ for the experimental data and under artificial signal degradation due to AWGN.We achieve an accuracy of 92% in the experiment for both neurons at 1 Gb/s.This result stands close to the 93% based on a fully-digital NN.As expected, the accuracy reduces when ε̅ increases.Particularly, the accuracy drops faster for node 2 (▲), while node 0 (•) shows just a slight degradation.This is due to the peculiar noise tolerance of the particular neurons.Finally, Fig. 3f shows how ε̅ elevates for higher data rates of up to 5 Gb/s.Results are shown for the noise-sensitive node 2 at layer 3. We see a slight increase in ε̅ from 0.10 to 0.12, which relates to a 80% NN accuracy at 5 Gb/s.

Conclusion
We have demonstrated an optical frequency-coded ReLU function in combination with a chirped laser and an optical interleaver.We have demonstrated operation at a marginal error when integrating the ReLU activation in a hybrid digital / optical NN implementation, amounting to a small 1% penalty in accuracy.We have further proven that the choice of sign for the synaptic weighting process can be accommodated with this ReLU function.

Fig. 1 .
Fig. 1.(a) MLP-based neural network architecture.(b) ReLU activation units performed by (i) an optical interleaver and (ii) a DML with (c) their corresponding activation functions.(d) Experimental setup to evaluate the photonic ReLU (Ψ) and a neuron of an optical neural network (Ω).
paper, subject to publisher revision Iris flowers (setosa, versicolor and virginica).Each sample of flower has four features, comprising of sepal (length and width) and petal (length and width).In our DNN, we use the multilayer perceptron (MLP) architecture shown in Fig.1a.The input layer has four nodes, each (x i ) represents one feature of the flower.The value of nodes in the following layers are determined from the weighted sum of the prior layer (Σ) and biased by b, followed by an activation function (ƒ).Finally, a Softmax function (