All-Optical ReLU as a Photonic Neural Activation Function

We experimentally demonstrate an alloptical rectified linear unit (ReLU) activation function for neuromorphic photonics applications enabled by optical frequency coding of signals. Furthermore, a comparison is made with an electro-optic approach building on a directly modulated laser.


I. INTRODUCTION
Photonic neural networks stand a chance to shift the computation paradigm to an analog optical processing method, driven by the inadequacy of digital electronic computers in fulfilling bandwidth demand and power consumption of an over-scaled data generation [1]. An analog processing with optics requires a set of operations that is equivalent to components in the neural network (NN) architecture, consisting of a linear weighted sum and a nonlinear activation function. By mapping these human brain functions to the optical realm, a multiply-accumulate operation (MAC) can be accomplished at a fast GHz signal rate, owing to the high parallelism and vast bandwidth of photonic technologies. Photonics-based weighting has been earlier demonstrated through ring-based modulators [2], interferometers [3], and SOA-REAM synaptic receptors [4]. Another important subsequent process in a NN is to activate the weighted sum via a nonlinear function such as sigmoid, hyperbolic tangent, or a rectified linear unit (ReLU). However, realizing an activation function in photonics is not straightforward. A hybrid system with all-optical neural network has been introduced, except for the nonlinearity, which has been performed electronically [5]. Several attempts to construct a nonlinear activation function in optical domain have been carried out using a saturable absorber [6], a graphene excitable laser [7] and an electro-absorption modulator (EAM) [8].
In this work, we experimentally demonstrate an implementation of an all-optical ReLU operation on synaptic signals represented in optical frequency coding. It is generated by a chirp-managed directly modulated laser (CML), in combination with an optical interleaver (IL) filter as the activation function. As a benchmark digital neural network (DNN), we use the well-known problem of Iris flower classification. We achieve a lower error with CML+IL and avoid an electro-optic conversion to perform the ReLU.

II. RELU FUNCTION BASED ON FREQUENCY CODING
In our DNN, we use a multilayer perceptron (MLP) scheme with the architecture shown in Fig. 1(a). Each neuron output is determined by the weighted sum Σ of inputs from the prior layer and biased by b, then processed by activation function f. Two ReLU schemes have been investigated, as introduced in Fig. 1 neural inputs modulated in their optical frequency ν, whose weighted sum Σ FM is processed by an all-optical activation function T(ν) contributed by an optical bandpass filter (BPF) with a linear filter slope, and (ii) an electro-optic ReLU constituted by the light-current (L-I) characteristic of a DML with threshold current I th . Through biasing either the wavelength λ of the FM inputs in the first scheme or the bias current of the RF signal that constitutes Σ el in the second scheme, we perform a photonic activation operation ( Fig. 1(c)).

III. EXPERIMENTAL EVALUATION
The all-optical ReLU is evaluated in the context of the Iris flower classification problem, where the MLP This work was supported by the Austrian FFG agency through the JOLLYBEE project (grant n° FO999887467). network has been trained offline. Fig. 2(a) shows the experimental setup to evaluate the performance of optical ReLU activation function. In our work, we send the weighted sum data Σ(t) (without being processed by the activation function f ) of the node 0 at the 2 nd layer of the DNN implementation ( Fig. 1(a); red) through an arbitrary waveform generator (AWG) to the optically implemented activation function, using a data rate of 1 Gb/s. The AWG drives a CML at 1600.6 nm whose internal chirp filter has been tuned to a neutral position to avoid frequency-to-intensity conversion. The CML therefore generates the frequency modulated weighted sum Σ FM (t). The inset in Fig. 2(a) shows the output of the CML with its negligible intensity extinction ratio of 0.3 dB. The weighted sum is subsequently processed by an optical 25/50 GHz IL. Fig. 2(b) shows the CML spectra before (red) and after the IL (green) when the FM signal is aligned to the slope of the IL, together with the transfer function of the IL (blue). The emitted signal is then detected by PIN receiver and digitized through a real-time oscilloscope to feed the optically performed f (Σ el (t)) back to the DNN. For comparison, we also use a DML at 1554 nm with I th = 6.5 mA. We use two different current biases: a sub-threshold bias of 6.2 mA, leading to rectification of the RF signal driving the laser, and a bias of 11.4 mA > I th . Deriving from the three classes of Iris flowers (setosa, virginica, versicolor), the output of node 0 at the 2 nd layer after ReLU stands in good agreement with the alldigital NN output (○). This is also evidenced by the very similar eye diagrams (i, ii) at the corresponding output. On the contrary in Fig. 2(d), a ReLU operation with the DML (▲) introduces large errors, especially towards performing the required rectification. This appears as surprising at first, considering the ideal L-I function of the DML; however, this large error is explained by the introduction of gain switching artifacts introduced when a sub-threshold bias of 5.5 mA is applied as required to perform rectification, as it is seen in the respective eye diagram (iii). Mitigating these artifacts through a suboptimal bias current of 11.2 mA towards to the linear L-I regime results in the large error, as it can be noticed from the unrectified eye diagram (iv). This renders the DML as unsuitable. This result is further emphasized in the error histogram presented in Fig. 2(e), where the error ε, is defined as the absolute difference between the normalized optical output and all-digital NN. For optical frequency coding (blue), this error is mostly populated within a range of ε < 0.15. The DML-based ReLU (red) yields an error range 0 < ε < 0.7.

IV. RESULTS AND DISCUSSION
We further calculated the accuracy of our Iris flower classification by propagating forwardly the optical measurement data from the node 0 of the 2 nd layer to the output layer. We achieve an accuracy of 90% and 62% when using the CML+IL and the DML as optical ReLU, respectively. This stands in comparison with a 93% accuracy of an all-digital NN and confirms that the optical ReLU function introduces a small 3% error for the application while operating at a 1 Gb/s input rate.