1. Introduction

Machine learning methods are extensively used in an increasing number of fields, e.g., automotive industry, medical science, internet security, air-traffic control etc. This field conveys many algorithms and structures ranging from simple linear regression to almost arbitrarily complex neural networks which are able to find solutions to highly complex problems. Recently, a considerable attention was drawn to the overlap between quantum physics and machine learning [1]. Depending on the type of input data and data processing algorithms, we can distinguish four types of quantum machine learning (QML), i.e., CC (classical data and classical data processing – classical limit of quantum machine learning), QC (quantum data and classical data processing), CQ (classical data and quantum data processing), and QQ (quantum data and quantum data processing).

QML offers reduced computational complexity with respect to its classical counterpart in solving some classes of problems [1]. Depending on the problem at hand the speedup can be associated with various features of quantum physics. A number of proposals and experiments focused on QML have been reported, such works include quantum support vector machines [2], Boltzmann machines [3], quantum autoencoders [4], kernel methods [5], and quantum reinforcement learning [6,7]. In reinforcement learning a learning agent receives feedback in order to learn an optimal strategy for handling a nontrivial task. Next, the performance of the agent is tested on cases that were not included in the training. If the agent performs well in these cases, learning is completed. The difference between machine learning and mere optimization is often very subtle. For instance in a recent Letter [3], Gao et al. have trained a neural network to classify quantum states according to their capability to violate the CHSH inequality. In that case, a classical computer learned a set of five real-valued numbers used as weight factors in the sum of correlation coefficients obtained in an already known measurement configuration. In contrast to that, our protocol relies on a feed-back between a classical control and a quantum gate rendering it genuinely quantum-classical machine learning.

Here we demonstrate experimentally that reinforcement learning can be used to train an optical quantum gate (see Fig. 1). This problem is related to the boson sampling [9–13], where one knows the form of the scattering matrix of a system and computes modulus squared of its permanent. However, here we optimize the probabilities of obtaining certain outputs of the gate by finding the optimal parameters of the scattering matrix. Calculating the probabilities (moduli squared of permanents of scattering matrix) is in general a computationally hard task while measuring them is much faster. This feature of quantum optics allows us to expect that complex integrated interferometers could be applied as special-purpose quantum computers (e.g. [14]). This sets our problem in the class of CQ quantum machine learning tasks. There are other QML approaches to optimizing quantum circuits. One approach uses classical machine learning to optimize the design of a quantum experiment in order to produce desired states [15]. Another QML approach consists of optimizing quantum circuits to improve the solution to some problems solved on a quantum computer [16]. The latter method can be applied even to minor computational tasks to save resources [17,18].

 

Fig. 1. Conceptual scheme of hybrid reinforcement learning of a quantum gate driven by a classical control. The transformation of the quantum register performed by the gate is evaluated by measurement providing a reward to the classical control that iteratively modifies the gate’s parameters. The core of this procedure can be viewed as boson sampling, classical simulation of which is known to be computationally #P-hard in $N$ [8].

Download Full Size | PDF

We applied online reinforcement learning methods to train an optimal quantum cloner. Quantum cloning is indispensable for safety tests of quantum cryptography systems or of other quantum communications protocols. Perfect quantum cloning of an unknown state is prohibited by the no-cloning theorem [19]. However, it is possible to prepare imperfect clones that resemble the original state to a certain degree. Usually the approach towards quantum cloning involves direct optimization of the experimentally implemented interaction between the system in the cloned state and another systems to maximize the fidelity of the output clones [20]. In contrast to that, we present a quantum gate (learning agent) that is capable to self-learn such interaction (policy) based on provided feedback (implicit setting of the parameters). For the purposes of this proof-of-principle experiment, we limit ourselves to equatorial qubits in the form of

2. Experimental realization

We constructed a device composed of a linear optical quantum gate and a computer performing classical information processing. While the gate itself is capable of a broad range of two qubit transformations, this paper focuses on its ability to act as a phase-covariant quantum cloner. Its figure of merit is the individual fidelity of the output copies. The fidelity of the $j$-th clone $F_{j} ={}_{\textrm {in}}\langle \psi |\hat {\varrho }_{j}|\psi \rangle _{\textrm {in}}$ is defined as overlap between the state of the input qubit $|\psi \rangle _{\textrm {in}}$ and the state of the clone $\hat {\varrho }_{j}$. In case of the state in Eq. (1), the maximum achievable fidelity of symmetric $1 \rightarrow 2$ cloning accounts for $F_{1} = F_{2} = \frac {1}{2} \left (1+\frac {1}{\sqrt {2}}\right ) \approx 0.8535$ [27,28].

The experimental setup is depicted in Fig. 2. Pairs of photons are generated in Type I spontaneous parametric down-conversion occurring in a nonlinear BBO crystal. This crystal is pumped by Coherent Paladin Nd-YAG laser with integrated third harmonic generation of wavelength at $\lambda = 355$ nm. The generated pairs of photons are both horizontally polarized and highly correlated in time.

 

Fig. 2. Experimental setup. Legend: PBS – polarization beam splitter, PC – polarization controller, BBO – beta barium borate, Det – detector, HWP – half-wave plate, QWP – quarter wave-plate, PS – piezoelectric stage, TAC&SCA – time-to-amplitude converter & single channel analyzer.

Download Full Size | PDF

These photons are then spectrally filtered by 10 nm wide interference filters and spatially filtered by two single mode optical fibers each guiding one photon from the pair. In our experimental setup, qubits are encoded into polarization states of the individual photons ($|0\rangle \Leftrightarrow |H\rangle$ and $|1\rangle \Leftrightarrow |V\rangle$). The photon in the upper path (spatial mode 2) represents the signal qubit, quantum state of which we want to clone, and the photon in the lower path (spatial mode 1) serves as the ancilla the state of which reads

Using polarization controllers (PC) we can ensure that both photons are horizontally polarized at the output of the fibers. Next, polarization states of the photons are set using a combination of half-wave plates (HWPs) and quarter-wave plates (QWPs). There are two stationary QWPs fixed at angle 45° and six motorized HWPs which make it possible to control the whole quantum gate using a computer. The first two half-wave plates HWP$_1$ and HWP$_2$ are used to set polarisation states of the ancilla and cloned photons, respectively.

The core part of the presented quantum gate is a Mach-Zehnder-type interferometer which consists of two polarizing beam splitters (PBS) and two reflective pentaprisms, one of which is attached to the piezoelectric stage (PS). With the addition of two HWPs (HWP$_3$ and HWP$_4$) placed in its arms, this whole interferometer implements a polarization dependent beam splitter with variable splitting ratio. Mathematically, the scattering matrix $\hat {U}$ of the gate transforms the bosonic modes

The two spatial modes at the output of the interferometer are subjected to polarization projection (QWP$_5$, QWP$_6$ and HWP$_7$, HWP$_8$) and then led to a pair of avalanche photodiodes by Perkin-Elmer running in Geiger mode. We use detection electronics to register both single photons at each of the detectors and coincident detections as successful operation of the gate is indicated by the presence of single photon in each output of the interferometer. The electronic signal is then sent to a classical computer.

For specific parameters of the presented linear-optical elements, this quantum gate functions as a $1 \rightarrow 2$ symmetric phase-covariant cloner, optimal analytical cloning transformation of which is well known [27]. On a linear-optical platform, this optimal cloning transformation can be achieved by a polarization dependent beam splitter with intensity transmissivities for horizontal and vertical polarization at $t_{\textrm {H}} \approx 0.21$ and $t_{\textrm {V}} \approx 0.79$, while setting the ancilla to be horizontally polarized. Note that our quantum gate is capable of implementing this transformation when set approximately to $\phi = 31.3, \theta = 13.7$. To showcase the capability of our gate to learn to clone phase-covariant states optimally, we deliberately ignore this analytical solution and employ self-optimization procedure seeking to maximize the cloning fidelities. The optimization process consists of a number of measurements (runs), each performed for a set of variable optimization parameters $\phi , \theta , \omega$. That is, variable splitting ratio for horizontal ($\phi$) and vertical ($\theta$) polarization as well as the state of the ancilla ($\omega$) [Eq. (2)]. In each run, output clones fidelities are evaluated and supplied to the classical Nelder-Mead algorithm [29] for a decision about the parameters of the future runs.

In between any two runs, the setup is stabilized. We first minimize temporal delay between the two individual photons. In this case, all HWPs are set to 0° with the exception of HWP$_4$ being at 22.5°. In this regime, we minimize the number of two-photon coincident detections (Hong-Ou-Mandel dip) by changing the temporal delay between the photons using a motorized translation stage MT. In the next step, the phase is stabilized in the interferometer. Moreover, we make use of the fact that the phase shift in the interferometer additively contributes to the phase $\eta$ of the signal state [Eq. (1)]. This allows us to use interferometer phase stabilization for setting of any signal state of the equatorial class. We achieve this task by setting HWP$_2$ to 22.5° and HWP$_8$ to the value corresponding to orthogonal state with respect to the required input signal state. All other HWPs are set to 0° and a minimum in single-photon detections on Det 2 is found by tuning the voltage applied to PS. Note that the entire stabilization procedure is completely independent of the learning process itself.

In this reinforced-learning scenario, the cloner is trained on a sequence of random equatorial signal states [Eq. (1)] different for each run. The phase $\eta$ is randomly picked from interval $\left ( 0; 2 \pi \right )$. The fidelities are obtained by measuring coincidence detections in four different projection settings. We label these coincident detections $cc_{ij}$, where $i,j \in \left \lbrace \parallel ; \perp \right \rbrace$. The $\parallel$ and the $\perp$ sign denote projection on the signal state $\vert {\psi _{\textrm {s}}}\rangle$ and its orthogonal counterpart $\vert {\psi _{\textrm {s}}^\perp }\rangle$. We calculate the fidelities as $F_1 = {\left (cc_{\parallel \parallel } + cc_{\parallel \perp }\right )}/{\Sigma }$ and $F_2 = {\left (cc_{\parallel \parallel } + cc_{\perp \parallel }\right )}/{\Sigma }$, where $\Sigma$ denotes $cc_{\parallel \parallel } + cc_{\parallel \perp } + cc_{\perp \parallel } + cc_{\perp \perp }$.

To see the connection between boson sampling (i.e., sampling probabilities corresponding to a permanent of scattering matrix) and the optimal quantum cloning let us consider scattering matrix $\hat U$ [see Eq. (3)] defined in Fock space of both spatial and polarization modes. This matrix, when optimized, describes the desired optimal quantum operation. Let us also consider a unitary scattering matrix $\hat {S},$ which transforms a single $H$–polarized photon into the sampled state $| \psi _s\rangle =\hat {S}^\dagger |1_{H,1},0_{V,1}\rangle =\alpha |1_{H,1},0_{V,1}\rangle +\beta |0_{H,1},1_{V,1}\rangle,$ \textrm{where} $|\alpha|^2 + |\beta|^2= 1.$ Global cloning fidelity $F_G$ for $1\to 2$ phase-covariant cloning can be expressed in terms of the measured detection rates as

Note that in the experiment, we do not measure the permanent itself but rather its modulus squared. Further to that, we only consider cases when 0 or 1 photon is present in an output mode. Thus, we avoid the problem of exponentially small detection probabilities of higher photon-number states [see Eq. (4)]. As a result, our measurement scales better that classical methods of calculating permanents.

3. Results

We demonstrate reinforcement-learned quantum cloner for a class of phase-covariant quantum states. The gate operates formally as a polarization dependent beam splitter with tunable splitting ratios. This tunability provides two parameters for self-learning, $\phi$ and $\theta$. The third learnable parameter $\omega$ is embedded in the state of the ancillary photon [Eq. (2)]. We have experimentally implemented two machine learning models using two and three parameters, respectively. In the first model, we fixed the ancilla state to its theoretically known optimum $\vert {\psi _{\textrm {a}}}\rangle = \vert {H}\rangle$. The remaining two parameters, $\phi$ and $\theta$, were machine learned. To minimize the cost function (i.e. optimize the performance of the cloner) we applied Nelder-Mead simplex algorithm which iteratively searches for a minimum of a cost function. We chose the cost function to be in the form of $C = (1-F_{1})^2+(1-F_{2})^2+(F_{1}-F_{2})^2$, where $F_{1}$ and $F_{2}$ stand for the fidelity of the first and second clone, respectively. This choice reflects the natural requirements to obtain maximum fidelities of both the clones as well as to force the cloner into a symmetrical cloning regime. Training of the gate consists of providing it with training instances of equatorial qubit states (randomly generated in each cost function evaluation, i. e. an online machine learning scenario) and with the respective fidelity of the clones. In each training run, the underlying Nelder-Mead algorithm sets the gate parameters to vertices of simplexes in the parameter space and then decides on a future action. In the case of a two-parameter optimization, these simplexes correspond to triangles as depicted in Fig. 3. In this figure, we plot the exact path taken by the Nelder-Mead simplex algorithm to minimize the cost function $C$ for the case of a real experiment and its simulation. The selected initial simplex was intentionally chosen well away from the optimal position – its first vertex resembles the trivial cloning strategy [26,33]. In Fig. 4(a), we illustrate the evolution of both the fidelities $F_{1}$ and $F_{2}$ during the training. After 40 runs (i.e. 40 instances from the training set), this model was deemed trained because the size of simplexes dropped to the experimental uncertainty level (i.e. $\sim 0.1$ degrees on rotation angles of wave plates). However, in general, setting the simplex to converge within a given precision is a nontrivial problem [34].

 

Fig. 3. Plot of the cost function $C$ for the angles $\phi$ and $\theta$ (corresponding to ${\textrm {HWP}}_{4}$ and ${\textrm {HWP}}_{3}$ in Fig. 2, respectively). The solid yellow lines denote the final triangles reached by the Nelder–Mead simplex minimization [29] algorithm in each of its iteration. The dashed lines mark intermediate steps. The circled numbers stand for gate runs and point F depicts the final state of the gate at the end of training. A simulation was performed prior to the experiment to verify our implementation of the learning algorithm. Note that the starting simplex is chosen in advance and is identical in both cases. Subsequent paths in case of the simulation and actual experiment differ due to setup imperfections.

Download Full Size | PDF

 

Fig. 4. Plots showing the evolution of fidelity of both the clones (top) as well as the cost function (bottom) throughout the training in case of (a) the first model with two free parameters $\phi$ and $\theta$ and (b) the second model with three free parameters $\phi$, $\theta$ and $\omega$. Fidelity of the first clone $F_{1}$ is visualized by a solid red line and the fidelity of the second clone $F_{2}$ is shown in blue (dashed line). The thick solid black line stands for the theoretical limit of $\approx 0.8535$. This theoretical limit bounds the value of fidelity averaged over both clones and over all equatorial states. It is legitimate and expected for $F_1$ to be close to 1 as the cloner starts in a highly asymmetric regime. Moreover, slight deviation from perfect sampling of equatorial input states can cause the average fidelity to surpass the theoretical limit by a few percent.

Download Full Size | PDF

In the second model, we let the gate learn the optimal setting of the ancilla $\omega$ along with the gate parameters $\phi$ and $\theta$. The training procedure ran similarly to the first model. The initial value of the $\omega$ parameter was set naively to $\omega = \tfrac {\pi }{8}$ so it lied on the equator of the Poincaré-Bloch sphere. We present evolution of the intermediate fidelities of this three-parameter model in Fig. 4(b). Using a similar stopping criterion as in the first model, the training of the second model was terminated after 60 runs.

We have tested the performance of both our models on independent random test sets each populated by 40 instances of equatorial states. We summarize the results of the two models in Table 1, where we provide the final learned parameters together with the mean values of the fidelities on the test sets $\langle F_{1} \rangle$ and $\langle F_{2} \rangle$. The observed fidelities on test sets are bordering on the theoretical limit (at most 0.013 below it) which renders our gate highly precise in context of previously implemented cloners [27,35–39].

Table 1. Summary of the Final Values for Both Models. ${\langle F_{1} \rangle }$ and ${\langle F_{2} \rangle }$ Denote the Mean Fidelities Observed on the Test Sets.

View Table

4. Conclusions

In our proof of principle experiment, we implemented a CQ reinforcement quantum machine learning algorithm driven by a hybrid of classical Nelder–Mead method and quantum computing performed as a measurement of modulus squared of a permanent. This approach was used to train a practical quantum gate (i.e., a quantum cloner). The task of the training was to optimize parameters of the gate (interferometer) $\phi$, $\theta$ and $\omega$ (setting of the ancilla in the second experiment) to perform phase-covariant cloning. The quality of both the clones measured by their fidelities $F_{1}$ and $F_{2}$ which were evaluated within both experiments (Fig. 4) successfully reached the theoretical limit for phase-covariant cloning $0.854$. Remarkably, the cloner managed to achieve almost optimal cloning by learning setup parameters, slightly different from analytical values, that counter all experimental imperfections including imperfections in the cloner itself and in the input state preparation.

To see the connection between boson sampling and our results, let us focus on computing the modulus squared of the permanent perm$[\hat {U}]$ of scattering matrix describing the gate operation. The unitary scattering matrix $\hat {U}$ performs linear transformation on the annihilation operators $\hat a_i$ of the input modes ($i$ can be an index labeling both the polarization and the spatial degrees of freedom). Then the input-output relation of an quantum-optical interferometer is given as $\hat b_j = \sum \hat U^\dagger _{i,j} \hat a_i.$ If all the input modes of an interferometer are injected with single photons and single photons are detected at specific outputs (no bunching) the probability of obtaining the desired detection coincidence is $p = |\mathrm {perm}[\hat U]|^2.$ However, this expression becomes more complex if some modes are occupied by more than one photon. Then factorials of mode-specific photon numbers appear as denominator and the respective rows/columns of $U$ must be repeated a corresponding number of times [9]. If some output modes are not to be populated, the respective row of $\hat U$ matrix is deleted. Calculating the moduli squared of permanents of the scattering matrix associated with our cloner by hand is already challenging (we have polarization and spatial degrees of freedom for two photons) and in general it falls into the $\#P$-hard complexity class. The quantum information processing performed by the setup is equivalent to boson sampling which, in complex systems, is predicted to manifest quantum supremacy over classical simulation of linear-optical setups. This makes our research a relevant application of so-called quantum circuit learning described in [40], Mitarai et al. The integrated-optics platform [13] or superconducting qubits [41] seem to be promising platforms for large-scale quantum machine learning of this type.

Our results also opens possibilities of further research or applications in the field of quantum key distribution. Suppose a typical attack on the key distribution scheme: Bob and Alice share quantum states and the attacker Eve is eavesdropping on them. Bob and Alice exchange quantum states and, via a classical line, they can decide to stop exchanging qubits (because of noise). Let us assume that Eve is eavesdropping on both quantum and classical communication. Eve can in principle use reinforcement learning to train a cloner to perform the attack by feeding it with information on the behavior of Bob and Alice, e.g., their decision on continuing or aborting the exchange of a quantum key and/or their decision on parameters of privacy amplification. For such application the proposed gate would have to be modified since Eve does not know the specific class of states used by Bob and Alice, but that is out of the scope of this paper.