## Abstract

Nonlinear impairments in optical communication system have become a major concern of optical engineers. In this paper, we demonstrate that utilizing a nonlinear filter based Decision Feedback Equalizer (DFE) with error detection capability can deliver a better performance compared with the conventional linear filter based DFE. The proposed algorithms are tested in simulation using a coherent 100 Gb/sec 16-QAM optical communication system in a legacy optical network setting.

© 2014 Optical Society of America

## 1. Introduction

Nonlinear distortion in the optical communication system attracts significant attentions in recent years [1–8]. The decision feedback equalizer (DFE) is widely used in optical communication systems to compensate for signal distortions and has become a standard optical communication component [9–11]. Due to the nature of decision device (also known as the “slicer”) of DFE, DFE can be considered as a nonlinear device although most of the DFE structures are based on linear filter equalizer. Besides the linear filter based DFE, the DFE based on nonlinear filters also attracts significant attentions in recent years [3–7]. Volterra filter is a popular nonlinear signal processing tool and is often used for nonlinear distortion compensation [8, 12]. The Volterra filter based DFE such as a DFE with a Volterra filter in the forward path and a 2nd-order polynomial filter in feedback path [4] and frequency-domain Volterra filters [5–7] has been investigated for optical communications. In this article, we present a more general DFE architecture consisting of time-domain Volterra filter in either forward path, feedback path, or both. The proposed DFE is also equipped with error detection capability to reduce the impact of error propagation. The decision error made by DFE’s slicer can significantly degrade DFE performances because DFE coefficients are updated based on the difference between slicer and equalizer outputs. This phenomenon is referred to as “error propagation” [13, 14]. In this study, we demonstrate that, by integrating error detection capability with the DFE, the negative impact of error propagation can be significantly reduced. Since the error correction code has been widely applied in optical communications [15, 16], it only requires moderate efforts to add an error detection capability to the DFE.

The rest of the paper is organized as follows. The Volterra filter based DFE will be introduced in the next section. A legacy network based coherent optical communication system simulation program used to investigate proposed DFE schemes is described in Section 3. Due to dispersion compensation fiber (DCF) used in a legacy network, its nonlinear compensation is a more difficult task compared with a greenfield network which relies on a digital filter at the receiver to compensate for dispersion. The fourth section summarizes simulation results and corresponding analysis and the fifth section concludes this article.

## 2. Volterra filter and DFE

The Volterra model is a popular nonlinear signal processing tool and is often used for nonlinear distortion compensation in communication systems [4–8]. A Volterra model can be considered as a Taylor series with memory. A general n^{th}-order discrete Volterra filter input-output relation is given in Eq. (1) [8, 17].

*x[n]*is filter input,

*y[n]*is filter output,

*M*is memory length, and

*h*is the r

_{r}[ ]^{th}-order Volterra kernel. In this project, we focus on a band-limited application and a 3rd-order band-limited Volterra model is utilized. The input-output relation of a 3rd-order band-limited Volterra model is provided in Eq. (2) [8, 18].

*x[n]*and

*y[n]*is the complex-valued filter input and output,

*M*is memory length,

*h*is the 1st-order Volterra kernel,

_{1}[ ]*h*is the 3rd-order kernel, and

_{3}[ ]*M*is the memory length.

A DFE consisting of a forward-path equalizer and feedback-path equalizer and a slicer is illustrated in Fig. 1. Its working principle can be explained as following. The slicer determines the most likely transmitted symbol based on the equalized symbol. The equalizer coefficients are then updated based on difference between equalizer output and slicer output. The update algorithm employed in this project is Recursive Least Square algorithm (RLS) [19]. The conventional DFE is based on linear filters (i.e. both equalizers in forward and feedback paths are linear filters). Since the slicer is a nonlinear device, a DFE can eliminate nonlinear impairments in optical fibers to some extent. However, the conventional linear filter based DFE, can only achieve limited success in compensating for nonlinear distortions [20]. In this paper, we expand the conventional linear filter based DFE to nonlinear filter based DFE and explore different combinations of linear and Volterra filters in the forward and feedback paths of DFE. Although studies on Volterra filter based DFE have been conducted, to the best of authors’ knowledge, this paper is the first investigation of different Volterra filter based DFE configurations for optical communications.

Because the equalizer coefficients are updated based on the difference between slicer output and equalizer output, a decision error made by the slicer can significantly deteriorate the DFE performance. Our simulation results show that error detection capabilities can help DFE improve its performance against the slicer’s decision error. Such a study is also the first attempt to address the Volterra filter based DFE’s error propagation issue for optical communication applications.

## 3. Simulation system setup

A coherent optical communication system simulation program built with the commercial optical communication simulation software, OptiSystem, is used to investigate proposed DFE architectures. The system configuration is illustrated in Fig. 2. The transmission data rate is 100 Gb/sec and the modulation scheme is 16-QAM. To make the compensation task more challenging, we tested the DFE in a legacy network configuration which uses DCF to compensate for chromatic dispersion. Compared with greenfield networks which do not use DCF, such a testing setting is more difficult due to additional nonlinearity of DCF. The Single Mode Fiber (SMF) is used for transmission. The transmission distance is 480km or 6 of 80-km loops. In each loop, the chromatic dispersion of SMF is fully compensated by the DCF and the attenuation of DCF and SMF is compensated by an optical amplifier. The SMF and DCF’s chromatic dispersion and nonlinearity parameters are listed in Table 1.

The carrier wavelength is set at 1550 nm and we only considered single channel transmission in this study. One set of transmission data has 65536 bits (16384 QAM symbols) and 5 sets of testing data are used for each laser power setting. The laser power is changed to evaluate the system performances under different signal power. Since we would like to investigate the effectiveness of error detection capacity in a DFE, the even-parity check is applied to encode transmitted data. As the result, only 8 out of 16-QAM symbols are actually used in transmission and the transmitted signal constellation is illustrated in Fig. 3(a). We set optical signal power to 0.75dBm and resulting un-compensated signal constellation is illustrated in Fig. 3(b). As shown in Fig. 3(b), the signal constellation is rotated and corrupted. The transmitted and received data is saved to test different DFE structures simulated with Matlab^{®}.

## 4. Results and discussions

In this study, we considered 4 different DFE structures: (1) linear FIR filter in both forward and feedback paths (referred to as *FF* in later discussions), (2) 3rd order Volterra filter in both forward and feedback paths (referred to as *VV* in later discussions), (3) linear FIR filter in forward path and 3rd-order Volterra filter in feedback path (referred to as *FV* in later discussions), and (4) 3rd-order Volterra filter in forward path and linear FIR filter in feedback path (referred to as *VF* in later discussions). The input-output relation of the 3rd-order Volterra filter is given in Eq. (2) and the input-output of the linear filter is given in Eq. (3) below.

*x[n]*and

*y[n]*is filter input and output,

*M*is memory length,

*h*is the linear filter coefficient.

_{1}[ ]As demonstrated in one co-author’s previous research work, the filter with memory length of 3 should be sufficient to compensate for nonlinear distortions in optical communication systems [8]; therefore, in this study, linear FIR filter has 4 coefficients and 3rd-order Volterra filter has 44 coefficients. The initial coefficients for forward path are all zero except for first coefficient (*h _{1}[0]*) which equals to one and the initial coefficients of feedback path are all zero. The RLS algorithm is used to update equalizer coefficients based on difference between slicer output and equalizer output. To compare the performance of DFE with equalizers with fixed coefficients, the first set of data is sent out as the training sequence to get coefficients of fixed-coefficients equalizers. We then used four different DFE structures (FF, VF, FV, and VV) to compensate the optical communication systems. The resulting BER values when optical signal power is 0.75dBm are listed in Table 2. As shown in Table 2, the fixed-coefficients DFE does not improve system performance and the DFE whose coefficients are updated based on the slicer output actually deteriorates system performance! The signal constellations of systems with DFEs whose coefficients are adaptively updated are illustrated in Fig. 4(a).

As mentioned previously, the DFE is sensitive to wrong decisions of the slicer, and it becomes more serious when the DFE coefficients are continuously updated because a decision error will update equalizer coefficients in a wrong direction. As the result, comparing Figs. 4(a) and 3(b), we can clearly see that if the DFE is updated continuously without error detection, the equalizer performance is even worse than the one without any compensation. To alleviate this issue, we integrated the error detection capacity into the DFE. When a decision error is detected via a wrong parity bit, the equalizer coefficients will not be updated and the previous equalizer coefficients are re-stored. The resulting signal constellations are shown in Fig. 4(b) and BER results are included in Table 2. As shown in Fig. 4(b) and Table 2, once the error detection capability is implemented, the DFE can reduce the system BER and, among different DFE architectures, the VF DFE performs the best. It is worthy of notice that, since error correction code is widely applied in optical communication systems, the BER can be dramatically further reduced once the error correction is conducted [15, 16]. The parity check is implemented in this study due to its simplicity and only serves as a proof-of-concept tool. In the real implementation, more advanced error checking coding might be applied.

Signal power has direct impact on the system performance. Signal with higher power has more tolerance for noise, but higher power can cause significant nonlinear distortions [8]. To study the impact of signal power, we changed optical signal power between 0.454 dBm and 3.72dBm by varying laser power. The resulting relations of BER vs. optical signal power for difference DFEs with parity check are illustrated in Fig. 5. For comparison, the BER of uncompensated system under different power is also included in Fig. 5. Figure 5 demonstrates that the DFE with parity check can improve system performance under different laser power. To further investigate the performance of DFEs with parity check, we show the magnified version of lower part of Fig. 5 in Fig. 6. For comparison, the BER of DFE with perfect error detections is also included in Fig. 6. The perfect error detection is achieved by comparing the slicer output with actually transmitted symbols. Although the perfect error detection is not feasible in real applications, we used it as a benchmark to evaluate the parity check code performance. As illustrated in Fig. 6, there is an optimal power to reach the minimum BER. In this simulation diagram, the optimal power is around 1.48dBm. Comparing DFE using the parity check with DFE using perfect error detection, we do not notice performance difference. For different DFE structures, VF structure (3rd-order Volterra filter in the forward path and linear filter in the feedback path) delivers the best performance among different DFE structures. One explanation is that the feedback path of DFE uses the output of slicer, a nonlinear system; therefore, a Volterra filter in the feedback path does not offer advantages and extra coefficients of Volterra filter might introduce over-fitting issues [21]. The DFE with the FF (linear filter in forward and feedback paths) structure delivers a good performance under low input power when system nonlinearity is not obvious but its performance sacrifices when signal power is high.

Based on the simulation results, we can conclude that error detection capability can greatly improve the DFE performance and a Volterra filter in the forward path will lower BER over a wide range of input power. Since error correction code such as feed-forward error correction (FEC) is widely used in current optical communication system [15, 16], it should be relatively straightforward to implement the error detection functionality in the DFE.

In this simulation, a simple parity check for 16-QAM symbols is used to test the proposed method and its overhead is 25%. The authors are aware that this overhead is comparable with other techniques such as soft decision coding scheme [22]; however, the method proposed in this article presents a new approach worthy of consideration and the parity check coding only serves as tools for proof of concept. Technologies presented in this article can also be used with other error detection method with a lower overhead.

## 5. Conclusion

In this paper, different DFE structures with Volterra and linear filter are investigated. The benefit of implementing error detection functionality in the DFE is also studied. Based on our research results, DFE with 3rd-order Volterra filter in forward path and linear filter in feedback path has stable optimal performance under different signal power. Error detection in the DFE can significantly improve the DFE performance. The results presented in this paper provide great insights for the implementation of DFE which can better compensate for nonlinear distortions.

## References and links

**1. **V. Mikhailov, R. I. Killey, and P. Bayvel, “Optical equalization of nonlinear signal distortion in 42.7-Gb/s RZ transmission,” IEEE Photon. Technol. Lett. **20**(5), 381–383 (2008). [CrossRef]

**2. **Y.-H. Hao, W.-W. Huang, W. Hu, and P. Jiang, “Fiber nonlinearity mitigation in long-haul coherent optical OFDM,” in Proceedings of 2nd International Conference on Information Engineering and Computer Science (ICIECS) (Wuhan, China, 2010), pp. 25–26. [CrossRef]

**3. **J. Leibrich, A. Ali, and W. Rosenkranz, “Decision feedback compensation of transmitter / receiver nonlinearity for DD-OFDM,” in Proceedings of European Conference of Optical Communication (Geneva, Switzerland, 2011) pp.1–3. [CrossRef]

**4. **C. Xia and W. Rosenkranz, “Nonlinear electrical equalization for different modulation formats with optical filtering,” J. Lightwave Technol. **25**(4), 996–1001 (2007). [CrossRef]

**5. **R. Weidenfeld, M. Nazarathy, R. Noe, and I. Shpantzer, “Volterra nonlinear compensation of 112 Gb/s ultra-long-haul coherent optical OFDM based on frequency-shaped decision feedback,” in Proceedings of European Conference of Optical Communication (Vienna, Austria2009) pp. 1–2.

**6. **F. P. Guiomar, J. D. Reis, A. L. Teixeira, and A. N. Pinto, “Mitigation of intra-channel nonlinearities using a frequency-domain Volterra series equalizer,” Opt. Express **20**(2), 1360–1369 (2012). [CrossRef] [PubMed]

**7. **G. Shulkind and M. Nazarathy, “Nonlinear digital back propagation compensator for coherent optical OFDM based on factorizing the Volterra series transfer function,” Opt. Express **21**(11), 13145–13161 (2013). [CrossRef] [PubMed]

**8. **J. Pan and C.-H. Cheng, “Nonlinear electrical compensation for the coherent optical OFDM system,” J. Lightwave Technol. **29**(2), 215–221 (2011). [CrossRef]

**9. **P. M. Watts, V. Mikhailov, S. Savory, P. Bayvel, M. Glick, M. Lobel, B. Christensen, P. Kirkpatrick, S. Shang, and R. I. Killey, “Performance of single-mode fiber links using electronic feed-forward and decision feedback equalizers,” IEEE Photon. Technol. Lett. **17**(10), 2206–2208 (2005). [CrossRef]

**10. **S. Otte and W. Rosenkranz, “A decision feedback equalizer for dispersion compensation in high speed optical transmission systems,” in Proceedings of International Conference on Transparent Optical Networks (Kielce, Poland, 1999), pp.19–22. [CrossRef]

**11. **G. Katz and D. Sadot, “A nonlinear electrical equalizer with decision feedback for OOK optical communication systems,” IEEE Trans. Commun. **56**(12), 2002–2006 (2008). [CrossRef]

**12. **R. de Vries, A. P. Berkhoff, C. H. Slump, and O. E. Herrmann, “Digital compensation of nonlinear distortion in loudspeakers,” in Proceedings of 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing (Institute of Electrical and Electronics Engineers, Minneapolis, MN, USA, 1993) pp. 165–168. [CrossRef]

**13. **V. Y. Krachkovsky, Y. X. Lee, G. Mathew, B. Liu, M. Y. Lin, R. W. Wood, and T. M. Chwee, “Error propagation evaluation for RLL-constrained DFE read channels,” IEEE Trans. Magn. **34**(1), 147–152 (1998). [CrossRef]

**14. **M. Reuter, J. C. Allen, J. R. Zeidler, and R. C. North, “Mitigating error propagation effects in a decision feedback equalizer,” IEEE Trans. Commun. **49**(11), 2028–2041 (2001). [CrossRef]

**15. **X. Yu, W. Jin, M. Sui, and Z. Lan, “Evaluation of forward error correction scheme for underwater wireless optical communication,” in Proceedings of third International Conference on Communications and Mobile Computing (Qingdao, China, 2011) pp. 527–530. [CrossRef]

**16. **T. Mizuochi, Y. Miyata, T. Kobayashi, K. Ouchi, K. Kuno, K. Kubo, K. Shimizu, H. Tagami, H. Yoshida, H. Fujita, M. Akita, and K. Motoshima, “Forward error correction based on block turbo code with 3-bit soft decision for 10-Gb/s optical communication systems,” IEEE J. Sel. Top. Quantum Electron. **10**(2), 376–386 (2004). [CrossRef]

**17. **V. J. Mathews, “Adaptive polynomial filters,” IEEE Signal Process. Mag. **8**(3), 10–26 (1991). [CrossRef]

**18. **A. Gutierrez and W. E. Ryan, “Performance of adaptive Volterra equalizers on nonlinear satellite channels,” in Proceedings of IEEE International Conference on Communications. (Institute of Electrical and Electronics Engineers, Seattle, WA, USA, 1995), pp. 488–492. [CrossRef]

**19. **S. Haykin, *Adaptive Filter Theory,* (Prentice-Hall, 1996).

**20. **A. Agarwal, S. N. Sur, A. K. Singh, H. Gurung, A. K. Gupta, and R. Bera, “Performance analysis of linear and non-linear equalizer in Rician channel,” Procedia Technology **4**, 687–691 (2012). [CrossRef]

**21. **W. J. Pei, Z.-Y. He, L. X. Yang, A. Song, S. S. Hull, and J. Y. Cheung, “A generalized Volterra series method for reconstructing deterministic dynamics from noisy chaotic time series,” in Proceedings of Asia-Pacific Conference on Circuits and Systems (Singapore, 2002) pp.491–494. [CrossRef]

**22. **S. Zhang, M.-F. Huang, F. Yaman, E. Mateo, D. Qian, Y. Zhang, L. Xu, Y. Shao, I. B. Djordjevic, T. Wang, Y. Inada, T. Inoue, T. Ogata, and Y. Aoki, “40×117.6 Gb/s PDM-16QAM OFDM transmission over 10,181 km with soft-decision LDPC coding and nonlinearity compensation,” in the Proceedings of Optical Fiber Communication Conference and Exposition and the National Fiber Optic Engineers Conference (Los Angeles, CA, USA, 2012) pp.1–3.