High-symbol-rate coherent optical transceivers suffer more from the critical responses of transceiver components at high frequency, especially when applying a higher order modulation format. Recently, we proposed in [20] a neural network (NN)-based digital pre-distortion (DPD) technique trained to mitigate the transceiver response of a 128~GBaud optical coherent transmission system. In this paper, we further detail this work and assess the NN-based DPD by training it using either a direct learning architecture (DLA) or an indirect learning architecture (ILA), and compare performance against a Volterra series-based DPD and a linear DPD. Furthermore, we willfully increase the transmitter nonlinearity and compare the performance of the three DPDs considered. The proposed NN-based DPD trained using DLA performs the best among the three contenders, providing more than 1~dB signal-to-noise ratio (SNR) gains for uniform 64-quadrature amplitude modulation (QAM) and PCS-256-QAM signals at the output of a conventional coherent receiver DSP. Finally, the NN-based DPD enables achieving a record 1.61~Tb/s net rate transmission on a single channel after 80~km of standard single mode fiber (SSMF).@en

We propose a novel digital pre-distortion (DPD) based on neural networks for high-baudrate optical coherent transmitters. We demonstrate experimentally that it outperforms an optimized linear DPD giving a 1.2 dB SNR gain in a 128GBaud PCS-256QAM single-channel transmission over 80km of standard single-mode fiber resulting in a record 1.61 Tb/s net data rate.@en

We demonstrate a record 54.5 Tb/s WDM transmission at 11.35 bit/s/Hz over 48 km of field-deployed SMF connecting business and academic parks enabled by a novel joint I-Q Neural Network-based transmitter digital pre-distortion technique.@en

We present a simple, efficient “direct learning” approach to train Volterra series-based pre-distortion filters using neural networks. We show its superior performance over conventional training methods using a 64-QAM 64 GBaud simulated transmitter with varying transmitter nonlinearity and noisy conditions.@en