High-symbol-rate coherent optical transceivers suffer more from the critical responses of transceiver components at high frequency, especially when applying a higher order modulation format. Recently, we proposed in [20] a neural network (NN)-based digital pre-distortion (DPD) technique trained to mitigate the transceiver response of a 128~GBaud optical coherent transmission system. In this paper, we further detail this work and assess the NN-based DPD by training it using either a direct learning architecture (DLA) or an indirect learning architecture (ILA), and compare performance against a Volterra series-based DPD and a linear DPD. Furthermore, we willfully increase the transmitter nonlinearity and compare the performance of the three DPDs considered. The proposed NN-based DPD trained using DLA performs the best among the three contenders, providing more than 1~dB signal-to-noise ratio (SNR) gains for uniform 64-quadrature amplitude modulation (QAM) and PCS-256-QAM signals at the output of a conventional coherent receiver DSP. Finally, the NN-based DPD enables achieving a record 1.61~Tb/s net rate transmission on a single channel after 80~km of standard single mode fiber (SSMF).

We demonstrate a record 54.5 Tb/s WDM transmission at 11.35 bit/s/Hz over 48 km of field-deployed SMF connecting business and academic parks enabled by a novel joint I-Q Neural Network-based transmitter digital pre-distortion technique.