Pursuing higher communication rates is a perpetual goal, especially in today's age of information explosion. To increase line rate without extending the optical and electrical bandwidth, advanced modulation formats such as probabilistic constellation shaping (PCS) and partial response signaling (PRS) have been researched. PCS offers up to 1.53 dB sensitivity gains by modifying the uniform distribution of transmitted symbols into the Maxwell-Boltzmann distribution to approximate a Gaussian distribution; PRS, also known as faster-than-Nyquist signaling, increases the symbol rate beyond the ideal Nyquist bandwidth limit by introducing controlled inter-symbol interference (ISI). However, when applied to PCS-enabled or PRS-enabled systems, digital signal processing (DSP) algorithms developed for conventional QAM with uniform probabilities and free ISIs often perform poorly, degrading the expected gain.

To maximize the benefits of PCS and PRS, thereby enhancing the performance of single-carrier coherent communication systems, this thesis focuses on addressing the problem of optimal degradation in the carrier recovery stage. In coherent optical communication systems, carrier recovery is a crucial DSP subsystem that compensates for time-varying carrier frequency offset and phase noise caused by both lasers at the transmitter and receiver.

We first propose a carrier recovery scheme using generalized maximum likelihood estimation with negligible pilot overhead (approximately 0.2%) in the context of PCS. Through simulations and 100 GBaud experiments with PCS-64QAM, our proposed scheme doubles computational efficiency, provides better estimation accuracy, and exhibits greater stability, leading to up to 0.25 dB sensitivity gain compared to other algorithms. We also determine the practical optimal shaping factors of PCS-64QAM for different SNR intervals to guide future experimental work.

Additionally, in a popular PRS scheme, Tomlinson-Harashima precoding combined with polybinary shaping (THP + Polybinary), we compare two 2M modulo formats and investigate their impact on carrier recovery. Based on simulations and 96 GBaud experiments with THP+Polybinary-16QAM, our carrier recovery scheme enhances both accuracy and stability, effectively mitigating issues caused by zero symbols and residual ISIs.

With its superior overall performance, the proposed carrier recovery scheme is a competitive algorithm for addressing carrier imperfections in both PCS-enabled and PRS-enabled high-speed coherent optical communications. For the future, modifying the timing recovery algorithm and the blind equalization algorithm to suit PCS/PRS systems is prioritized to maximize PCS/PRS gains. Given the vast amount of communication data, machine learning-based DSP algorithms also present an interesting avenue for future research.