With the development of 5G/6G communication technology, an increasing number of communication protocols for data transmission are created and implemented. At the same time, the redundancy of packet headers has become a matter to be optimized when transferring packets across different protocol layers, which causes high resource occupancy, low transmission efficiency and high latency, especially for real-time communications like Voice over IP (VoIP). The currently standardized Robust Header Compression (RoHC) Algorithm seeks to provide an approach to compressing the protocol headers inscribed in data packets to decrease the amount of data transmission without reducing the communication quality. However, the existing RoHC algorithm is not efficient enough for the requirements of present header compression, as it's initially designed for specific headers like RTP, UDP, and IP. The packet headers must be optimized over the protocol stack for the real-time data stream and the GPRS Tunneling Protocol (GTP) tunnel sections in the 5G network. An Optimized Header (OH) Compression Algorithm based on the existing RoHC algorithm is proposed in this thesis to compress the headers over the 5G protocol stack in a coordinated fashion for real-time transmission and GTP tunnels. Besides, further development of the OH algorithm is done to optimize protocol headers by merging duplicate header fields for both multiple QoS flows and multi-layer Protocol Data Unit (PDU) flows in the same PDU session. The overall optimization is conducted in the 5G System (5GS) over F1-U and N3 reference points. The simulation results indicate that the OH algorithm improves the average header compression rate by 45% compared with the RoHC algorithm, as demonstrated via scenario-based 5GS simulations. The average number of transported bits is decreased by 30%, 15% and 26% for Opus coding audio stream, H.264-Opus coding video-audio stream and BLE Version 4.2 triple packets for smartwatch telemedicine stream, respectively, compared with the RoHC algorithm. Besides, the average algorithm execution latency is decreased by 2 μs compared with the RoHC algorithm. Moreover, the multi-stream and multi-layer optimization for the OH algorithm brings about a higher header compression rate and lower execution time in specific scenarios. The overall results indicate that the OH algorithm is capable of compressing the packet size effectively, decreasing the number of transported bits, and shortening the execution time, resulting in higher transmission efficiency and lower latency for VoIP services.