Optimization of ship energy efficiency is an efficient measure to decrease fuel usage and emissions in the shipping industry. The accurate prediction model of ship energy usage is the basis to achieve optimization of ship energy efficiency. This study investigates the sequential properties of the actual voyage data from a VLOC. On this basis, a model for predicting ship energy consumption is established by adopting a LSTM neural network that has better prediction performance for sequential datasets. To further enhance the performance of the established LSTM-based model, the network structures and hyperparameters are optimized by using Genetic Algorithm. Lastly, the application analysis is conducted to validate the established GA-LSTM-based model for ship fuel usage prediction. The established model for ship energy usage shows a significant improvement in prediction accuracy, compared to the original LSTM-based model. Meanwhile, the developed prediction model is more accurate than the existing BP, SVR, and ARIMA-based energy consumption models. The prediction errors for the ship's operational energy efficiency adopting the established GA-LSTM-based model can reach as low as 0.29%. Therefore, the established model can effectively predict the ship fuel usage under different conditions, which is essential for the optimization and improvement of ship energy efficiency.