Improper fuel loading decision results in carrying excessive dead weight during flight operation, which will burden the airline operation cost and cause extra waste emission. Existing works mainly focused on the post-event fuel consumption based on flight trajectory. In this work, a novel deep learning model, called FCPNet, is proposed to achieve the fuel consumption prediction (FCP) before the flight departure. Considering the influential factors for aircraft performance, the multi-modal information sources, including the planned route, weather information, and operation details, are selected as the model input to predict fuel consumption. Correspondingly, three modules are innovatively proposed to learn embedding features from multi-modal inputs. Based on the planned route, the graph convolutional network is proposed to mine the spatial correlations in the non-Eulerian route network. Considering the grid attributes of the weather information, the ConvLSTM is applied to learn abstract representations from both the temporal and spatial dimensions, in which the three-dimensional convolution neural networks are also designed to fine-tune intermediate feature maps. The fully connected layer is also proposed to learn informative features from operation details. Finally, an attention-based fusion network is presented to generate the final embedding by considering the unique contributions of the multi-modality sources, which are further applied to predict flight fuel consumption. A binary encoding representation is proposed to formulate the FCP task as a multi-binary classification problem. The proposed model is validated on a real-world dataset, and the results demonstrate that it outperforms other baselines, i.e., achieving a 6.50% mean absolute percentage error, which can practically support the airline operation and global emission control before flight departure.