The future trajectories of surrounding agents are critical for the motion planning and control of autonomous vehicles. Thus, this study employs Transformer to develop a multi-agent trajectory prediction model named Multi-agent Trajectory Vector Transformer (MaTVT). MaTVT features a lightweight architecture, comprising a dual-level encoder formed by a low-level encoder and a high-level encoder, along with a multi-modal decoder. Once input enters MaTVT, the low-level encoder first constructs polar coordinate systems centered on target agents and then projects historical trajectories and map elements to each agent-centered coordinate system. Next, it utilizes attention mechanisms to encode motion features, agent-agent interactions, and agent-infrastructure constraints independently and fuses them into the agent encoding sequence. Considering the agent response delay, the low-level encoder extracts heterogeneous spatial-temporal features from agent encoding sequences as the local encodings for target agents. Afterward, the high-level encoder treats all agents as the nodes in a directed graph and utilizes a Graph Attention Network to convert inter-agent relationships into global encodings, which are fused with the local encodings of target agents. Finally, the multi-modal decoder translates these fusion encodings into multi-modal trajectory predictions for target agents. This study selects complex traffic scenarios from the Argoverse Motion Forecasting dataset to create a dedicated dataset for MaTVT training, validation, and testing. The test results demonstrate that MaTVT outperforms advanced benchmark methods in prediction performance, revealing its superb accuracy, efficiency, and robustness. In addition, ablation studies further explain the interpretability of the main functional components of MaTVT and their contributions to prediction performance.