Asphalt mixture is a skeleton filling system consisting of aggregate, asphalt, and mineral powder. High performance asphalt mixture design is directly affected by internal stress transmission of skeleton filling system. In this paper, the discrete element method based on digital image was employed to study the stress transmission. By analyzing the effect of skeleton structure on the contact force between aggregate and mortar, it is concluded that the skeleton aggregate in the asphalt mixture is the main stress transmission medium. The lower the degree of disruption, the higher the contact force of skeleton aggregate. At the same time, the variation coefficient of contact force of asphalt mortar with low filling coefficient is larger and the localization phenomenon is more serious. By analyzing the effect of mortar properties on the contact force between aggregate and mortar, it is concluded that the lower the modulus of asphalt mortar, the higher the degree of localization of contact force and the more likely to have local damage.