The growth and invasion of solid tumors are associated with changes in their viscoelastic properties, influenced by both internal cellular factors and physical forces in the tumor microenvironment. Due to the lack of a comprehensive investigation of tumor tissue viscoelasticity, the relationship between such physical properties and cancer malignancy remains poorly understood. Here, the viscoelastic properties of breast cancer spheroids, 3D (in vitro) tumor models, are studied in relation to their metastatic potentials by imposing controlled, dynamic compression within a microfluidic constriction, and subsequently monitoring the relaxation of the imposed deformation. By adopting a modified Maxwell model to extract viscoelastic properties from the compression data, the benign (MCF-10A) spheroids are found to have higher bulk elastic modulus and viscosity compared to malignant spheroids (MCF-7 and MDA-MB-231). The relaxation is characterized by two timescales, captured by a double exponential fitting function, which reveals a similar fast rebound for MCF-7 and MCF-10A. Both the malignant spheroids exhibit similar long-term relaxation and display residual deformation. However, they differ significantly in morphology, particularly in intercellular movements. These differences between malignant spheroids are demonstrated to be linked to their cytoskeletal organization, by microscopic imaging of F-actin within the spheroids, together with cell-cell adhesion strength.