Common optimization approaches for solving the muscle redundancy problem in musculoskeletal simulations can predict shoulder contact forces that either violate or barely satisfy joint stability requirements, with force directions falling outside or near the perimeter of the glenoid cavity. In this study, several glenohumeral stability formulations were tested against in vivo measurements of glenohumeral contact forces from the Orthoload dataset on one participant data in lateral, posterior, and anterior dumbbell raises. The investigated formulations either constrained the contact force direction to remain within different shapes of a stability perimeter, or added a penalty term that discouraged contact force directions from deviating from the glenoid cavity center. All stability formulations predicted contact force magnitudes that agreed relatively well to the in vivo measured forces except for the strictest formulation that constrained the joint contact force directly to the glenoid cavity center. Constraint and conditional penalty models estimated force vectors that largely lay along the perimeters. Continuous penalty models estimated relatively more accurate contact force directions within the glenoid cavity than constraint models. Our findings support the proposed penalty formulations as more reasonable and accurate than other investigated existing glenohumeral stability formulations.