The superconducting proximity effect leads to strong modifications of the local density of states in diffusive or chaotic cavity Josephson junctions, which displays a phase-dependent energy gap around the Fermi energy. The so-called minigap of the order of the Thouless energy ETh is related to the inverse dwell time in the diffusive region in the limit ETh??, where ? is the superconducting energy gap. In the opposite limit of a large Thouless energy ETh??, a small new feature has recently attracted attention, namely, the appearance of a further secondary gap, which is around two orders of magnitude smaller compared to the usual superconducting gap. It appears in a chaotic cavity just below the superconducting gap edge ? and vanishes for some value of the phase difference between the superconductors. We extend previous theory restricted to a normal cavity connected to two superconductors through ballistic contacts to a wider range of contact types. We show that the existence of the secondary gap is not limited to ballistic contacts, but is a more general property of such systems. Furthermore, we derive a criterion which directly relates the existence of a secondary gap to the presence of small transmission eigenvalues of the contacts. For generic continuous distributions of transmission eigenvalues of the contacts, no secondary gap exists, although we observe a singular behavior of the density of states at ?. Finally, we provide a simple one-dimensional scattering model which is able to explain the characteristic “smile” shape of the secondary gap.