In order to account explicitly for the existence of long-periodic layered structures and the strong structural relaxations in the most common binary and ternary alloys of the Bi-Sb-Te-Se system, we have developed a one-dimensional cluster expansion (CE) based on first-principles electronic structure calculations, which accounts for the Bi and Sb bilayer formation. Excellent interlayer distances are obtained with a van der Waals density functional. It is shown that a CE solely based on pair interactions is sufficient to provide an accurate description of the ground-state energies of Bi-Sb-Te-Se binary and ternary systems without making the data set of ab initio calculated structures unreasonably large. For the binary alloys A1?xQx (A=Sb, Bi; Q=Te, Se), a ternary CE yields an almost continuous series of (meta)stable structures consisting of consecutive A bilayers next to consecutive A2Q3 for 0