Identifying subgroup markers in heterogeneous populations