Rank-based optimization techniques for estimation problems in optics