Instabilities and fractal basins of attraction in optical system optimization