Extracting hysteresis from nonlinear measurement of wavefront-sensorless adaptive optics system