Optomechanical response of a nonlinear mechanical resonator