Quantifying the nanomechanical properties of soft-matter using multi-frequency atomic force microscopy (AFM) is crucial for studying the performance of polymers, ultra-thin coatings, and biological systems. Such characterization processes often make use of cantilever's spectral components to discern nanomechanical properties within a multi-parameter optimization problem. This could inadvertently lead to an over-determined parameter estimation with no clear relation between the identified parameters and their influence on the experimental data. In this work, we explore the sensitivity of viscoelastic characterization in polymeric samples to the experimental observables of multi-frequency intermodulation AFM. By performing simulations and experiments we show that surface viscoelasticity has negligible effect on the experimental data and can lead to inconsistent and often non-physical identified parameters. Our analysis reveals that this lack of influence of the surface parameters relates to a vanishing gradient and non-convexity while minimizing the objective function. By removing the surface dependency from the model, we show that the characterization of bulk properties can be achieved with ease and without any ambiguity. Our work sheds light on the sensitivity issues that can be faced when optimizing for a large number of parameters and observables in AFM operation, and calls for the development of new viscoelastic models at the nanoscale and improved computational methodologies for nanoscale mapping of viscoelasticity using AFM.