The current evaluation of signs relies on quantitative comprehensibility testing. Such testing yields extensive findings about signs’ effectiveness. However, a shortcoming of comprehensibility testing is that it does not provide qualitative information relevant to sign modification and does not facilitate interactions between designers and users. This article advocates the use of visual data to evaluate signs by examining the similarities between signs and drawings produced by end users based on a sign referent given to them. A new evaluation index is developed to measure the extent to which a sign conforms to users’ mental images and to determine whether it should be redesigned. It is calculated by using the learned perceptual image patch similarity. To illustrate the modified approach, a study of safety signs is presented in the article. The article provides an example of how evaluation using visual data can be conducted.