Large Multimodal Modals can be subjected to similar psychophysical paradigms as human observers, affording comparison between human and machine vision. In this context, we explored material perception. We created 32 stimuli of a constant 3D shape but with various material properties. Then we presented them in 1193 triplets in an odd-one-out task for both humans (N=18) and machine. The machine judgements were performed with gpt-4o, which has vision capabilities. Triplet data was both analysed directly, and also used to create perceptual embeddings using Soft Ordinal Embedding (SOE). The raw triplet data revealed an interesting commonality between human and machine judgements when we compared the ‘popularity scores’ of odd-ones-out: a group of 6 stimuli was substantially more different from the remaining 26 stimuli. Furthermore, we found that 47% of the triplet judgements were similar for the human and gpt-4o data, which is well above chance level (33%). The SOE analysis revealed that the accuracy (agreement between raw triplet data and multidimensional embeddings) was substantially higher for machine than human vision, indicating a higher degree of internal consistency. Also, we found a full saturation at 6 dimensions for the machine data: all triplets could be accounted for by the embedding. Besides various commonalities, the embeddings themselves revealed some peculiar differences. Firstly, translucent stimuli were close for humans but distant for the machine. Secondly, the machine embedding showed a clear cluster of achromatic stimuli, while this was entirely absent in the human data. This suggests that computers use colour for material perception, while humans do not. With some imagination, one could argue that human material perception partly prepares for physical interaction where colour is irrelevant, while the algorithm does not (yet) have a body to interact with the outside world.

We present a framework that connects ideas from the visual arts and visual perception. It adapts two existing frameworks for the analysis of form and content so that it can be used in an educational context for teaching perception through visual arts. The basis is the formal analysis of texture, colour, light, space, and material. This analysis can be conducted both on the medium and the motif, which adds a second level in addition to the formal level. Thirdly, a conte(n/x)t level is discussed which combines a basic notion of semiotics and iconography. We share our experience of implementing pictorial analysis in design and perception education and discuss how the framework is used both in a quantitative and a qualitative fashion. Next to education, the framework provides a basis for further pictorial research.

We investigated the influence of the medium on the perception of depicted objects and materials. Oil paintings and their reproductions in engravings were chosen because they are vastly distinctive media while having completely identical content. A total of 15 pairs were collected, consisting of 88 fragments depicting different materials, including fabric, skin, wood and metal. Besides the original condition, we created three manipulations to understand the effect of colour (a greyscale version) and contrast (equalised histograms towards both painting and engraving). We performed rating experiments on five attributes: three-dimensionality, glossiness, convincingness, smoothness and softness. An average of 25 participants finished each of the 20 online experimental sessions (five attributes X four conditions). Besides clear correlations between the two media, the differences mainly show in their means (different levels of perceived attributes) and standard deviations (perceived range). In most sessions, paintings depict a wider range than engravings. In addition, it was the histogram equalisation (global contrast) that made the most impact on perceived attributes, rather than colour removal. This suggests that engravers compensated for the lack of colour by exploiting the possibilities of local contrast.

Humans can rapidly identify materials, such as wood or leather, even within a complex visual scene. Given a single image, one can easily identify the underlying "stuff," even though a given material can have highly variable appearance; fabric comes in unlimited variations of shape, pattern, color, and smoothness, yet we have little trouble categorizing it as fabric. What visual cues do we use to determine material identity? Prior research suggests that simple "texture" features of an image, such as the power spectrum, capture information about material properties and identity. Few studies, however, have tested richer and biologically motivated models of texture. We compared baseline material classification performance to performance with synthetic textures generated from the Portilla-Simoncelli model and several common image degradations. The textures retain statistical information but are otherwise random. We found that performance with textures and most degradations was well below baseline, suggesting insufficient information to support foveal material perception. Interestingly, modern research suggests that peripheral vision might use a statistical, texture-like representation. In a second set of experiments, we found that peripheral performance is more closely predicted by texture and other image degradations. These findings delineate the nature of peripheral material classification.

Pictorial research can rely on computational or human annotations. Computational annotations offer scalability, facilitating so-called distant-viewing studies. On the other hand, human annotations provide insights into individual differences, judgments of subjective nature. In this study, we demonstrate the difference in objective and subjective human annotations in two pictorial research studies: one focusing on Avercamp’s perspective choices and the other on Rembrandt’s compositional choices. In the first experiment, we investigated perspective handling by the Dutch painter Hendrick Avercamp. Using visual annotations of human figures and horizons, we could reconstruct the virtual viewpoint from where Avercamp depicted his landscapes. Results revealed an interesting trend: with increasing age, Avercamp lowered his viewpoint. In the second experiment, we studied the compositional choice that Rembrandt van Rijn made in Syndics of the Drapers’ Guild. Based on imaging studies it is known that Rembrandt doubted where to place the servant, and we let 100 annotators make the same choice. Subjective data was in line with evidence from imaging studies. Aside from having their own merit, the two experiments demonstrate two distinctive ways of performing pictorial research, one that concerns the picture alone (objective) and one that concerns the relation between the picture and the viewer (subjective).

The spectral shape, irradiance, direction, and diffuseness of daylight vary regularly throughout the day. The variations in illumination and their effect on the light reflected from objects may in turn provide visual information as to the time of day. We suggest that artists' color choices for paintings of outdoor scenes might convey this information and that therefore the time of day might be decoded from the colors of paintings. Here we investigate whether human viewers' estimates of the depicted time of day in paintings correlate with their image statistics, specifically chromaticity and luminance variations. We tested time-of-day perception in 17th- to 20th-century Western European paintings via two online rating experiments. In Experiment 1, viewers' ratings from seven time choices varied significantly and largely consistently across paintings but with some ambiguity between morning and evening depictions. Analysis of the relationship between image statistics and ratings revealed correlations with the perceived time of day: higher "morningness" ratings associated with higher brightness, contrast, and saturation and darker yellow/brighter blue hues; "eveningness" with lower brightness, contrast, and saturation and darker blue/brighter yellow hues. Multiple linear regressions of extracted principal components yielded a predictive model that explained 76% of the variance in time-of-day perception. In Experiment 2, viewers rated paintings as morning or evening only; rating distributions differed significantly across paintings, and image statistics predicted people's perceptions. These results suggest that artists used different color palettes and patterns to depict different times of day, and the human visual system holds consistent assumptions about the variation of natural light depicted in paintings.

In European painting, a transition took place where artists started to consciously introduce blurred or soft contours in their works. There may have been several reasons for this. One suggestion in art historical literature is that this may have been done to create a stronger sense of volume in the depicted figures or objects. Here we describe four experiments in which we tried to test whether soft or blurred contours do indeed enhance a sense volume or depth. In the first three experiments, we found that, for both paintings and abstract shapes, three dimensionality was actually decreased instead of increased for blurred (and line) contours, in comparison with sharp contours. In the last experiment, we controlled for the position of the blur (on the lit or dark side) and found that blur on the lit side evoked a stronger impression of three dimensionality. Overall, the experiments robustly show that an art historical conjecture that a blurred contour increases three dimensionality is not granted. Because the blurred contours can be found in many established art works such as from Leonardo and Vermeer, there must be other rationales behind this use than the creation of a stronger sense of volume or depth.

Most studies on the perception of style have used whole scenes/entire paintings; in our study, we isolated a single motif (an apple) to reduce or even eliminate the influence of composition, iconography, and other contextual information. In this article, we empirically address two fundamental questions of the existence (Experiment 1) and description (Experiment 2) of style. We chose 48 cut-outs of mostly Western European paintings (15th to 21st century) that showed apples. In Experiment 1, 415 unique participants completed online triplet similarity tasks. Multidimensional scaling (MDS) reached a nonrandom three-dimensional (3D) embedding, showing that participants are able to judge stylistic differences in a systematic way. We also found a strong correlation between creation year and embedding, both a linear correlation with Dimension 2, and a rotational correlation in the first two dimensions. To interpret the embedding further, in Experiment 2, we fitted three color statistics and nine attribute ratings (glossiness, three-dimensionality, convincingness, brush coarseness, etc.) to the 3D perceptual style space. Results showed that Dimension 1 is associated with spatial attributes (Smoothness, Brushstroke coarseness) and Convincingness, Dimension 2 is related to Hue, and Dimension 3 is related to Chroma. The results suggest that texture and color are two important variables for style perception. By isolating the motifs, we could exclude higher levels of information such as composition and context. Interestingly, the results reinforce previous findings using whole scenes, suggesting that style can already be perceived in sometimes very small fragments of paintings.

We present a method to capture the 7-dimensional light field structure, and translate it into perceptually-relevant information. Our spectral cubic illumination method quantifies objective correlates of perceptually relevant diffuse and directed light components, including their variations over time, space, in color and direction, and the environment’s response to sky and sunlight. We applied it “in the wild”, capturing how light on a sunny day differs between light and shadow, and how light varies over sunny and cloudy days. We discuss the added value of our method for capturing nuanced lighting effects on scene and object appearance, such as chromatic gradients.

If two painters paint the same scene, the appearance difference can be referred to as style difference. The distinguishing features result from artists’ use of composition, color, brushstroke etc. We are interested in how people perceive different depiction styles, when they are presented with different levels of information. Whole paintings contain mid-level information (depicted scenes, etc.) and low-level information (brushstroke, colors, etc.). Square cut-outs of single objects contain only low-level information. The same cut-outs in grayscale contain low-level information but without colors. We collected 42 digitized oil paintings as stimuli, the creation years varied from 15th to 21st century, and their location of production varied from southern Spain to the northern Netherlands. All paintings contain at least one apple. We gathered similarity judgement data using a triplet comparison method from three online experiments, where observers were presented the whole paintings (condition 1), square cut-outs of painted apples (condition 2) and the same cut-outs in grayscale (condition 3). 20 observers completed each experiment (60 observers in total). We applied soft ordinal embedding to achieve multidimensional embeddings. We reached a 3D space for condition 1 and 3, and a 4D space for condition 2. Condition 2 has less information than condition 1, but has one more dimension, suggesting that different criteria might be involved. Condition 3 has one less dimension than condition 2, suggesting that color is one of the attributes for style perception judgement. In addition, having the same dimensionality, around 64% of the raw data was in line with the 3D embedding in condition 1 and 58% in condition 3. This difference suggests that although the whole scene and a grayscale cut-out both need three dimensions to describe their style differences, the implicit style criteria for grayscale cut-outs are apparently more ambiguous than those used to judge the whole paintings.

In everyday scenes, the effective light (the actual light in a space) can be defined as a complex light field, resulting from a mixture of emissive light sources and indirect mutual surface (inter-)reflections. Hence, the light field typically consists of diffuse and directional illumination and varies in spectral irradiance as a function of location and direction. The spatially varying differences between the diffuse and directional illumination spectra induce correlated colour temperature (CCT) and colour rendition variations over the light fields. Here, we aim to investigate the colourimetric properties of the actual light, termed the effective CCT and colour rendition, for spaces of one reflectance (uni-chromatic spaces). The spectra of the diffuse light-field component (light density) and the directional light-field component (light vector) were measured in both physical and simulated uni-chromatic spaces illuminated by ordinary white light sources. We empirically tested the effective CCT and colour rendition for the light density and the light vector, separately. There were significant differences between the lamp-specified CCT and colour rendition and the actual light-based effective CCT and effective colour rendition. Inter-reflections predominantly affected the CCT and colour rendition of the light density relative to the light vector. Treating the diffuse and directional light-field components in a linear model reveals the separate influences of the light source and scene. These effects show the importance of a 3D version of colour checkers for lighting designers, architects or in general computer graphics applications, for which we propose simple Lambertian spheres.

Visualizing biosignals can be important for social Virtual Reality (VR), where avatar non-verbal cues are missing. While several biosignal representations exist, designing effective visualizations and understanding user perceptions within social VR entertainment remains unclear. We adopt a mixed-methods approach to design biosignals for social VR entertainment. Using survey (N=54), context-mapping (N=6), and co-design (N=6) methods, we derive four visualizations. We then ran a within-subjects study (N=32) in a virtual jazz-bar to investigate how heart rate (HR) and breathing rate (BR) visualizations, and signal rate, influence perceived avatar arousal, user distraction, and preferences. Findings show that skeuomorphic visualizations for both biosignals allow differentiable arousal inference; skeuomorphic and particles were least distracting for HR, whereas all were similarly distracting for BR; biosignal perceptions often depend on avatar relations, entertainment type, and emotion inference of avatars versus spaces. We contribute HR and BR visualizations, and considerations for designing social VR entertainment biosignal visualizations.

Research has shown that disentangling surface and illuminant colors was possible based on various scene statistics. This study investigates the statistical cues induced by the chromatic effects of interreflections. We present a numerical analysis of ambiguous spectral pairs, in which the spectral power distribution of the illuminant in one scene matched the surface reflectance function in the other scene and vice versa. If the scenes are flat or convex and perfectly matte (Lambertian), the reflected light spectra of both cases are identical. However, the incident light undergoes interreflections for concave scenes. The spectral power of interreflections will be absorbed spectrally in an exponential way, dependent on the number of interreflections. We found that this causes systematic shifts towards the spectral reflectance peaks, resulting in brightness, saturation and hue shifts. Those paired cases' color differences (CIEDE2000) are so large that humans would be able to observe them if viewed simultaneously. In addition, we find that the color shifts cause qualitatively different gradients for chromatic materials and achromatic light and vice versa. Further psychophysical testing is necessary to see whether the different color shifts for the two cases can be recognized in isolation due to material or light properties. Moreover, the light densities and light vectors are spectrally different for these cases, creating different appearances of 3D objects in non-empty rooms.

Before the invention of photography, paintings were reproduced in a graphic and linear medium, engravings. To compare material perception across two modalities, paintings and engravings, we conducted two online experiments. We collected 15 pairs of color oil paintings and their engraving reproductions. Then we selected 40 elements from these 15 pairs, including fabric and skin, which resulted in 80 stimuli in total. In experiment 1, we used original (colored) versions for both paintings and engravings. In experiment 2, we used the same stimulus set, but achromatic (luminance only). Two attributes were rated in both experiments: glossiness and softness. 30 observers completed online rating tasks for each attribute in each experiment (120 observers in total). For glossiness, independent of color or black and white versions, engravings scored higher than paintings. In experiment 1, engravings were rated as glossier in 28 out of 40 pairs of variations, with 11 out of these 28 pairs showing significant differences. In experiment 2 (achromatic), engravings scored higher in glossiness than paintings in 33 pairs with 21 pairs showing significant differences. Both numbers increased when colors were removed. For softness, 21 elements in experiment 1 and 29 elements in experiment 2 were judged glossier in engravings than in paintings. Surprisingly, engravings performde well in presenting both gloss and softness. Moreover, when colors were removed, the performance of engravings in conveying glossiness and softness got even better. The increased number of significant cases underscores the robustness of this trend.

While automatic computational techniques appear to reveal novel insights in digital art history, a complementary approach seems to get less attention: that of human annotation. We argue and exemplify that a 'human in the loop' can reveal insights that may be difficult to detect automatically. Specifically, we focused on perceptual aspects within pictorial art. Using rather simple annotation tasks (e.g. delineate human heights, indicate highlights and classify gaze direction) we could both replicate earlier findings and reveal novel insights into pictorial conventions. We found that Canaletto depicted human figures in rather accurate perspective, varied viewpoint elevation between approximately 3 and 9 m and highly preferred light directions parallel to the projection plane. Furthermore, we found that taking the averaged images of leftward-looking faces reveals a woman, and for rightward-looking faces showed a male, confirming earlier accounts on lateral gender bias in pictorial art. Lastly, we confirmed and refined the well-known light-from-the-top-left bias. Together, the annotations, analyses and results exemplify how manual annotations can contribute and complement to digital art history.

The effective illumination incident on an object in a three-dimensional scene is a geometrically-weighted sum of direct and indirect light. The luminous and chromatic properties of the light field vary spatially and directionally, inducing luminance and chromatic gradients - smooth color variations over objects. When a color combination of a step or gradient produces a pleasing effect, it is said to be harmonious. Previous studies have shown that perception of color harmony is dependent on a complex interplay between hue, chroma and lightness (Ou and Luo, 2006). Further, the visual cues from luminance and chromatic gradients might assist three-dimensional shape recovery (Ruppertsberg et al., 2008). The aim of this research is to investigate the influence of chromatic furnishing materials on the perception of object color harmony and shape, through inter-reflections. Box spaces were rendered with uni-chromatic surfaces and a colored sphere, acting as a probe, in its center, illuminated by a planar white illuminant. 24 room surface colors were sampled systematically in RGB space. The sphere’s color was sampled from the 15 CIE CRI color checker samples. Participants had to rate perceived three-dimensionality (flat disk vs. sphere) and color harmony (disharmonious vs. harmonious) of the rendered sphere under carefully calibrated conditions. Before each session and between trials, participants adapted to an animated noisy mask. A short training session introduced randomly selected stimuli after which the main experiment took place. Of the tested furnishing hues, the bluish rooms resulted in the highest mean color harmony and three-dimensionality scores. Decreasing the furnishing brightness resulted in a major three-dimensionality enhancement, as expected. Reducing the saturation and even more so the brightness of the chromatic furnishing colors enhanced the perceived color harmony of the probe. These effects show the importance of 3D versions of color checkers, here we used spheres, for color testing.

Food appearance sets intentions and expectations. When designing packaged food much attention is devoted to packaging elements like color and shape, but less to the characteristics of the images used. To our awareness, no study has yet investigated how the appearance of the food shown on the package affects consumers’ preferences. Often, orange juice packages depict an orange. Juiciness being one of the most important parameters to assess oranges’ quality, we hypothesized that an orange with a juicier appearance on the package would improve the overall evaluation of the juice. Using image cues found to trigger juiciness perception of oranges depicted in 17th century paintings, we designed four orange juice packages by manipulating the highlights on the pulp (present vs. absent) and the state of the orange (unpeeled vs. peeled). In an online experiment, 400 participants, each assigned to one condition, rated expected naturalness, healthiness, quality, sweetness and tastiness of the juice, package attractiveness and willingness to buy. Finally, they rated juiciness of the orange for all four images. A one-way ANOVA showed a significant effect of the highlights on juiciness. A MANOVA showed that the presence of highlights, both in isolation and in interaction with the peeled side, also significantly increased expected quality and tastiness of the juice. The present study shows the importance of material perception and food texture appearance in the imagery of food packaging. We suggest that knowledge from vision science on image features and material perception should be integrated into the process of packaging design.

In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists’ eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at materialsinpaintings.tudelft.nl. We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.