The spectral shape, irradiance, direction, and diffuseness of daylight vary regularly throughout the day. The variations in illumination and their effect on the light reflected from objects may in turn provide visual information as to the time of day. We suggest that artists' color choices for paintings of outdoor scenes might convey this information and that therefore the time of day might be decoded from the colors of paintings. Here we investigate whether human viewers' estimates of the depicted time of day in paintings correlate with their image statistics, specifically chromaticity and luminance variations. We tested time-of-day perception in 17th- to 20th-century Western European paintings via two online rating experiments. In Experiment 1, viewers' ratings from seven time choices varied significantly and largely consistently across paintings but with some ambiguity between morning and evening depictions. Analysis of the relationship between image statistics and ratings revealed correlations with the perceived time of day: higher "morningness" ratings associated with higher brightness, contrast, and saturation and darker yellow/brighter blue hues; "eveningness" with lower brightness, contrast, and saturation and darker blue/brighter yellow hues. Multiple linear regressions of extracted principal components yielded a predictive model that explained 76% of the variance in time-of-day perception. In Experiment 2, viewers rated paintings as morning or evening only; rating distributions differed significantly across paintings, and image statistics predicted people's perceptions. These results suggest that artists used different color palettes and patterns to depict different times of day, and the human visual system holds consistent assumptions about the variation of natural light depicted in paintings.

@en

We explored the underpinnings of providing positive listener experiences for intensive care unit (ICU) patients with compositions of added sounds. Our objective was to derive an approach to such interventions based on soundscape perception and need fulfillment. In one study, we gathered qualitative empirical data about imagined soundscapes where nine fundamental needs were fulfilled. Hierarchical clustering and thematic analysis showed that imagined soundscapes clustered into four types of sonic ambiances, i.e., affective connotations with soundscapes: Comfortable, Pleasurable, Motivating, or Stimulating ambiances. We derived four design parameters to achieve these ambiances with sound compositions: eventfulness, sonic ambiance qualities, narrative structure, and sound distribution. A sound artist was asked to use these parameters to create sound compositions. In a listening experiment, we examined their effects on the perceived pleasantness and eventfulness of soundscapes and on listeners' experienced pleasure and arousal. Soundscapes were perceived as pleasant with varying eventfulness in line with our structured approach. We found a strong correlation between pleasantness and with listener's pleasure and a moderate correlation between eventfulness and with listener's arousal. Finally, we suggested that in future research, three sonic ambiance types should be considered rather than four. Concluding, we showed that our need-driven approach could form a promising way to support ICU patients.@en

Visual perception of materials is crucial for our interactions with the world around us, and understanding the optical and perceptual mechanisms behind it has applications from spectral tuning to holistic appearance tuning. How real materials appear optically is determined by the real light they are in and described by ecological optics. With formal psychophysical experiments we test the perception of their color and texture, how matte/glossy, rough/smooth, hard/soft, glittery, velvety, etcetera we perceive them to be – and how those percepts are related to the ecological optics. I will present a series of studies into light-material-interactions and material perception of computer rendered, real, and painted materials. In addition, I will present research into effects of materials on light, and how materials can be used to make spatial and form-giving qualities of light visible in empty space, using "light probes."@en

We present a method to capture the 7-dimensional light field structure, and translate it into perceptually-relevant information. Our spectral cubic illumination method quantifies objective correlates of perceptually relevant diffuse and directed light components, including their variations over time, space, in color and direction, and the environment’s response to sky and sunlight. We applied it “in the wild”, capturing how light on a sunny day differs between light and shadow, and how light varies over sunny and cloudy days. We discuss the added value of our method for capturing nuanced lighting effects on scene and object appearance, such as chromatic gradients.

@en

Research has shown that disentangling surface and illuminant colors was possible based on various scene statistics. This study investigates the statistical cues induced by the chromatic effects of interreflections. We present a numerical analysis of ambiguous spectral pairs, in which the spectral power distribution of the illuminant in one scene matched the surface reflectance function in the other scene and vice versa. If the scenes are flat or convex and perfectly matte (Lambertian), the reflected light spectra of both cases are identical. However, the incident light undergoes interreflections for concave scenes. The spectral power of interreflections will be absorbed spectrally in an exponential way, dependent on the number of interreflections. We found that this causes systematic shifts towards the spectral reflectance peaks, resulting in brightness, saturation and hue shifts. Those paired cases' color differences (CIEDE2000) are so large that humans would be able to observe them if viewed simultaneously. In addition, we find that the color shifts cause qualitatively different gradients for chromatic materials and achromatic light and vice versa. Further psychophysical testing is necessary to see whether the different color shifts for the two cases can be recognized in isolation due to material or light properties. Moreover, the light densities and light vectors are spectrally different for these cases, creating different appearances of 3D objects in non-empty rooms.@en

Chromatic properties of the effective light in a space are hard to predict, measure and visualise. This is due to complex interactions between materials and illuminants. Here, we describe, measure and visualise the effects of inter-reflections on the structure of the physical light field for diffusely scattering scenes. The spectral properties of inter-reflections vary as a function of the number of bounces they went through. Via a computational model, these spectral variations were found to be systematic and correspond with brightness, saturation and hue shifts. We extended our light-field methods to measure and understand these spectral effects on the first-order properties of light fields, the light density and light vector. We tested the model via a set of computer renderings and cubic spectral illuminance measurements in mock-up rooms under different furnishing scenarios for two types of illuminants. The predicted spectral variations were confirmed and indeed varied systematically within the resulting light field, spatially and directionally. Inter-reflections predominantly affect the light density spectrum and have less impact on the light vector spectrum. It is important to consider these differential effects for their consequences on the colour rendering of 3-dimensional objects and people.

@en

Flavobacteria, which can be found in marine environments, are able to grow in highly organized colonies producing vivid iridescent colorations. While much is known about the biology of these organisms, their design potential as responsive media in user interfaces has not been explored. Our paper aims at bridging this gap by providing insights into the type, degree, and duration of change in Flavobacteria's expression, i.e., their living aesthetics. We present a tool to capture and characterize these changes concerning form, texture and iridescent color. To support the long-term study of their living aesthetics, we designed Flavorium. This bio-digital artifact provides the necessary habitat conditions for Flavobacteria to thrive for a month. Granting insights into the responsive behavior of this organism, this work presents a design space, vocabulary, and application concepts to inspire HCI and design scholars to investigate the complex temporal qualities of living media for future user interfaces.

@en

In everyday life our visual system is continuously exposedto a wide range of motionflow patterns. Our knowledgeof low-level motion processing is substantial, while higher-level processes such as future state prediction, motion con-stancy, and behavioural property estimation remain poorlyunderstood. Here, we concentrate on behavioural cues ofcollectiveflow. Collectiveflow exists of a body of individualagents that show both collective and individual behavioursfollowing a coordinated set of rules. In nature there aremany occurrences of collectiveflow on various scales, var-ious levels of complexity, across both animate and inani-mate systems (e.g., swarms of insects, cars on highways).Using a real-time browser-based simulator of a relativelysimple six-dimensional parametric model we displayed arange of collective behaviours. In a variety of experimentswith free naming, name selection, similarity judgements,and rating tasks we started exploring the parametricspace and its perceived behavioural dimensions. Wefindthat observers can name a wide range of behaviours despitethe abstraction of the simulations. Observers found thewords expressing the spacing between agents to be themost descriptive. However, in the rating experiment it wasfound to be a challenge to differentiate between more dis-tinct definitions of this spacing such as grouping or dispersal.Moreover, the six-dimensional parametric space containedmultiple instances of the same perceived behaviour, makingdirect mappings between the parametric space and percep-tual space even more complex. The challenge will be toclearly tease apart the perceived behaviours with their non-linear interactions across the explored parametric space.[This work was supported by a Marie-Skłodowska-CurieActions Individual Fellowship (H2020-MSCA-IF-2019-FLOW, Project ID: 896434) and a Marie-Skłodowska-CurieActions Innovative Training Network (MSCA-ITN-ETN,grant number 765121, 2017) DyViTo.]@en

In everyday scenes, the effective light (the actual light in a space) can be defined as a complex light field, resulting from a mixture of emissive light sources and indirect mutual surface (inter-)reflections. Hence, the light field typically consists of diffuse and directional illumination and varies in spectral irradiance as a function of location and direction. The spatially varying differences between the diffuse and directional illumination spectra induce correlated colour temperature (CCT) and colour rendition variations over the light fields. Here, we aim to investigate the colourimetric properties of the actual light, termed the effective CCT and colour rendition, for spaces of one reflectance (uni-chromatic spaces). The spectra of the diffuse light-field component (light density) and the directional light-field component (light vector) were measured in both physical and simulated uni-chromatic spaces illuminated by ordinary white light sources. We empirically tested the effective CCT and colour rendition for the light density and the light vector, separately. There were significant differences between the lamp-specified CCT and colour rendition and the actual light-based effective CCT and effective colour rendition. Inter-reflections predominantly affected the CCT and colour rendition of the light density relative to the light vector. Treating the diffuse and directional light-field components in a linear model reveals the separate influences of the light source and scene. These effects show the importance of a 3D version of colour checkers for lighting designers, architects or in general computer graphics applications, for which we propose simple Lambertian spheres.

@en

Product lighting design is a laborious and time-consuming task. With product illustrations being increasingly rendered, the lighting challenge transferred to the virtual realm. Our approach targets lighting design in the context of a scene with fixed objects, materials, and camera parameters, illuminated by environmental lighting. Our solution offers control over the depiction of material characteristics and shape details by optimizing the illuminating environment-map. To that end, we introduce a metric that assesses the shape and material cues in terms of the designed appearance. We formalize the process and support steering the outcome using additional design constraints. We illustrate our solution with several challenging examples.

@en

Psychological needs of users as a basis for design are at the core of design practice, yet the importance of fundamental human needs when designing soundscapes has not been studied specifically. This paper investigates the relationship between nine fundamental human needs and the affective qualities and categories of soundscapes. In a free-labeling survey study, we collected descriptions of imagined sound environments for the fulfilment of the needs, as well as ratings of the perceived affective quality of these environments. We found that needs were associated with pleasant soundscapes, while their eventfulness varied. ‘Human’ sounds were a common category for each of the nine needs considered in this study, but systematic variations of the categories were found dependent on the need. Results suggest that designing categorically different soundscapes dependent on the users’ needs will have beneficial effects.@en

Deep learning has paved the way for strong recognition systems which are often both trained on and applied to natural images. In this paper, we examine the give-and-take relationship between such visual recognition systems and the rich information available in the fine arts. First, we find that visual recognition systems designed for natural images can work surprisingly well on paintings. In particular, we find that interactive segmentation tools can be used to cleanly annotate polygonal segments within paintings, a task which is time consuming to undertake by hand. We also find that FasterRCNN, a model which has been designed for object recognition in natural scenes, can be quickly repurposed for detection of materials in paintings. Second, we show that learning from paintings can be beneficial for neural networks that are intended to be used on natural images. We find that training on paintings instead of natural images can improve the quality of learned features and we further find that a large number of paintings can be a valuable source of test data for evaluating domain adaptation algorithms. Our experiments are based on a novel large-scale annotated database of material depictions in paintings which we detail in a separate manuscript.

@en

This paper aims to provide first insights into flash characteristics of bioluminescent microalgae as a potential media for future living light interfaces. A growing number of HCI and interaction design researchers show interest in living material interfaces, which incorporate living organisms for novel responsive behaviour and interaction possibilities in digital and biological hybrids. While much is known about the science of these organisms, their gliving aesthetics', i.e., how humans experience the unique temporal changes in a living media, have hardly been explored. To bridge this gap in designing living light interfaces, this paper presents a study of bioluminescent flash characterisation. A DIY shaking device was designed to interact with the liquid living media, providing a range of stimuli including orbital rotation, pulsation and vibration. The living light aesthetics is presented with rich visuals illustrating the intensity variations over time, textural qualities and spatial distribution.

@en

Dutch 17th century painters were masters in depicting materials and their properties in a convincing way. Here, we studied the perception of the material signatures and key image features of different depicted fabrics, like satin and velvet. We also tested whether the perception of fabrics depicted in paintings related to local or global cues, by cropping the stimuli. In Experiment 1, roughness, warmth, softness, heaviness, hairiness, and shininess were rated for the stimuli shown either full figure or cropped. In the full figure, all attributes except shininess were rated higher for velvet, whereas shininess was rated higher for satin. This distinction was less clear in the cropped condition, and some properties were perceived significantly different between the two conditions. In Experiment 2 we tested whether this difference was due to the choice of the cropped area. On the basis of the results of Experiment 1, shininess and softness were rated for multiple crops from each fabric. Most crops from the same fabric differed significantly in shininess, but not in softness perception. Perceived shininess correlated positively with the mean luminance of the crops and the highlights’ coverage. Experiment 1 showed that painted velvet and satin triggered distinct perceptions, indicative of robust material signatures of the two fabrics. The results of Experiment 2 suggest that the presence of local image cues affects the perception of optical properties like shininess, but not mechanical properties such as softness.

@en

In this paper, we capture and explore the painterly depictions of materials to enable the study of depiction and perception of materials through the artists’ eye. We annotated a dataset of 19k paintings with 200k+ bounding boxes from which polygon segments were automatically extracted. Each bounding box was assigned a coarse material label (e.g., fabric) and half was also assigned a fine-grained label (e.g., velvety, silky). The dataset in its entirety is available for browsing and downloading at materialsinpaintings.tudelft.nl. We demonstrate the cross-disciplinary utility of our dataset by presenting novel findings across human perception, art history and, computer vision. Our experiments include a demonstration of how painters create convincing depictions using a stylized approach. We further provide an analysis of the spatial and probabilistic distributions of materials depicted in paintings, in which we for example show that strong patterns exists for material presence and location. Furthermore, we demonstrate how paintings could be used to build more robust computer vision classifiers by learning a more perceptually relevant feature representation. Additionally, we demonstrate that training classifiers on paintings could be used to uncover hidden perceptual cues by visualizing the features used by the classifiers. We conclude that our dataset of painterly material depictions is a rich source for gaining insights into the depiction and perception of materials across multiple disciplines and hope that the release of this dataset will drive multidisciplinary research.

@en

Food appearance sets intentions and expectations. When designing packaged food much attention is devoted to packaging elements like color and shape, but less to the characteristics of the images used. To our awareness, no study has yet investigated how the appearance of the food shown on the package affects consumers’ preferences. Often, orange juice packages depict an orange. Juiciness being one of the most important parameters to assess oranges’ quality, we hypothesized that an orange with a juicier appearance on the package would improve the overall evaluation of the juice. Using image cues found to trigger juiciness perception of oranges depicted in 17th century paintings, we designed four orange juice packages by manipulating the highlights on the pulp (present vs. absent) and the state of the orange (unpeeled vs. peeled). In an online experiment, 400 participants, each assigned to one condition, rated expected naturalness, healthiness, quality, sweetness and tastiness of the juice, package attractiveness and willingness to buy. Finally, they rated juiciness of the orange for all four images. A one-way ANOVA showed a significant effect of the highlights on juiciness. A MANOVA showed that the presence of highlights, both in isolation and in interaction with the peeled side, also significantly increased expected quality and tastiness of the juice. The present study shows the importance of material perception and food texture appearance in the imagery of food packaging. We suggest that knowledge from vision science on image features and material perception should be integrated into the process of packaging design.@en

A common strategy for improving model robustness is through data augmentations. Data augmentations encourage models to learn desired invariances, such as invariance to horizontal flipping or small changes in color. Recent work has shown that arbitrary style transfer can be used as a form of data augmentation to encourage invariance to textures by creating painting-like images from photographs. However, a stylized photograph is not quite the same as an artist-created painting. Artists depict perceptually meaningful cues in paintings so that humans can recognize salient components in scenes, an emphasis which is not enforced in style transfer. Therefore, we study how style transfer and paintings differ in their impact on model robustness. First, we investigate the role of paintings as style images for stylization-based data augmentation. We find that style transfer functions well even without paintings as style images. Second, we show that learning from paintings as a form of perceptual data augmentation can improve model robustness. Finally, we investigate the invariances learned from stylization and from paintings, and show that models learn different invariances from these differing forms of data. Our results provide insights into how stylization improves model robustness, and provide evidence that artist-created paintings can be a valuable source of data for model robustness. Code and data are available at: https://github.com/hubertsgithub/style_painting_robustness@en

In natural scenes, the wavelength-dependent sunlight scattering in the atmosphere and the presence of occluders and mutual reflections cause variations in the local illumination's magnitude, direction, and spectral composition as a function of time and space. Yet, unlike the characterization of temporal changes of natural light, these spatial variations have received relatively limited attention. In this paper, we employed the light field concept to quantify those variations and empirically demonstrated that they are significant and how the different contributions of the diffuse and directed parts can be measured.@en