This paper evaluates the influence of an additional visual aesthetic layer on the experience of concert goers during a live event. The additional visual layer incorporates musical features as well as bio-sensing data collected during the concert, which is coordinated by our audience engagement monitoring technology. This technology was used during a real Jazz concert. The collected measurements were used in an experiment with 32 participants, where two different forms of visualization were compared: one factoring in music amplitude, audience engagement collected by the sensors and the dynamic atmosphere of the event, the other one purely relying on the beat of the music. The findings indicate that the visual layer could add value to the experience if used during a live concert, providing a higher level of immersion and feeling of togetherness among the audience.

@en

Crowdworker online communities -- operating in fora like mTurkForum and TurkerNation -- are an important actor in microwork markets. Albeit central to market dynamics, how the behavior of crowdworker communities and the dynamics of online marketplaces influence each other is yet to be understood. To provide quantitative evidence of such influence, we performed an analysis on 6-years worth of mTurk market activities and community discussions in six fora. We investigated the nature of the relationships that exist between activities in fora, tasks published in mTurk, requesters for such tasks, and task completion speed. We validate -- and expand upon -- results from previous work by showing that (i) there are differences between market demand and community activities that are specific to fora and task types; (ii) the temporal progression of HIT availability in the market is predictive of the upcoming amount of crowdworker discussions, with significant differences across fora and discussion categories; (iii) activities in fora can have a significant positive impact on the completion speed of tasks available in the market.@en

Many studies have indicated that predicting users’ perception of visual quality depends on various factors other than artifact visibility alone, such as viewing environment, social context, or user personality. Exploiting information on these factors, when applicable, can improve users’ quality of experience while saving resources. In this paper, we improve the performance of existing no-reference image quality metrics (NR-IQM) using image semantic information (scene and object categories), building on our previous findings that image scene and object categories influence user judgment of visual quality. We show that adding scene category features, object category features, or the combination of both to perceptual quality features results in significantly higher correlation with user judgment of visual quality. We also contribute a new publicly available image quality dataset which provides subjective scores on images that cover a wide range of scene and object category evenly. As most public image quality datasets so far span limited semantic categories, this new dataset opens new possibilities to further explore image semantics and quality of experience.@en

Objective video quality metrics are designed to be as reliable as the subjective quality assessments on which they are calibrated and validated. However, existing standard methodologies for subjective video quality assessment provide low reliable results for some conditions. We investigate whether an extension of the quality ruler experimental methodology, originally defined for images and shown to be more reliable than, e.g., standard single stimulus (SS) methods, can be adapted to reliably assess the quality of videos. The video quality ruler methodology allows subjects to assess video quality using a set of reference anchor images (the ruler), spanning a wide range of quality altogether, but closely spaced in function of quality one from the other. Subjects are asked to compare the quality of the displayed test video with the quality of these anchor images, displayed on a tablet, and indicate which of the reference images matches in quality the test video. As a result, the video quality assessment task is reduced to a set of visual comparisons between video and reference image quality. We describe how to adapt the original quality ruler methodology to video quality assessment, and we compare the proposed methodology with two other, widely used experimental methodologies: the single stimulus (SS) and the double stimulus (DS) method. Our results show that video quality ruler is a reliable method to assess video quality according to a multitude of criteria.

@en

This paper presents an effective method based on Vector Regression and Object oriented Pooling (VROP) for blind image quality assessment (BIQA). Unlike previous models which map the extracted features directly to a quality score, the proposed vector regression framework yields a vector of belief scores for the input image. We explore the uncertainty factors in quality assessment and design the belief scores to measure the confidences of an image to be assigned to the corresponding quality grades. Moreover, we propose an object oriented pooling strategy to further improve the performance by incorporating semantic information of image contents. According to this strategy, regions occupied by objects will be assigned more weights in the pooling phase, leading to a more accurate quality assessment. Extensive experiments on benchmark datasets demonstrate that our approach achieves state-of-the-art performance and shows a great generalization ability.

@en

Video-conferencing is becoming an essential part in everyday life. The visual channel allows for interactions which were not possible over audio-only communication systems such as the telephone. However, being a de-facto over-the-top service, the quality of the delivered video-conferencing experience is subject to variations, dependent on network conditions. Video-conferencing systems adapt to network conditions by changing for example encoding bitrate of the video. For this adaptation not to hamper the benefits related to the presence of a video channel in the communication, it needs to be optimized according to a measure of the Quality of Experience (QoE) as perceived by the user. The latter is highly dependent on the ongoing interaction and individual preferences, which have hardly been investigated so far. In this paper, we focus on the impact video quality has on conversations that revolve around objects that are presented over the video channel. To this end we conducted an empirical study where groups of 4 people collaboratively build a Lego® model over a video-conferencing system. We examine the requirements for such a task by showing when the interaction, measured by visual and auditory cues, changes depending on the encoding bitrate and loss. We then explore the impact that prior experience with the technology and affective state have on QoE of participants. We use these factors to construct predictive models which double the accuracy compared to a model based on the system factors alone. We conclude with a discussion how these factors could be applied in real world scenarios.

@en

As crowdsourcing gains popularity, organisations seek ways to systematically and reliably involve their workforce with data processing pipelines. Mobile crowdsourcing allows for opportunistic task executions and thus, potentially, for higher throughput. However, how to engage and to retain employees in enterprise crowdsourcing campaigns is still an open research topic. .is paper discusses the results of a study performed in IBM Benelux. We surveyed 93 employees to discover the factors that might a.ect engagement in mobile enterprise crowdsourcing. .e survey informed the design of an experiment that aimed at investigating the e.ectiveness of di.erent task noti€cation strategies. We studied how factors such as time and context of noti€cation can a.ect the participation and retention of employees. Results show that break times are the most suitable for crowd work, and that "aggressive" noti€cation strategies act as deterrent for participation, while moderate yet regular nudges are the most likely to retain contributors.@en

The driving force behind digital crowdsourcing are its workers: working, hidden behind the scenes, churning out data in experiments, participating in research studies, completing little tasks to accomplish HITs online. Understanding workers and crowdwork better is therefore key to develop a more effective and fair use of crowdsourcing for research. This chapter attempts to help develop an understanding of the various aspects of the crowd by drawing parallels between workers of different platforms (AMT, Microworkers and Crowdee) through quantitative and qualitative analysis of current and newly collected data. A picture of the crowd is drawn by uncovering their motivations, workplaces, skills and infrastructure, issues and perspectives about the design of microtasks, the employers and the microtask-based platforms. Legal and ethical perspectives on crowdwork are also discussed, and online resources are reviewed that researchers can use as a primer to employ crowdworkers in an ethical and fair way. The chapter provides information, a review of internationally recognised ethical principles and practical advice to those who would like to use crowdsourcing for experiments and to carry out research studies as an informed researcher and crowd employer.

@en

This article showcases our effort to explore the music club of the future. We present the development and results of an end-to-end system which enhances the club-going experience through the use of wearable technology. Each party guest wearing one of the wristbands actively contributes to the overall experience with their movement and location patterns. The system collects acceleration data from each of the attendees in real-time and feeds it into a pluggable network infrastructure, which processes the data, affecting the environment via data visualization or controlling of the light and sound system of a curated space within the club. Finally we describe the results of a two night, 450 person per night deployment.@en

Although compression and transmission artifacts are likely to appear simultaneously in digital videos, their annoyance has been traditionally studied and modeled in isolation. So, while blockiness, blurriness, and packet-loss metrics exist, hardly any attempt has been made at modeling their joint impact on visual perception. In this paper, we evaluate the perceptual impact of those three artifacts on video quality. Based on data from three different experiments, in which a pool of participants evaluated videos impaired with packet-loss, blurriness and blockiness (in isolation and in combination), we analyze how the different artifacts combine to produce annoyance and propose several models for predicting the annoyance of videos impaired with combinations of packet-loss, blurriness and blockiness.@en

Most video platforms deliver HD video in high bitrate encoding. Modern video-conferencing systems are capable of handling HD streams, but using multiparty conferencing, average internet connections in the home are on their bandwidth limit. For properly managing the encoding bitrate in videoconferencing, we must know what is the minimum bitrate requirement to provide users an acceptable experience, and what is the bitrate level after which QoE saturates?. Most available subjective studies in this area used rather dated technologies. We report on a multiparty study on video quality with HD resolution. We tested different encoding bitrates (256kbs, 1024kbs and 4096kbs) and packet loss rates (0, 0.5%) in groups of 4 participants with a scenario based on the ITU building blocks task. We discuss the influence of group interaction and individual idiosyncrasies based on different mixed models, and look at covariates engagement and enjoyment as further explanatory factors. We found that 256kbs is still sufficient to provide a fair overall experience, but video quality is noticed to be poor. On the higher bitrate end, most people will not perceive the difference between 1024kbs and 4096kbs, considering in both cases the quality to be close to excellent. Independent on bitrate, packet loss has a small but significant impact, quantifiable in, on average, less than half a point difference on a 5-point ITU scale.@en

Recognizing what makes an image aesthetically pleasing is crucial to the effectiveness of many multimedia systems. Several works have attempted to build image aesthetic appeal predictors, and created their own set of ground truth data for the purpose, either by using rated images from photo sharing websites, or by asking a pool of users to rate images in lab or crowdsourcing experiments. Literature has shown that the way these experiments are conducted can influence their results: poor experimental setup can result in poorly reliable outcomes (i.e., highly imprecise aesthetic appeal measures). A question then arises whether the different choices made to collect ground truth of aesthetic appeal data are appropriate. In this paper, we propose a systematic study that looks into how different experimental environments and rating scales used to collect image aesthetic appeal ground truth data influence the reliability and repeatability of aesthetic appeal assessments. Our findings show that discrete and continuous scales with five-point absolute category rating labels yield more reliable results, with the continuous scale being more reliable for abstract images. We also show that image aesthetic appeal assessments could be repeatable across different experimental environments (i.e., lab and crowdsourcing). We finally formulate concrete recommendations to guide the collection of large sets of ground truth data for training models of aesthetic appeal appreciation.@en

We hypothesize that the semantics of image content affects how humans judge the perceptual quality of images. The recognition of image content has been shown to be processed within the first 500 ms of observation (and mostly in a pre-attentive stage). We look at whether or not participants are also able to detect impairments and judge their annoyance at early attentive stages. As the presence of impairments may slow down the early semantic recognition process, we investigate whether or not different semantic content impacts people’s judgment of image quality. Our results show that participants do recognize image content despite the presence of impairments even at very early stages of vision (within the first fixation). In addition, we show that semantic categories have an influence on people’s detection of image impairments at early attentive stages. People seem to be able to correctly detect very obvious impairments within one fixation, but more subtle impairments are not perceived. Finally, we show that people are more tolerant toward impairments on images portraying outdoor scenes than images portraying indoor scenes; additionally, users seem to be more critical toward images containing animate objects (humans or animals) in the region of interest compared with those with inanimate objects.@en

Complexity is crucial to characterize tasks performed by humans through computer systems. Yet, the theory and practice of crowdsourcing currently lacks a clear understanding of task complexity, hindering the design of effective and efficient execution interfaces or fair monetary rewards. To understand how complexity is perceived and distributed over crowdsourcing tasks, we instrumented an experiment where we asked workers to evaluate the complexity of 61 real-world re-instantiated crowdsourcing tasks. We show that task complexity, while being subjective, is coherently perceived across workers; on the other hand, it is significantly influenced by task type. Next, we develop a high-dimensional regression model, to assess the influence of three classes of structural features (metadata, content, and visual) on task complexity, and ultimately use them to measure task complexity. Results show that both the appearance and the language used in task description can accurately predict task complexity. Finally, we apply the same feature set to predict task performance, based on a set of 5 years-worth tasks in Amazon MTurk. Results show that features related to task complexity can improve the quality of task performance prediction, thus demonstrating the utility of complexity as a task modeling property.@en

Multimedia experiences allow us to access other worlds, to live other people's stories, to communicate with or experience alternate realities. Different spaces, times or situations can be entered thanks to multimedia contents and systems, which coexist with our current reality, and are sometimes so vivid and engaging that we feel we are living in them. Advances in multimedia are making it possible to create immersive experiences that may involve the user in a different or augmented world, as an alternate reality. AltMM 2016, the 1st International Workshop on Multimedia Alternate Realities at ACM Multimedia, aims at exploring how the synergy between multimedia technologies and effects can foster the creation of alternate realities and make their access an enriching, valuable and real experience. The workshop program will contain a combination of oral and invited keynote presentations, and poster, demo and discussion sessions, altogether enabling interactive scientific sharing and discussion between practitioners and researchers.@en

Understanding the perceptual impact of compression artifacts in video is one of the keys for designing better coding schemes and appropriate visual quality control chains. Although compression and transmission artifacts, such as blockiness, blurriness, and packet-loss, appear simultaneously in digital videos, traditionally they have been studied in isolation. In this paper, we report the results of three subjective quality assessment experiments aimed at studying perceptual characteristics of a set of artifacts common in digital videos. With this goal, first, we study the annoyance of each of three artifacts (blockiness, blurriness, and packet-loss) in isolation and then in combination. Based on the subjective evaluations, we design several models of the annoyance caused by the joint presence of these three artifacts on digital video.@en

The persistence of negative moods (sadness and anxiousness) induced by three visual Mood Induction Procedures (MIP) was investigated. The evolution of the mood after the MIP was monitored for a period of 8 min with the Self-Assessment Manikin (SAM; every 2 min) and with recordings of skin conductance level (SCL) and electrocardiography (ECG). The SAM pleasure ratings showed that short and longer film fragments were effective in inducing a longer lasting negative mood, whereas the negative mood induced by the IAPS slideshow was short lived. The induced arousal during the anxious MIPs diminished quickly after the mood induction; nevertheless, the SCL data suggest longer lasting arousal effects for both movies. The decay of the induced mood follows a logarithmic function; diminishing quickly in the first minutes, thereafter returning slowly back to baseline. These results reveal that caution is needed when investigating the effects of the induced mood on a task or the effect of interventions on induced moods, because the induced mood diminishes quickly after the mood induction.@en

In modern video conferencing services, just as in commonvideo delivery, most of the resource optimization is taken careof in the codec layer. Modern codecs like H.264 use detailedperceptual models to optimize the data reduction in way that itis least noticed by us. Already early evaluations oftelecommunication systems could establish that there aredifferent thresholds for a good quality depending on thesituation. It is further known that subjective quality perceptionsvary from user to user. But the space of user and context factorsis still largely unexplored. To gain insight in which parametersare key in differentiating quality perception, we need to explorethe interaction in different situations while keeping a tightcontrol over the system parameters. In this paper we explorehow clustering participants by their interaction or ratingbehavior can reveal subgroups that show significantly differentperception of the QoE delivered by the same videoconferencingsystem. While for a cluster of users we find video quality toinfluence other QoE dimensions such as audio, for anothercluster this is not the case. We explore whether this effect is dueto conversational dynamics (contextual factor) or individualpreferences (user factor) and discuss what this would mean forthe design of future video-conferencing systems, that want todynamically adapt to situation and participants.@en

Most automatic Quality of Experience (QoE) assessment models have so far aimed at predicting the QoE of a video as experienced by an average user, and solely based on perceptual characteristics of the video being viewed. The importance of other characteristics, such as those related to the video content being watched, or those related to an individual user have been largely neglected. This is suboptimal in view of the fact that video viewing experience is individual and multifaceted, considering the perceived quality (related to coding or network-induced artifacts), but also other -- more hedonic - aspects, like enjoyment. In this paper, we propose an expanded model which aims to assess QoE of a given video, not only in terms of perceived quality but also of enjoyment, as experienced by a specific user. To do so, we feed the model not only with information extracted from the video (related to both perceived quality and content), but also with individual user characteristics, such as interest, personality and gender. We assess our expanded QoE model based on two publicly available QoE datasets, namely i_QoE and CP-QAE-I. The results show that combining various types of characteristics enables better QoE prediction performance as compared to only considering perceptual characteristics of the video, both when targeting perceived quality and enjoyment.@en