Point clouds (PCs) have attracted researchers and developers due to their ability to provide immersive experiences with six degrees of freedom (6DoF). However, there are still several open issues in understanding the Quality of Experience (QoE) and visual attention of end users while experiencing 6DoF volumetric videos. First, encoding and decoding point clouds require a significant amount of both time and computational resources. Second, QoE prediction models for dynamic point clouds in 6DoF have not yet been developed due to the lack of visual quality databases. Third, visual attention in 6DoF is hardly explored, which impedes research into more sophisticated approaches for adaptive streaming of dynamic point clouds. In this work, we provide an open-source Compressed Point cloud dataset with Eye-tracking and Quality assessment in Mixed Reality (ComPEQ - MR). The dataset comprises four compressed dynamic point clouds processed by Moving Picture Experts Group (MPEG) reference tools (i.e., VPCC and GPCC), each with 12 distortion levels. We also conducted subjective tests to assess the quality of the compressed point clouds with different levels of distortion. The rating scores are attached to ComPEQ - MR so that they can be used to develop QoE prediction models in the context of MR environments. Additionally, eye-tracking data for visual saliency is included in this dataset, which is necessary to predict where people look when watching 3D videos in MR experiences. We collected opinion scores and eye-tracking data from 41 participants, resulting in 2132 responses and 164 visual attention maps in total. The dataset is available at https://ftp.itec.aau.at/datasets/ComPEQ-MR/.

@en

Immersive technologies like eXtended Reality (XR) are the next step in videoconferencing. In this context, understanding the effect of delay on communication is crucial. This article presents the first study on the impact of delay on collaborative tasks using a realistic Social XR system. Specifically, we design an experiment and evaluate the impact of end-to-end delays of 300, 600, 900, 1,200, and 1,500 ms on the execution of a standardized task involving the collaboration of two remote users that meet in a virtual space and construct block-based shapes. To measure the impact of the delay in this communication scenario, objective and subjective data were collected. As objective data, we measured the time required to execute the tasks and computed conversational characteristics by analyzing the recorded audio signals. As subjective data, a questionnaire was prepared and completed by every user to evaluate different factors such as overall quality, perception of delay, annoyance using the system, level of presence, cybersickness, and other subjective factors associated with social interaction. The results show a clear influence of the delay on the perceived quality and a significant negative effect as the delay increases. Specifically, the results indicate that the acceptable threshold for end-to-end delay should not exceed 900 ms. This article additionally provides guidelines for developing standardized XR tasks for assessing interaction in Social XR environments.@en

Extended Reality (XR) has emerged as a transformative and immersive technology with versatile applications in content creation and consumption. As XR gains popularity, companies eager to adopt it often possess a surface-level understanding, investing significant resources without effectively addressing the genuine needs of end-users. This study explores the current workflows of XR production companies, and the potential of social XR in mitigating challenges throughout the XR production workflow. We present the outcomes of three respective focus group workshops conducted with three XR production companies and their experts (N=17). The results indicate that at every stage of the production, namely pre-production, production, post-production, and post-release, there are communication challenges between producers and clients, as well as different production and post-production specialists. We discuss various aspects of XR concerning the problem and propose novel opportunities offered by social XR to ameliorate those challenges, improving communication and making development more agile.

@en

Point clouds represent one of the prevalent formats for 3D content. Distortions introduced at various stages in the point cloud processing pipeline affect the visual quality, altering their geometric composition, texture information, or both. Understanding and quantifying the impact of the distortion domain on visual quality is vital to driving rate optimization and guiding post-processing steps to improve the quality of experience. In this paper, we propose a multi-task guided multi-modality no reference metric (M3-Unity), which utilizes 4 types of modalities across attributes and dimensionalities to represent point clouds. An attention mechanism establishes inter/intra associations among 3D/2D patches, which can complement each other, yielding local and global features, to fit the highly nonlinear property of the human vision system. A multi-task decoder involving distortion type classification selects the best association among 4 modalities, aiding the regression task and enabling the in-depth analysis of the interplay between geometrical and textural distortions. Furthermore, our framework design and attention strategy enable us to measure the impact of individual attributes and their combinations, providing insights into how these associations contribute particularly in relation to distortion type. Extensive experimental results on 4 datasets consistently outperform the state-of-the-art metrics by a large margin.@en

As 3D immersive media continues to gain prominence, Point Cloud Quality Assessment (PCQA) is essential for ensuring high-quality user experiences. This paper introduces ViSam-PCQA, a no-reference PCQA metric guided by visual saliency information across three modalities, which facilitates the performance of the quality prediction. Firstly, we project the 3D point cloud to acquire 2D texture, depth, and normal maps. Secondly, we extract the saliency map based on the texture map and refine it with the corresponding depth map. This refined saliency map is used to weight low-level feature maps to highlight perceptually important areas in the texture channel. Thirdly, high-level features from the texture, normal, and depth maps are then processed by a Transformer to capture global and local point cloud representations across the three modalities. Lastly, saliency along with global and local embeddings, are concatenated and processed through a multi-task decoder to derive the final quality scores. Our experiments on the SJTU, WPC, and BASICS datasets show high Spearman rank order correlation coefficients/Pearson linear correlation coefficients of 0.953/0.962, 0.920/0.920 and 0.887/0.936 respectively, demonstrating superior performance compared to current state-of-the-art methods.@en

In recent years, a large variety of online communication tools have emerged, including social Virtual Reality (VR) platforms for interacting in a virtual world with participants being represented as virtual avatars. Given their popularity, an active area of research focuses on improving the user experience in these virtual experiences. To enable experimentation at large scale on online platforms, it is however essential to collect behavioural data (e.g. movements and audio information). In this work, we present a toolchain that enables the running of experiments using a modified version of the social VR platform Mozilla Hubs. Specifically, our toolkit enables collection and tracking or user positions and movements at a central location, enabling fine-grained analysis of user behaviour during a social VR experience.

@en

The latest social VR technologies have enabled users to attend traditional media and arts performances together while being geographically removed, making such experiences accessible despite budget, distance, and other restrictions. In this work, we aim at improving the way remote performances are shared by designing and evaluating a VR theatre lobby which serves as a space for users to gather, interact, and relive the common experience of watching a virtual opera. We conducted an initial test with experts ($\mathrm{N}=10$, i.e., designers and opera enthusiasts) in pairs using our VR lobby prototype, developed based on the theoretical lobby design concept. A unique aspect of our experience is its highly realistic representation of users in the virtual space. The test results guided refinements to the VR lobby structure and implementation, aiming to improve the user experience and align it more closely with the social VR lobby's intended purpose. With the enhanced prototype, we ran a between-subject controlled study ($\mathrm{N}=40$) to compare the user experience in the social VR lobby between individuals and paired participants. To do so, we designed and validated a questionnaire to measure the user experience in the VR lobby. Results of our mixed-methods analysis, including interviews, questionnaire results, and user behavior, reveal the strength of our social VR lobby in connecting with other users, consuming the opera in a deeper manner, and exploring new possibilities beyond what is common in real life. All supplemental materials are available at https://github.com/cwi-dis/IEEEVR2024-VRLobby.

@en

Volumetric video (VV) is a novel form of video that allows recreation of real-world scenes in 3D with users consuming the content from any viewpoint they desire. This makes VV best suited for augmented reality (AR) or virtual reality (VR) applications. This freedom necessitates increased user interaction with the VV itself, which brings new challenges to its visual quality assessment. In this chapter, various aspects of VV quality assessment using subjective user studies and objective quality estimation methods are discussed. These aspects include the manner of representing 3D models, mode of interaction, display settings (e.g., whether AR or VR headsets are used), rendering parameters, and how the characteristics of point clouds or meshes are used in quality estimation. The chapter discusses the advantages and disadvantages of different methods and provides take away messages for researchers.

@en

Immersive reality technologies, such as Virtual and Augmented Reality, have ushered a new era of user-centric systems, in which every aspect of the coding-delivery-rendering chain is tailored to the interaction of the users. Understanding the actual interactivity and behaviour of the users is still an open challenge and a key step to enabling such a user-centric system. Our main goal is to extend the applicability of existing behavioural methodologies for studying user navigation in the case of 6 Degree-of-Freedom (DoF). Specifically, we first compare the navigation in 6-DoF with its 3-DoF counterpart highlighting the main differences and novelties. Then, we define new metrics aimed at better modelling behavioural similarities between users in a 6-DoF system. We validate and test our solutions on real navigation paths of users interacting with dynamic volumetric media in 6-DoF Virtual Reality conditions. Our results show that metrics that consider both user position and viewing direction better perform in detecting user similarity while navigating in a 6-DoF system. Having easy-To-use but robust metrics that underpin multiple tools and answer the question "how do we detect if two users look at the same content?"open the gate to new solutions for a user-centric system.

@en

The development and widespread adoption of immersive XR applications has led to a renewed interest in representations that are capable of reproducing real-world objects and scenes with high fidelity. Among such representations, point clouds have attracted the interest of industry and academia alike, and new compression solutions have been developed to facilitate their adoption in mainstream applications. To ensure the best quality of experience for the end-user in limited bandwidth scenarios, new full-reference objective quality metrics have been proposed, promoting features designed specifically for point cloud contents. However, the performance of such features to predict the quality of point cloud contents when the reference is not available is largely unexplored. In this paper, we evaluate the performance of features commonly used to model point cloud distortions in a no-reference framework. The obtained features are integrated into a quality value through a support vector regression model. Results demonstrate the potential of full-reference features for no-reference assessment.

@en

Watching HMD-based 360° video has become in-creasing popular as a medium for immersive viewing of photo-realistic content. To evaluate subjective video quality, researchers typically prompt users to provide an overall Quality of Experience (QoE) score after viewing a stimulus. However, since users can adjust their viewport throughout a 360° video, a higher level of spatiotemporal granularity is needed for adaptive 360° video streaming. To address this, we design several real-time, continuous QoE annotation input and peripheral visualization techniques, with the goal of minimizing mental workload and distraction during score acquisition. Drawing on two parallel co-design sessions with seven experts, we find that touchpad and joystick are most suitable for continuous input, with DotMorph (circle with tick label that varies in filling) for peripheral state feedback. We contribute design findings for testing QoE score acquisition techniques during HMD-based 360° video watching, which enable more precise optimization of adaptive video streaming quality.@en

Technological advances in head-mounted displays and novel real-time 3D acquisition and reconstruction solutions have fostered the development of 6 Degrees of Freedom (6DoF) teleimmersive systems for social VR applications. Point clouds have emerged as a popular format for such applications, owing to their simplicity and versatility; yet, dense point cloud contents are too large to deliver directly over bandwidth-limited networks. In this context, user-adaptive delivery mechanisms are a promising solution to exploit the increased range of motion offered by 6DoF VR applications to yield gains in perceived quality of 3D point cloud user representations, while reducing their bandwidth requirements. In this paper, we perform a user study in VR to quantify the gains adaptive tile selection strategies can bring with respect to non-adaptive solutions. In particular, we define an auxiliary utility function, we employ established methods from the literature and newly-proposed schemes for distributing the bit budget across the tiles, and we evaluate them together with non-adaptive streaming baselines through subjective QoE assessment. Results confirm that considerable gains can be obtained with user-adaptive streaming, achieving bit rate gains of up to 65% with respect to a non-adaptive approach to deliver comparable quality. Our analysis provides useful insights for the design and development of social VR applications.

@en

This work presents an explorative behavioural analysis of users navigating in an immersive space aimed at enabling the next-generation multimedia systems. Our main goal is to understand how the user experience of immersive content with 6-Degrees-of-Freedom (DoF) is affected not only by the visual content and its quality but also by the disposition of the user. We based our investigations on traditional statistical metrics, on techniques that have been already used for 6-DoF, as well as adapted 3-DoF tools to be used in this new context. We show the limitation of each metric in giving a complete interpretation of user behaviour, and we draw insights on important factors to be considered when analysing and predicting navigation trajectories. Specifically, we have noticed in our behavioural investigations that the user disposition plays an important role in the way of interacting with the immersive content. This opens the gate to user profiles (i.e., a collection of key information that describes the behavioural features of a single or group of users) that would be beneficial for different purposes in future immersive applications such as enabling new modalities for live streaming services optimised per user profiles but also for user-based quality assessment methods.

@en

Social Virtual Reality (VR) applications represent a big step forward in the field of remote communication. Social VR provides the possibility for participants to explore and interact with virtual environments and objects, feelings of a full sense of immersion, and being together. Understanding how user behaviour is influenced by the shared virtual space and its elements becomes the key to design and optimize novel immersive experiences. This paper presents a behavioural analysis of user navigating in 6 degrees of freedom social VR movie. Specifically, we analyse 48 user trajectories from a photorealistic telepresence experiment, in which subjects watch a crime movie together in VR. We investigate how users are affected by salient agents (i.e., virtual characters) and by narrative elements of the VR movie (i.e., dialogues versus interactive part). We complete our assessment by conducting a statistical analysis of the collected data. Results indicate that user behaviour is affected by different narrative and interactive elements. We conclude by presenting our observations and drawing conclusions on future paths for social VR experiences. This work has been supported by Royal Society under grant IES R1180128 and by Cisco under Cisco Research Center Donation scheme.

@en

Point cloud representation has seen a surge of popularity in recent years, thanks to its capability to reproduce volumetric scenes in immersive scenarios. New compression solutions for streaming of point cloud contents have been proposed, which require objective quality metrics to reliably assess the level of degradation introduced by coding and transmission distortions. In this context, reduced reference metrics aim to predict the visual quality of the transmitted contents, while requiring only a small set of features to be sent in addition to the streamed media. In this paper, we propose a reduced reference metric to predict the quality of point cloud contents under compression distortions. To do so, we extract a small set of statistical features from the reference point cloud in the geometry, color and normal vector domain, which can be used at the receiver side to assess the visual degradation of the content. Using publicly available ground-truth datasets, we compare the performance of our metric to widely-used full reference metrics. Results demonstrate that our metric is able to effectively predict the level of distortion in the degraded point cloud contents, achieving high correlation values with respect to subjective scores.

@en

In recent years, there has been an increased interest in point cloud representation for visualizing digital humans in cross reality. However, due to their voluminous size, point clouds require high bandwidth to be transmitted. In this paper, we propose a temporal interpolation architecture capable of increasing the temporal resolution of dynamic digital humans, represented using point clouds. With this technique, bandwidth savings can be achieved by transmitting dynamic point clouds in a lower temporal resolution, and recreating a higher temporal resolution on the receiving side. Our interpolation architecture works by first downsampling the point clouds to a lower spatial resolution, then estimating scene flow using a newly designed neural network architecture, and finally upsampling the result back to the original spatial resolution. To improve the smoothness of the results, we additionally apply a novel technique called neighbour snapping. To be able to train and test our newly designed network, we created a synthetic point cloud data set of animated human bodies. Results from the evaluation of our architecture through a small-scale user study show the benefits of our method with respect to the state of the art in scene flow estimation for point clouds. Moreover, correlation between our user study and existing objective quality metrics confirm the need for new metrics to accurately predict the visual quality of point cloud contents.

@en