We present eSense - an open and multi-sensory in-ear wearable platform to detect and monitor human activities. eSense is a true wireless stereo (TWS) earbud with dual-mode Bluetooth and Bluetooth Low Energy and augmented with a 6-axis inertial measurement unit and a microphone. We showcase the eSense platform, its data APIs to capture real-time multi-modal sensory data in a data exploration tool, and its manifestation in a 360◦ workplace well-being application.@en

We present eSense - an open and multi-sensory in-ear wearable platform for personal-scale behaviour analytics. eSense is a true wireless stereo (TWS) earbud and supports dual-mode Bluetooth and Bluetooth Low Energy. It is also augmented with a 6-axis inertial measurement unit and a microphone. We demonstrate the eSense platform, the data exploration tool with the open APIs for the real-time visualisation of multi-modal sensory data, and its manifestation in a 360 ◦ workplace well-being application. @en

We propose a cross-modal approach for conversational well-being monitoring with a multi-sensory earable. It consists of motion, audio, and BLE models on earables. Using the IMU sensor, the microphone, and BLE scanning, the models detect speaking activities, stress and emotion, and participants in the conversation, respectively. We discuss the feasibility in qualifying conversations with our purpose-built cross-modal model in an energy-efficient and privacy-preserving way. With the cross-modal model, we develop a mobile application that qualifies on-going conversations and provides personalised feedback on social well-being.@en

In this paper, we explore audio and kinetic sensing on earable devices with the commercial on-the-shelf form factor. For the study, we prototyped earbud devices with a 6-axis inertial measurement unit and a microphone. We systematically investigate the differential characteristics of the audio and inertial signals to assess their feasibility in human activity recognition. Our results demonstrate that earable devices have a superior signal-to-noise ratio under the influence of motion artefacts and are less susceptible to acoustic environment noise. We then present a set of activity primitives and corresponding signal processing pipelines to showcase the capabilities of earbud devices in converting accelerometer, gyroscope, and audio signals into the targeted human activities with a mean accuracy reaching up to 88% in varying environmental conditions.@en

In this paper, we introduce inertial signals obtained from an earable placed in the ear canal as a new compelling sensing modality for recognising two key facial expressions: Smile and frown. Borrowing principles from Facial Action Coding Systems, we first demonstrate that an inertial measurement unit of an earable can capture facial muscle deformation activated by a set of temporal microexpressions. Building on these observations, we then present three different learning schemes - shallow models with statistical features, hidden Markov model, and deep neural networks to automatically recognise smile and frown expressions from inertial signals. The experimental results show that in controlled non-conversational settings, we can identify smile and frown with high accuracy (F1 score: 0.85).@en

The increasing availability of multiple sensory devices on or near a human body has opened brand new opportunities to leverage redundant sensory signals for powerful sensing applications. For instance, personal-scale sensory inferences with motion and audio signals can be done individually on a smartphone, a smartwatch, and even an earbud - each offering unique sensor quality, model accuracy, and runtime behaviour. At execution time, however, it is incredibly challenging to assess these characteristics to select the best device for accurate and resource-efficient inferences. To this end, we look at a quality-aware collaborative sensing system that actively interplays across multiple devices and respective sensing models. It dynamically selects the best device as a function of model accuracy at any given context. We propose two complementary techniques for the runtime quality assessment. Borrowing principles from active learning, our first technique runs on three heuristic-based quality assessment functions that employ confidence, margin sampling, and entropy of models' output. Our second technique is built with a siamese neural network and acts on the premise that runtime sensing quality can be learned from historical data. Our evaluation across multiple motion and audio datasets shows that our techniques provide 12% increase in overall accuracy through dynamic device selection at the average expense of 13 mW power on each device as compared to traditional single-device approaches.@en

We explore a new variability observed in motion signals acquired from modern wearables. Wearing variability refers to the variations of the device orientation and placement across wearing events. We collect the accelerometer data on a smartwatch and an earbud and analyse how motion signals change due to the wearing variability. Our analysis shows that the wearing variability can bring an unexpected change to motion signals, not only from different users but also from different wearing sessions of the same user. We also provide empirical ranges of changes in device orientations.@en

Recent research has successfully shown brand new models with Wi- Fi signals explaining space dynamics, assessing social environments, and even tracking people's posture, gesture and emotion. However, these models are seldom used in real execution and operating environments, i.e., on residential gateways with networking tasks. In this paper, we present the first, albeit preliminary, measurement study of common Wi-Fi sensing models on a residential gateway. This investigation aims to understand the performance characteristics, resource requirements, and execution bottlenecks for Wi-Fi sensing when being used in parallel with communication tasks. Based on our findings, we propose two optimisation techniques - i) dynamic sampling and ii) dynamic planning of inference execution - for optimum Wi-Fi sensing performance without compromising the quality of communication service. The results and insights lay an empirical foundation for the development of optimisation methods and execution environments that enable sensing models to be more readily integrated into next-generation residential gateways.@en