Toward Reliable and Interpretable Tactile Material Classification

Abstract

Deep learning has led to strong performance in recognizing materials from touch signals, but it is often difficult to understand which parts of those signals drive a model’s decision. This thesis studies accurate and interpretable tactile material classification on the public SENS3 dataset using three dominant cue sources: thermal transients during static contact, deformation dynamics during pressing, and friction/vibration cues during sliding. An adjective-mediated pipeline that first predicts psychophysical rating distributions and then classifies materials is compared against a direct multimodal classifier that fuses modality-specific encoders. The direct multimodal model shows strong generalization, reaching 0.896 test accuracy across seven retained material classes. To better understand the learned decision process, Integrated Gradients was used as the main explanation method, combined with Temporal Saliency Rescaling for models with a single classification output. The resulting attribution maps were then summarized into contributions at the level of signal type, interaction phase or bin, and individual measurement channel. The resulting explanations reveal class-conditional cue usage aligned with interaction structure: transient thermal phases dominate for metal-like materials, pressing dynamics contribute strongly for compliant/textile-like classes, and sliding evidence concentrates in low-force regimes where friction and vibration cues are most informative. Overall, the results demonstrate that high-performing tactile material recognition can be combined with interpretable, physically grounded attribution summaries, improving trust in model decisions for haptic interfaces and embodied systems.