This paper presents a method for semiautonomous teleimpedance where the control is shared between the human operator and the robot. The human commands the position of the teleoperated robotic arm end-effector while the robot autonomously adjusts the impedance depending on the object with which the end-effector interacts. We developed a vision system that calculates the appropriate robot stiffness based on the detected object geometry and material and object’s relation to the environment. This system uses an RGB-D camera near the robot’s end-effector to capture different perspectives of the scene. To validate the proposed method, we conducted experiments on a teleoperation system where a Force Dimension Sigma7 haptic device was used to operate a KUKA LBR iiwa robotics arm. At the same time, the Intel RealSense D455 depth camera provided the visual input. We examined two practical tasks: engaging with bolts on a plate and polishing a stripe.