Teleoperation of a robot is often necessary when the remote site is not safe for humans. Moreover, to interact with dynamic environments safely, a teleoperation method called teleimpedance, which allows the human operator to control the impedance of the robot, is used. The main drawback of this method is that the human workload may increase. This could be tackled by using autonomous impedance controllers to relieve the human operator from this added workload. However, most of the existing autonomous impedance controllers require physical contact before adjusting to unexpected environmental changes. This study presents a novel semi-autonomous impedance control method that includes a vision-based autonomous impedance controller and a voice-based impedance control interface. The first element allows the robot to adjust to the environment before contact, whereas the second element allows the human operator to interact with the impedance controller when the vision-based autonomy is performing poorly or not sufficient under the environment. The method has four modalities: (i) Perturbation rejection mode, (ii) Object property detection mode, (iii) Verbal confirmation mode, (iv) Voice control mode. To provide a proof-of-concept of the proposed method, experiments were performed on a teleoperation setup that uses a Force Dimension Sigma.7 as the slave robot, a computer mouse as the master device, and a camera device. The proposed method was analyzed with a position tracking task and contact establishing task, where changing impedance can be crucial and beneficial.