We propose a real-time multi-step view reconstruction algorithm and we tune its implementation to a virtual teleconference application. Theoretical motivations and practical implementation issues of the algorithm are detailed. The proposed algorithm can be used to reconstruct novel views at arbitrary poses (position and orientation) in a way that is geometrically valid. The algorithm is applied to a virtual teleconference system. In this system, we show that it can provide high-quality nearby virtual views that are comparable with the real perceived view. We experimentally show that, due to the modular approach, a real-time implementation is feasible. Finally, it is proved that it is possible to seamlessly integrate the proposed view reconstruction approach with other parts of the teleconference system. This integration can speed up the virtual view reconstruction.