We present a collision model for particle-particle and particle-wall interactions in interface-resolved simulations of particle-laden flows. Three types of interparticle interactions are taken into account: (1) long- and (2) short-range hydrodynamic interactions, and (3) solid-solid contact. Long-range interactions are incorporated through an efficient and second-order-accurate immersed boundary method (IBM). Short-range interactions are also partly reproduced by the IBM. However, since the IBM uses a fixed grid, a lubrication model is needed for an interparticle gap width smaller than the grid spacing. The lubrication model is based on asymptotic expansions of analytical solutions for canonical lubrication interactions between spheres in the Stokes regime. Roughness effects are incorporated by making the lubrication correction independent of the gap width for gap widths smaller than ?1% of the particle radius. This correction is applied until the particles reach solid-solid contact. To model solid-solid contact we use a variant of a linear soft-sphere collision model capable of stretching the collision time. This choice is computationally attractive because it allows us to reduce the number of time steps required for integrating the collision force accurately and is physically realistic, provided that the prescribed collision time is much smaller than the characteristic time scale of particle motion. We verified the numerical implementation of our collision model and validated it against several benchmark cases for immersed head-on particle-wall and particle-particle collisions, and oblique particle-wall collisions. The results show good agreement with experimental data.