We report numerical simulations of assisting and opposing mixed convection in a side-heated, side-cooled cavity packed with relatively large solid spheres. The mixed convection is generated by imposing a movement on the isothermal vertical walls, either in or opposite to the direction of natural convection flow. For a fluid Prandtl number of 5.4 and fluid Rayleigh numbers of 10⁶ and 10⁷, we varied the modified Richardson number from 0.025 to 500. As in fluids-only mixed convection, we find that the mutual interaction between forced and natural convection, leading to a relative heat transfer enhancement in assisting - and a relative heat transfer suppression in opposing - mixed convection, is most prominent at a Richardson number of approximately one, when the Richardson number is modified with the Darcy number Da and the Forchheimer coefficient C_f = 0.1 as Ri_m = Ri × Da^0.5/C_f. We focus on local flow and heat transfer variations in order to explain differences in local and average heat transfer between a coarse grained and fine grained (Darcy-type) porous medium, at equal porosity and permeability. We found that the ratio between the thermal boundary layer thickness at the isothermal walls and the average pore size plays an important role in the effect that the grain and pore size have on the heat transfer. When this ratio is relatively large, the thermal boundary layer is locally disturbed by the solid objects and these objects cause local velocities and flow recirculation perpendicular to the walls, resulting in significant differences in the wall-averaged heat transfer. The local nature of the interactions between flow and solid objects cannot be captured by a volume averaged approach, such as a Darcy model.