This paper describes our work on the problem of reconstructing the original visual appearance of underpaintings (paintings that have been painted over and are now covered by a new surface painting) from noninvasive X-ray fluorescence imaging data of their canvases. This recently-developed imaging technique yields data revealing the concentrations of various chemical elements at each spatial location across the canvas. These concentrations in turn result from pigments present in both the surface painting and the underpainting beneath. Reconstructing a visual image of the underpainting from this data involves repairing acquisition artifacts in the dataset, underdetermined source separation into surface and underpainting features, identification and inpainting of areas of information loss, and finally estimation of the original paint colors from the chemical element data. We will describe methods we have developed to address each of these stages of underpainting recovery and show results on lost underpaintings.