Lunar magnetism has remained an unsolved mystery since the Apollo days, with its origin debated between an internal core-dynamo or transient magnetic fields from impacts. While most investigations studying magnetic anomalies with orbital data use methodologies inherently limited by relying on available geological/geophysical context, this study overcomes these constraints with a novel approach. It implements the versatile Parker Inversion method to estimate the spatial distribution, intensity, and direction of surface magnetization. Correlative analyses of the results with impact-related processes indicate a prominent role of impacts in shaping lunar magnetism: most isolated near-side anomalies are probably magnetized ejecta from the Imbrium impact, while complex far-side and North Pole anomalies likely originate from antipodal effects of Imbrian and Nectarian-aged impacts. Magnetized material also strongly correlates with lunar swirls. Findings overall suggest that a transient origin alone is unlikely, favoring a core-dynamo as the primary magnetic field source, potentially amplified by impact events.