The high-precision scrap sorting for effective metal recycling can bring substantial environmental and economic benefits. This article presents a magnetic image sensor that can help to identify the ferrous contaminants inside nonferrous scraps of large sizes. First, the concept and the theory for detecting ferrous contaminants are described. In particular, an inversion algorithm is proposed to characterize the size and position of ferrous contaminants inside the main scrap bodies. Then, based on computed and measured results, the feasibility of sensor design using either 1-D Hall arrays or 1-D pickup coils is demonstrated. Finally, methods are suggested to minimize disturbing signals from very large nonferrous pieces passing through the slightly uneven magnetic field. The obtained findings in this study may apply not only to nonferrous scraps but many other materials of which the mass ratio of the ferrous contaminant to the main material is small.