Advancements in deep learning (DL) and machine learning (ML) have improved the ability to model complex, nonlinear relationships, such as those encountered in complex material inverse problems. However, the effectiveness of these methods often depends on large datasets, which are not always available. In this study, the incorporation of domain-specific knowledge of the mechanical behaviour of material microstructures is investigated to evaluate the effect on the predictive performance of the models in data-scarce scenarios. To overcome data limitations, a two-step framework, learning latent hardening (LLH), is proposed. In the first step of LLH, a deep neural network (DNN) is employed to reconstruct full stress–strain curves from randomly selected portions of the stress–strain curves to capture the latent mechanical response of a material based on key microstructural features. In the second step of LLH, the results of the reconstructed stress–strain curves are leveraged to predict key microstructural features of porous materials. The performance of six DL and/or ML models trained with and without domain knowledge are compared: convolutional neural networks (CNNs), DNN, extreme gradient boosting (XGBoost), K-nearest neighbours (KNN), long short-term memory (LSTM) and random forest (RF). The results from the models with domain-specific information consistently achieved higher 𝑅2 values compared to models without prior knowledge. When the models did not include domain knowledge, meaningful patterns in the model result, such as the link between stress–strain behaviour and underlying microstructural changes not being recognized, while those enhanced with domain knowledge insights showed better feature selection, in which they identified key stress–strain characteristics that are most relevant for predicting microstructure. These findings reveal the critical role domain-specific knowledge can provide in guiding DL models, further highlighting the need to combine domain expertise with data-driven approaches to achieve reliable and accurate outcomes in materials science and related fields.