This study evaluates how the explainer for a Graph Neural Network creates explanations for chemical property prediction tasks. Explanations are masks over input molecules that indicate the importance of atoms and bonds toward the model output. Although these explainers have been evaluated for accuracy, no information exists on how faithful they are to the model (faithfulness), or how closely they correspond to human rationale (plausibility). Using explainability metrics to measure this, the per formance of the explainer is evaluated on different subsets based on the presence of benzene rings and halogens respectively, and on molecular weight. This study reveals that benzene rings influence the plausibility performance of the explainer, showing that performance is better at higher thresholds but worse at lower thresholds. Molecular weight and the presence of halogens do have no impact on plausibility. The ratio of positive samples in a set is shown to influence the metrics used for faithfulness. To ac
curately evaluate the faithfulness of different subsets, they should be changed to have equal positive rates or different metrics should be used. This research can be used as a starting point to research the influence of dataset properties on explainer performance. This is useful to create better explainers, leading to better acceptance of these models.