Side-channel attacks leverage the unintentional leakage of information that indirectly relates to cryptographic secrets such as encryption keys. Previous settings would involve an attacker conducting some manual-statistical analysis to exploit this data and retrieve sensitive information from the target. With the adoption of deep learning techniques, side-channel attacks have become more powerful and require less manual analysis; hence, approaches involving deep learning have become the de facto standard for side-channel analysis. Especially the convolutional neural network has been highly effective in bypassing side-channel-specific countermeasures. The research surrounding the application of deep learning in the side-channel domain has so far primarily focused on either introducing architectures that perform well on specific datasets, data-preprocessing techniques, or the assessment of model output. Only a few attempts have been made toward the methodology aspect involving the generation of convolutional neural network architectures. The negligence of this part lends itself to the challenge of interpreting the model’s decision-making process and the high number of tunable parameters and design choices. In this work, we assess the architectural components and the hyper-parameters encountered when constructing a CNN and attempt to determine the steps of a conceivable methodology for building well-performing CNNs in the side-channel domain regardless of the dataset used.