Side-channel attacks (SCA), which use unintended leakage to retrieve a secret cryptographic key, have become more sophisticated over time. With the recent successes of machine learning (ML) and especially deep learning (DL) techniques against cryptographic implementations even in the presence of dedicated countermeasures, various methods have been utilized to construct better and less complex neural network architectures. However, this process takes significant manual effort and expertise, where new architectures are constructed by adapting existing architectures or by following some methodology and filling the gaps with experimentation. While automated neural architecture search (NAS) exists and has been applied in the image classification domain, the side-channel analysis domain requires different metrics, as the machine learning metrics can be misleading in this context. In this work, we present a NAS method based on MetaQNN, which utilizes the Q-Learning reinforcement learning (RL) algorithm to generate Convolutional Neural Networks (CNNs). We define two reward functions based on the guessing entropy (GE) metric, where one of these also rewards less complex networks. We use this NAS method to generate CNNs that rival the current state-of-the-art CNNs while reducing the complexity in terms of trainable parameters significantly. We also consider a naive ensemble, which manages to keep the combined complexity below the state of the art while improving the SCA performance. Since the goal of SCA research is to improve security, there should be a balance in research on improving attacks as opposed to research on how to improve defense mechanisms. In line with this balance, we adapt our Q-Learning based reinforcement learning neural architecture search method to generate sets of countermeasures, apply them a posteriori on existing datasets, and evaluate them against existing state-of-the-art CNNs. Since implementing countermeasures is not without its costs, we also define synthetic cost functions to countermeasures based on their parameters, and both restrict the countermeasure budget and reward unused budget. We use this method to generate cost-effective countermeasure sets capable of defeating different state-of-the-art CNNs.