Globally, potatoes are one of the major crops that significantly contribute to food security; hence, the field of machine learning has opened the gate for many advances in plant disease detection. For real-time agricultural applications, it has been found that real-time data processing is challenging; this is due to the limitations and constraints imposed by hardware platforms. However, such challenges can be handled by deploying simple and optimized AI models serving the need of accurate data classification while taking into consideration hardware resource limitations. Hence, the purpose of this study is to implement a customized and optimized convolutional neural network model for deployment on hardware platforms to classify both potato early blight and potato late blight diseases. Lastly, a thorough comparison between both embedded and PC simulation implementations was conducted for the three models: the implemented CNN model, VGG16, and ResNet50. Raspberry Pi3 was chosen for the embedded implementation in the intermediate stage and NVIDIA Jetson Nano was chosen for the final stage. The suggested model significantly outperformed both the VGG16 and ResNet50 CNNs, as evidenced by the inference time, number of FLOPs, and CPU data usage, with an accuracy of 95% on predicting unseen data.