The advent of Artificial Intelligence (AI) and Internet-of-things (IoT) has led to a significant demand for edge computing and enabling Neural Network Implementation on edge devices. However, due to large MAC operations involved in the implementation of Neural networks, the traditional digital hardware based on the von-Neumann architecture is not well suited for an edge device. Computation-in-Memory (CIM) is an attractive alternative in mitigat- ing the challenges involved with traditional hardware by directly processing the data within the memory. It utilizes emerging memory devices such as Resistive Random Access Memory (RRAM) to perform in-place computations in the analogue domain, thereby, eliminating the bottlenecks associated with the constant movement of data in the von-Neumann architecture. However, the standard implementation of CIM comes with several challenges, with the pri- mary being high power consumption, which debate its implementation on an edge device in accelerating Neural network computation. Thus, this work proposes a novel CIM crossbar that has the potential to alleviate the challenges associated with the standard CIM cross- bar. Subsequently, an ultra-low power micro-architecture design is proposed, based on the novel CIM crossbar, that can accelerate Binary Neural Networks (BNN) and Spiking Neural Networks (SNN) with high power efficiency. The benchmark results obtained over the imple- mentation of custom-developed BNN and SNN, trained over the MNIST dataset, indicate a power reduction of 13x and 26x respectively for the proposed micro-architecture, compared to its standard CIM crossbar counterpart. In addition, the proposed micro-architecture exhibits energy savings of around 4-5x over both BNN and SNN, making it a promising alternative for accelerating neural network computation over edge devices.