Computing hardware like field programmable gate arrays (FPGAs), microcontrollers and microprocessors can have limited compute and on-chip storage resources. This is especially true for computing hardware in Internet of Things (IoT) and low end embedded systems. With the growth in machine and deep learning, it is imperative to build intelligence in these devices. Therefore, this paper proposes exploiting weight statistics to compress floating point based weights in neural networks without any loss in accuracy. The proposed method has been implemented as an optimization pass in open source N2D2 framework. The proposed method thus does not make an assumption that the application can tolerate some accuracy loss which is the case with other methods like quantization, binary weights etc. However, it can also be considered as a further step in optimization after applying existing quantization based methods. The proposed method is able to save nearly 10% on-chip storage requirement, thus reducing the number of Block RAMs (BRAM) in case of FPGAs and the size of on-chip memory (OCM) in case of microcontrollers and microprocessors. We show that layer wise compression gives slightly better compression than global compression. This compression is traded off for execution time overhead in microcontrollers.