The Capacitated and Time-Constrained Vehicle Routing Problem (CTCVRP) is regarded as a complex but essential, optimization mission in logistics and transportation systems. In this paper, we propose a novel approach to use deep reinforcement learning to solve the CTCVRP in an e-fulfilment center environment. Our approach aims to deal with both capacity and time constraints, ensuring optimal resource allocation and timely deliveries. Deep reinforcement learning algorithms are developed in Python environment to guide the learning agent towards optimal decisions while satisfying constraints. Experimental evaluations on benchmarking instances demonstrate the viability and effectiveness of our approach, surpassing state-of-the-art techniques in terms of solution quality and computational efficiency. The contributions of this work include a reinforcement learning formulation for CTCVRP, a deep reinforcement learning-based approach and experimental analysis. This research provides a scalable and adaptable solution for solving capacitated and time-constrained vehicle routing problems with high practicality in a real-life environment.