Federated learning is an advanced technique that outperforms traditional centralized approaches in machine learning. It allows participants to keep their data locally and obtain a global model through aggregation. However, this technique introduces parameter exchange processes, increasing privacy risks. In this study, we investigate two privacy concerns: protecting users’ data during training and verifying the accuracy of aggregation results from the server. To address these issues, we propose a verifiable Federated Learning aggregation scheme based on the RSA accumulator. We employ double-masking techniques for secure federated learning and to accommodate disconnected users. Using the RSA accumulator and homomorphic hash function, our scheme offers efficient correctness verification with minimal storage requirements and constant verification time, even with more users. Through experiments and analysis, we demonstrate that our scheme reduces computation and communication costs while ensuring lightweight verifiability.