In recent years, federated learning is the most commonly used framework for collaborative training under the protection of privacy, and has been successfully applied to smart healthcare. The distribution of data in the federated network is usually non-independent and identically distributed (non-IID) and imbalanced, which worsens the performance and increases the gap between the local and global models. Additionally, the similarity of the medical images makes it challenging to identify. In this paper, we propose a dual aggregated federated learning with depthwise separable convolution for smart healthcare. Specifically, a diagnostic network based on depthwise separable convolution is designed, and the residual connection is introduced, which can make full use of the feature information in the medical image and improve the accuracy of disease diagnosis. Meanwhile, we design a dual federated aggregation algorithm to reduce the impact of parameter differences on multi-client model federated aggregation and improve the performance of the global model. Finally, the experimental results illustrate that our proposed algorithm achieves significant performance advantages compared with other existing methods.