Influence prediction methods based on specific diffusion models are not suitable for information spread in real social networks. In addition, influence prediction methods based on information such as structure graphs and text content are difficult to promote due to limitations in information acquisition. Aiming at the two problems faced in the research of influence prediction, the sequential and non-sequential dependencies in the information diffusion process are captured through the information cascades time-series information, and the Multi-Dependency Diffusion Attention Neural (MDDAN) network is proposed. The sequential dependency and non-sequential dependency of cascades are obtained through recurrent neural networks and attention mechanisms, respectively. At the same time, the model captures the user’s dynamic preferences through the attention mechanism because the information has a time decay characteristic. To reduce the noise interference in the information diffusion, a Cascade-based Adversarial Optimization (CAO) strategy is proposed. To prove that this strategy effectively enhances the generalization ability of cascade-based influence prediction models, we apply it to MDDAN and propose an Adversarial Multi-Dependency Diffusion Attention Neural (AMDDAN) network. Experiments on three real social network datasets show that MDDAN outperforms state-of-the-art cascade prediction models, and the addition of adversarial perturbation to AMDDAN improves the robustness of MDDAN.