As one of the major diseases that pose a threat to human health, brain tumors' diagnosis plays a vital role in the intervention and rehabilitation process. Classification models based on deep neural networks (DNNs) have been proven effective in aiding doctors to diagnose brain tumors through MRI images. However, the injection of tiny perturbations to the images can have a significant impact on the diagnosis, indicating the vulnerability of DNNs. Therefore, research on adversarial attack methods are highly significant in the medical diagnosis field. In this paper, we combined the non-sign gradient attack method and average gradient to propose a non-sign average gradient attack method(ABINM). By performing attack experiments on brain tumor MRI images under three different classification models, it was verified that the ABINM method can achieve higher rate of attack success and the generated adversarial sample shows stronger transferability on white-box and black-box testing compared to the baseline gradient-based adversarial attack methods.