In neutron radiography, the gamma ray associated with the neutron beam produced by the neutron source has a strong penetration ability. Some $\gamma$ rays pass through the detected object together with neutrons, and then hit the neutron conversion screen and participate in the imaging process of the object. And some $\gamma$ rays go directly through the shield to the detection surface of CCD camera. These two kinds of Y rays will cause a large number of random high brightness gamma white spots in the neutron images, which greatly interferes with the neutron imaging non-destructive detection and the subsequent quantitative analysis. Thus, eliminating gamma white spot noise in the neutron images is of great significance to the application and development of neutron radiography system. Traditional noise removal algorithms are difficult to remove the noise completely, and they may destroy the image texture information. In this paper, an improved self-adaptive weighted median filtering algorithm was proposed, which realized the dynamic detection of image noise and the weighted median filtering algorithm of self-adaptive window expansion. The weight values and spatial position of pixels in the filtering process were stored in the spiral data structure, which reduced the computational redundancy. The experimental results showed that the proposed algorithm could effectively remove the high brightness gamma white spot noise in neutron images and protect the image details. In addition, the algorithm can also be used in the application of high intensity impulse noise removal, with good stability and universality.