In Recirculating Aquaculture System (RAS), feeding frequency is an important factor affecting fish growth, and precise feeding according to the state of fish is the key to improving the aquaculture efficiency. The current problems in detecting the feeding frequency of fish include low efficiency, high technical requirements, and the influenced by the aquaculture environment. In this paper, we adopt a computer vision method to calculate feeding frequency to achieve objective with high accuracy. Firstly, we process the video of the fish according to the inter-frame difference method to obtain the image of the feeding state of the fish. Then we propose a modified VGG16 model to determine the feeding state of the fish, transform it into a 0–1 classification problem and calculate the feeding frequency of the fish. The feeding frequency and the growth status of the fish are then used to develop an intelligent feeding strategy to improve the growth rate of the fish and the conversion rate of the bait. Tests have shown that the accuracy of identifying feeding state by the modified VGG16 model can reach 92.4%. The method has a positive effect on the development of recirculating aquaculture system.