A optimization algorithm is put forward based on the brightness and frequency domain characteristics of macro block, including the algorithm of model segmentation and the prediction algorithm of starting motion vectors(MV). The basic idea of the mode segmentation is that to the unit of 16×16 block, discrete wavelet transform to frames, the after low frequency coefficients as the research object, define the coefficients sum ∑. If ∑ is greater or equal to threshold, the corresponding areas will be divided into texture areas, or smooth areas. The current block is compared with the corresponding block in reference frames. If the result is equal, the current block will be marked to background block, or sport block, and the corresponding segmentation mode is selected at the same time. The basic idea of prediction starting motion vectors algorithm is that base on the marked sport blocks, the starting MV of current block is the median value of MV to adjacent sport blocks in initial MV prediction. The experimental results show that the efficiency of algorithm in this paper has been improved.