Design of sparse finite impulse response (FIR) filter is of great importance to reduce the implementation cost. However, design of sparse FIR filter under the prescribed constraints is a highly non-convex problem. Traditional methods generally relax the non-convex design problem to a convex one, which leads the obtained solutions suboptimal. In this paper, the non-convex design problem is modeled as a combinatorial optimization problem and an algorithm based on simulated annealing (SA) is presented to solve it. At each stage of the proposed algorithm, with a fixed sparsity of the filter coefficients, SA is employed for finding the possible sparse pattern of the FIR filter that satisfies the prescribed constraints. Once the design constraints have been satisfied, the sparsity is added by one and the algorithm moves to the next stage. The algorithm successively increases the sparsity of the filter coefficients until no sparser solution could be obtained. The proposed algorithm is evaluated by two sets of examples, and better results can be achieved than other existing algorithms.