The drag coefficient and Magnus coefficient are different along with the geometric parameters of target object when calculating and simulating the trajectory. Generally, the coefficients are measured in wind-tunnel via repeated experiments, and only for object which have the same shape and geometric parameters. This paper proposes a numerical iterative algorithm based on hierarchical optimization method. First, all feasible solutions are calculated with the cost function of carry distance. Then a gradient cost function was added to get the optimal coefficient. In the experiment section, golf projectile motion was employed to verify the algorithm. The result shows that the algorithm is low cost and low error, and just relies on the carry distance of different initial velocity, is more versatility in application.