The improvement of network performance needs to optimize network topology (that is adding network link) and expand link capacity. This paper comprehensively considers delay constraints, demand constraints and link capacity constraints and researches the joint optimization problem of network topology and link capacity under complex constraints. A new heuristic algorithm is proposed, called greedy-mutation genetic algorithm. Our method, based on a conventional genetic algorithm, conducts mutations on original solutions based on various constraints and the greedy algorithm, therefore, it can find better optimized solutions and fulfill all the constraints better. We applied the greedy-mutation genetic algorithm into two public backbone networks’ topology optimization and link expansion cases. Our results show that the proposed algorithm can effectively decrease the total cost of network optimization. The proposed method can also be applied in the network design and planning of wide area networks under complicated constraints, which is helpful to network operators.