Manganese (Mn) contamination in soil is a serious problem in China, especially around mining area. Camellia oleifera (C. oleifera) is one of the four major woody oil plants in the world and has become an important ecological and economic resource. Different P fertilization levels on the morphological and physiological characteristics of C. oleifera in two soils with different degrees of Mn contamination were evaluated. C. oleifera saplings were planted individually in plastic pots that were filled with 5.0 kg of the two types of Mn-contaminated soil. P fertilizer was applied at the rates of 0 (control), 100, 200 and 400 mg P kg−1 at the beginning of the pot experiment. Ninety days after transplantation, C. oleifera were harvested, and soils (on root surface within 1–2 mm) were collected to analyse. Compared to the control, P fertilization significantly increased the content of photosynthetic pigments (p < 0.05) and affected the growth of C. oleifera in both soils. In the soils from mining area (high contamination level), low level of P fertilizer (100 mg kg−1) significantly increased the antioxidant enzyme content in plant (p < 0.05) and had appropriate nonenzymatic antioxidant activities. In recovery area soil, high level of P fertilizer (400 mg kg−1) could significantly increase the antioxidant enzyme content in plant (p < 0.05), which reduced the toxicity of high Mn concentration to plant. The recommended P fertilization levels for maintaining the biomass of plant while increasing Mn accumulation in mining area and recovery area soils were 100 and 400 mg kg−1, respectively. At these levels, the efficiency of Mn phytoremediation by C. oleifera can be further enhanced.