An infiltration-reaction (IR) method can fabricate a dense MgB$_{2}$ bulk under ambient pressure, thus being a promising method for mass production. However, the IR-processed MgB$_{2}$ bulks are known to contain the unreacted Mg and B and the intermediate product, Mg$_{2}$B$_{25}$, which degrades the trapped field, $B_\mathrm{T}$, and critical current density, $J_\mathrm{c}$, of the bulks. To expedite the creation of the MgB$_{2}$ phase during the IR process, we prepared the refined B powder by ball-milling and fabricated the MgB$_{2}$ bulks by the IR method. We achieved both improved $B_\mathrm{T}$ and $J_\mathrm{c}$ for the MgB$_{2}$ bulks made from the ball-milled B powders with sub-micron size. The microstructural analysis suggested that these results originated from the refinement of MgB$_{2}$ grains and the increase of the volume fraction of MgB$_{2}$ phase. The refinement of the starting B powder is quite effective to enhance the magnetic properties for the IR-processed MgB$_{2}$ bulks.