This longitudinal study aimed to clarify the longstanding controversy over whether variations in paraspinal muscle morphology (e.g., size, composition and asymmetry) are predictors of low back pain (LBP). A sample of 99 Finnish men were included in this population-based longitudinal study. Data were collected through a structured interview, physical examination and magnetic resonance imaging (MRI). Baseline measurements of the lumbar multifidus and erector spinae muscles were obtained from T2-weighted axial images at L3–L4 and L5–S1, and interview data were obtained at baseline, 1- and 15-year follow-ups. Few of the paraspinal muscle parameters investigated were predictors of change in LBP frequency, intensity or sciatica at 1- and 15-year follow-ups in the population-based sample, and findings were not consistent across muscles and spinal levels. However, greater multifidus and erector spinae fatty infiltration at L5–S1 was associated with a higher risk of having continued, frequent, persistent LBP at 1-year follow-up. None of the relationships observed was confounded by body mass index or the amount of physical activity at work or leisure. This longitudinal study provided evidence that variations in paraspinal muscle morphology on MRI have a limited, if not uncertain, role in the short- and long-term predictions of LBP in men.