Physical distancing is a vital health measure to prevent pathogens from spreading during pandemics. Many factors, including personality characteristics, may influence the ability to adhere to these mitigation measures. However, no longitudinal research has studied temperament dimensions influencing physical distancing. Therefore, we aimed to investigate if childhood temperament dimensions as predictors of later difficulty in physical distancing during the COVID-19 pandemic. This study used data from the Brazilian High-Risk Cohort Study (BHRC). Five temperament dimensions were assessed at baseline with the self-report version of the Early Temperament Questionnaire-Revised Version (EATQ-R): effortful control (EC), fear, frustration, shyness, and surgency. In addition, the Coronavirus Health Impact Survey (CRISIS) Youth Self-Report assessed the difficulty in physical distancing during the COVID-19 pandemic, ten years after baseline. The factor structure of EATQ-R was generated by confirmatory factor analysis, and logistic regression models were used to determine temperament associations with difficulty distancing. Effortful control dimension emerged as a factor associated with lower levels of perceived difficulty in adhering to physical distancing (OR = 0.823, 95% CI: 0.691–0.979, p