We report the optical behaviors of the Fe-doped CaMnO3 family of compounds at low doping concentrations x ≤ 5%. The study aims at assisting the evaluation of the competition between ferroand antiferromagnetic orderings, which is believed to be a cause of many interesting properties of this class of compounds, including the magnetization reversal effect recently discovered. The structural characterization showed a predominant orthorhombic phase with slightly increased cell constants due to doping. The Raman spectra revealed changes associated with the Mn sites, and the IR absorption spectrum showed a characteristic Fe band at 1.2 eV, which should be accompanied by a change of spin. The analysis of the magnetization data allowed us to predict that while the doping reduced the ferromagnetic coupling strength, and therefore the TC, the maximal doping concentration for the effective exchange to be zero was around 14%.