$\rm{PtSe_2}$ is a van der Waals material transitioning from an indirect bandgap semiconductor to a semimetal with increasing thickness. Its absorption threshold has been conjectured to originate from interband indirect transitions. By quantitative comparison between broadband ($0.8 - 3.0\,\rm{eV}$) optical absorption of high-quality exfoliated crystals and DFT ab-initio simulations, we prove instead that the optical absorption arises only from direct transitions. This understanding allows us to shed light on the semiconductor to semimetal transition and to explore the effect of stacking and excitons on the optical absorption.