Bi$_2$Se$_3$ is a widely studied 3D topological insulator having potential applications in optics, electronics, and spintronics. When the thickness of these films decrease to less than approximately 6 nm, the top and bottom surface states couple, resulting in the opening of a small gap at the Dirac point. In the 2D limit, Bi$_2$Se$_3$ may exhibit quantum spin Hall states. However, growing coalesced ultra-thin Bi$_2$Se$_3$ films with a controllable thickness and typical triangular domain morphology in the few nanometer range is challenging. Here, we explore the growth of Bi$_2$Se$_3$ films having thickness down to 4 nm on sapphire substrates using molecular beam epitaxy that were then characterized with Hall measurements, atomic force microscopy, and Raman imaging. We find that substrate pre-treatment -- growing and decomposing a few layers of \BiSe before the actual deposition -- is critical to obtaining a completely coalesced film. In addition, higher growth rates and lower substrate temperatures led to improvement in surface roughness, in contrast to what is observed for conventional epitaxy. Overall, coalesced ultra-thin Bi$_2$Se$_3$ films with lower surface roughness enables thickness-dependent studies across the transition from a 3D-topological insulator to one with gapped surface states in the 2D regime.