The sawtooth lattice shares some structural similarities with the kagome lattice and may attract renewed research interest. Here, we report a comprehensive study on the physical properties of Fe$_2$SiSe$_4$, an unexplored member in the olivine chalcogenides with the sawtooth lattice of Fe. Our results show that Fe$_2$SiSe$_4$ is a magnetic semiconductor with band gap of 0.66~eV. It first undergoes an antiferromagnetic transition at T$_{m1}$=110~K, then an ferrimagnetic-like one at T$_{m2}$=50~K and finally a magnetic transition at T$_{m3}$=25~K which is likely driven by the thermal populations of spin-orbit manifold on the Fe site. Neutron diffraction analysis reveals a non-collinear antiferromagnetic structure with propagation vector $\mathbf{q_1}$=(0,0,0) at T$_{m2}$Comment: 8 pages, 7 figures