We present a single-crystal neutron diffraction study on the Shastry-Sutherland lattice system TbB$_4$ at zero magnetic field and under pulsed high magnetic fields up to 35 T applied along the crystallographic $c$ axis. While our results confirm the magnetic structures at zero-field as well as those at the half- and full-magnetization plateaus, they offer new insight into the $\frac{2}{9}$- and $\frac{1}{3}$-magnetization plateaus observed in this system. A stripe model of polarized 4-spin-plaquettes whose stripe density proportionally increases with the macroscopic magnetization is in full agreement with the neutron diffraction data. Equally well suited alternative models exist which explain the observed Bragg peaks which are inherently limited in a pulsed high magnetic field experiment. We discuss the different intensity distribution in $Q$ space which can be used to distinguish these models in future experiments.