The space of invariant affine connections on every $3$-Sasakian homogeneous manifold of dimension at least $7$ is described. In particular, the remarkable subspaces of invariant affine metric connections, and the subclass with skew-torsion, are also determined. To this aim, an explicit construction of all $3$-Sasakian homogeneous manifolds is exhibited. The unique $3$-Sasakian homogeneous manifolds which admit nontrivial Einstein with skew-torsion invariant affine connections are those of dimension $7$, that is, $\mathbb{S}^7=\mathrm{Sp} (2)/ \mathrm{Sp(1)}$, $\mathbb{R} P^7=\mathrm{Sp}(2)/ \mathrm{Sp(1)}\times \mathbb{Z}_{2}$ and the Aloff-Wallach space $\mathfrak{W}^{7}_{1,1}= \mathrm{SU}(3)/ \mathrm{U}(1)$. For $\mathbb{S}^7$ and $\mathbb{R} P^7$, the set of such connections is in one to one correspondence with two copies of the conformal linear transformation group of the Euclidean space, while it is strictly bigger for $\mathfrak{W}^{7}_{1,1}$. In addition, the set of invariant connections with totally skew-symmetric torsion whose Ricci tensor is multiple of the metric, with different factors, on the canonical vertical and horizontal distributions, is fully described on every $3$-Sasakian homogeneous manifold. An affine connection satisfying these conditions is distinguished, characterized by parallelizing all the characteristic vector fields associated to the $3$-Sasakian structure. This connection is Einstein with skew-torsion for the $7$-dimensional examples. Several results have also been adapted to the nonnecessarily homogeneous setting. In this case, the above mentioned sets of affine connections are, in general, only proper subsets satisfying the properties.