The main objects of this paper include some degenerate and nonlocal elliptic operators which naturally arise in the conformal invariant theory of Poincar\'e-Einstein manifolds. These operators generally reflect the correspondence between the Riemannian geometry of a complete Poincar\'e-Einstein manifold and the conformal geometry of its associated conformal infinity. In this setting, we develop the quantitative differentiation theory that includes quantitative stratification for the singular set and Minkowski type estimates for the (quantitatively) stratified singular sets. All these, together with a new $\epsilon$-regularity result for degenerate/singular elliptic operators on Poincar\'e-Einstein manifolds, lead to uniform Hausdorff measure estimates for the singular sets. Furthermore, the main results in this paper provide a delicate synergy between the geometry of Poincar\'e-Einstein manifolds and the elliptic theory of associated degenerate elliptic operators.