The direction of in situ stress is an important factor influencing the fracture characteristics of the surrounding rock of underground tunnels. In this study, to investigate the influence of the principal stress direction on the fracture characteristics of underground tunnels and the formation mechanism of spalling failure around hard rock tunnels, a series of uniaxial compression tests were first carried out on granite specimens containing an inverted U-shaped cavity. Then, a two-dimensional numerical model was constructed by particle flow code (PFC) to further investigate the influence of the principal stress direction and aspect ratio of the cavity on the fracture characteristics of the specimens under biaxial compression. The results show that the uniaxial compressive strength and elastic modulus of granite specimens with an inverted U-shaped cavity are 23.97% and 6.66% lower than those of the intact specimen, and the maximum value of the number of microcracks is 36.4% larger than the minimum value for different loading directions when the aspect ratio of the cavity is 2. When the direction of the maximum principal stress is parallel to or at a small angle to the straight sidewall of the cavity, obvious spalling failure occurs at the cavity’s straight sidewall, indicating that stronger support structures should be applied to the straight sidewalls of underground tunnels in this situation. Moreover, the specimens containing a cavity with a larger aspect ratio show more obvious spalling failure and experience more severe damage. The results of this study can provide not only insight into the spalling failure mechanism but also a scientific basis for the design and support of underground tunnels.