Recently, body part as an intuitive movement unit has received increasing attention in skeleton-based action. However, the part-level embedding is hard to be fully exploited, especially for fine-grained actions, as the body joints are aggregated into parts. To address this problem, we propose a novel transformer-based network (IIP-Transformer). Different from previous models that rely on specially designed partition method, our proposed IIPA mechanism which incorporates joint-level (intra-part) and part-level (inter-part) interactions simultaneously is the keypoint for IIP-Transformer to fully exploit part-level data, making considerable improvements in both coarse-grained and fine-grained action recognition. Ablation studies on three typical partition methods show that IIP-Transformer is a relatively general solution for part-level data and thus we could choose the simplest hand-craft partition embedding to significantly reduce computational complexity and model size. Besides, The proposed IIP-Transformer exceeds the state-of-the-art methods with much less computational complexity on NTU RGB+D, NTU RGB+D120 and NW-UCLA datasets.