A novel hybrid particles Srp@FeO/OA, composed of phyllosilicate Serpentine (Srp), magnetic FeO and oleic acid (OA), has been explored via a two-step process. Then the as-prepared Srp@FeO/OA particles were firstly mixed with bismaleimide resin (BMI) to constructe a series of Srp@FeO/OA/BMI composites, the mechanical properties, tribological properties and thermal stability of the Srp@FeO/OA/BMI composites are subsequently investigated. The characterization results indicate that the 0.3 wt% Srp@FeO/OA/BMI composite shows the maximum impact strength (19.0 kJ·m) and minimum friction coefficient (0.21), higher 52.7% and lower 55% than those of the neat BMI resin, respectively. The significantly enhanced toughness and tribological performance of the Srp@FeO/OA/BMI composites are mainly due to the increase of the free volume and the uniformly distribution of Srp@FeO/OA, as well as the good interfacial adhesion between BMI matrix and Srp@FeO/OA particles. [ABSTRACT FROM AUTHOR]