Stoner and Heisenberg excitations in magnetic materials are inherently different. The former involves an effective reduction of the exchange splitting, whereas the latter comprises excitation of spin-waves. In this work, we test the impact of these two excitations in the hybrid Stoner-Heisenberg system FePd. We present a microscopic picture of ultrafast demagnetization dynamics in this alloy, which represents both components of strong local exchange splitting in Fe, and induced polarization in Pd. We identify spin-orbit coupling and optical inter-site spin transfer as the two dominant factors for demagnetization at ultrashort timescales. By tuning the external laser pulse, the extrinsic inter-site spin transfer can be manipulated for site selective demagnetization on femtosecond time scales providing the fastest way for optical and selective control of the magnetization dynamics in alloys. Remarkably, the drastic difference in origin of the magnetic moment of the Fe and Pd species is not deciding the initial magnetization dynamics in this alloy.