The ultrafast optical dynamics of plasmon-quantum emitter systems have rarely been explored. Here, the authors study those dynamics by interferometrically probing nonlinear optical emission from porous gold nanosponges with zinc oxide emitters, allowing sum-frequency generation enhancement from individual hot spots.