In this paper, we investigate the novel nonlinear vector network analyzer (NVNA) setup for testing millimeter-wave antenna systems using spectrally-overlapped in-band full-duplex (IBFD) stimuli. Different from the commercial NVNAs, the experimental setup is established with user-defined digitally-modulated test stimuli and multisine phase reference, where two periodical test stimuli are designed with identical carrier frequency and the same tone spacing, and are simultaneously applied to simulate the realistic IBFD scenario. According to the experimental results, the components of received signals can be extracted for error vector magnitude (EVM) analysis from the NVNA raw measurements, which are the vector superposition of received and reflected contributions.