Predicting accurate human future trajectories is of critical importance for self-driving vehicles if they are to navigate complex scenarios. Trajectories of humans are not only dependent on the humans themselves, but also the interactions with surrounding agents. Previous works mainly model interactions among agents by using a diversity of polymerization methods that integrate various learned agent states hit or miss. In this article, we propose social self-attention generative adversarial networks (Social SAGAN), which generate socially acceptable multimodal trajectory predictions. Social SAGAN incorporates a generator that predicts future trajectories of pedestrians, a discriminator that classifies trajectory predictions as real or fake, and a social self-attention mechanism that selectively refines the most interactive information and helps the overall model to capture what to pay attention to. Through extensive experiments, we demonstrate that our model achieves competitive prediction accuracy and computational complexity compared with previous state-of-the-art methods on all trajectory forecasting benchmarks.