Recently, adversarial perturbations have been used to reassign cost which can enhance the security of steganography, called as adversarial embedding. However, existing methods selected costs to be modified by self-defined rules which were hard to achieve the optimal security against steganalyzers. In this paper, we propose an automatic adversarial embedding scheme called RLAE (deep Reinforcement Learning-based content-adaptive Adversarial Embedding). In RLAE, an agent network utilizes a generative network which generates an embedding policy for cost reassignment automatically according to a basic steganography cost map. Then, an environment network employs a steganalyzer as an attack target that offers rewards for optimizing the agent network. To provide more comprehensive information, we design a joint reward by considering both the adversarial perturbations calculated from the environment network and noise residual signal representing image textures. Experimental results show that the security of the proposed RLAE is superior than state-of-the-art works, especially steganography with for the large payloads.