For the application of multiusers, the arrangement and distribution of the keys is a much concerningproblem in a cryptosystem. In this paper, we propose an optical encryption scheme with multiple usersbased on computational ghost imaging (CGI) and orthogonal modulation. The CGI encrypts the secret imageinto an intensity vector rather than a complex-valued matrix. This will bring convenience for post-processingand transmission of the ciphertext. The orthogonal vectors are taken as the address codes to distinguishusers and avoid cross-talk. Only the decryption key and the address code owned by an authorized userare matched, the secret image belonging to him/her could be extracted from the ciphertext. Therefore, thereare two security levels in the encryption scheme. The feasibility and property are verified by numericalsimulations.