The increasing demand of mobile Internet and the Internet of Things poses challenging requirements for 5G wireless communications, such as high spectral efficiency and massive connectivity. For the promising technology of non-orthogonal multiple access (NOMA), which serves as the standard for future 5G and is different from conventional orthogonal multiple access (OMA) technologies, it can accommodate much more users via non-orthogonal resource allocation. In this thesis, the performance, which in terms of rate and outage, of NOMA for cooperative relay (CR) system are discussed. Firstly, three convention works of cooperative NOMA system are studied which are the cooperative NOMA for downlink, the selective-DF NOMA and the cooperative NOMA with simultaneous wireless information and power transfer (SWIPT). And then, a simple CR system with one source node, one relay and one destination for NOMA named the novel receiver design for CR-NOMA system is proposed which improves the performance in terms of the date rate and the outage probability by using the maximum ratio combining (MRC). Since the MRC requires the symbol with less power to be decoded first, which is not strictly following the NOMA principle and limited the outage probability, a two-stage power allocation NOMA is investigated. Unlike the existing works, where the relay forwards the latter decoded symbol only, our proposed scheme allows the relay to forward two symbols simultaneously with a different power allocation from that employed at the source. In addition, the destination will not immediately decode the received signal from the source until it receives the superposition coded signal from the relay. At the second phase, the destination jointly decodes the information symbols by using a simple linear combination. Finally, two more general and challenger works, two-way relay networks (TWRN) and multi relay systems, are discussed. In these systems, different from conventional works, the performance are relatively improved for two different kinds of the NOMA cases in which one is the immediately decode and the other one is the jointly decode. Furthermore, since that the power allocations in the multi relay systems are quite involved, it is possible to find the relations between the source power allocation factors and the relay power allocation factors. By this way, the proposed sub-optimal strategy is practical and simple which has the closed performance in the term of the date rate/capacity compared to the optimal solution at the high signal-to-noise ratio (SNR).