In this letter, an integrated waveform based on 4 quadrature amplitude modulation (QAM)-non-orthogonal multiple access (NOMA)-linear frequency modulation (LFM) is proposed and experimentally demonstrated in W-band integrated sensing and communication (ISAC) system. And multiple user communication and multi-target detection can be realized by using NOMA. In addition, a machine learning scheme is proposed for the ISAC system based on 4QAM-NOMA-LFM. The experimental results show that, the proposed scheme can effectively improve the performance of bit error rate (BER). The BER can be re duced to $5\times 10\,\,^{\mathbf {-4}}$ when the total aggregate rate for two users is 800Mbps. And centimeter level radar ranging is realized.