The exponential growth of the Internet of Multimedia Things (IoMT) traffic has posed a threat of service quality degradation due to the limitation of current communication, networking, and computing advances in mobile networks. In this regard, managing the Quality-of-Experience (QoE) for IoMT services is a vital challenge to meet user satisfaction. To cope with this problem, we investigate the joint optimization of video quality variation and latency in multiuser downlink rate-splitting multiple-access (RSMA) networks, especially within imperfect network conditions and state information. To accomplish this, we first formulated the joint optimization problem into a Markov decision process framework, then exploited a deep reinforcement learning approach to adaptively calculate the optimal configuration of the RSMA against environment dynamics. As a result, the proposed deep deterministic policy gradient on RSMA-based video streaming system (DDPG-RMAVS) provides QoE maintenance by minimizing video resolution reduction and latency. Extensive simulation results revealed that the proposed DDPG-RMAVS algorithm surpasses existing algorithms by achieving higher video quality, lower delay, larger buffer capacity, and limited stalling events, representing a significant breakthrough in IoMT streaming optimization.