Since visible light communication (VLC) could support high-speed broadband transmission and be combined with existing illumination system, it is an optional technology for multimedia broadcasting, especially for indoor scenario. Further-more, hybrid modulation could be applied in VLC system to assist data transmission or expand new functions. Researches and experiments show that conventional on-off key (OOK) and orthogonal frequency division multiplexing (OFDM) could be implemented successfully in VLC system respectively, thus it seems that a direct adding of OOK and OFDM signal is a feasible method for hybrid modulation. However, nonlinearity of electro-optical converting in VLC transmitter makes the relationship between transmission performance and amplitude of OOK or OFDM signal different from that in radio frequency (RF) system. In this paper, a mathematical model is built to analyze its performance and predict the bit error rate (BER) using OOK and OFDM signal amplitude in a given VLC system in order to find proper parameters for higher transmission quality. Simulation is also implemented to prove the validity and accuracy of the model.