Semantic communication is regarded as the breakthrough beyond the Shannon paradigm, which transmits the semantic information only to improve the communication efficiency significantly. This article first introduces a framework for the generalized semantic communication system, which exploits the semantic information in both the multimodal source and the wireless channel environment. Subsequently, the deep learning enabled end-to-end semantic communications and the environment semantics aided wireless communications are demonstrated through two user cases. The article is concluded with several research challenges to boost the development of such a generalized semantic communication system.