The paper has investigated the metadata for annotating the video information, and put up with the framework architecture of a semantic metadata describing video information based on ontology. Firstly a domain ontology library is constructed, the video information is described by the ontology, and the video library system with architecture is developed. The paper builds a semantic ontology library for Peking Opera, and gets the conclusion that it is feasible to describing video information with semantic metadata based on ontology.