Given the substantial economic repercussions resulting from smart contract security vulnerabilities, the detection and prevention of such vulnerabilities have emerged as critical issues warranting robust solutions. Recently, many researchers try to apply deep learning methods to the vulnerability detection task of smart contracts. However, deep learning-based approaches often focus on a singular feature of the smart contract source code, inhibiting a more comprehensive extraction of semantic and structural information embedded within the smart contract. The datasets employed for smart contract vulnerability detection are limited in size, which also restricts the learning capacity of the model, leading to suboptimal performance in identifying vulnerabilities related to smart contracts. To overcome these challenges, this paper introduces a novel vulnerability detection model centered on the fusion of semantic and structural features. These features, extracted from abstract syntax trees and contract graphs by Text Convolutional Neural Networks (TextCNN) and Temporal Message Propagation Network (TMP) respectively, are integrated to construct a classification prediction model. In addition, we added the pre-trained vector of SmartEmbed (smart contract similarity measurement model) to the training, and used the rich knowledge captured by the smart contract pre-trained word vector as the prior knowledge of our model, which can enhance the characterization of security features, and improve the performance of the vulnerability detection model, especially identifying hidden vulnerability rules when using limited labeled datasets. We conduct extensive experiments on approximately 5,000 smart contracts deployed on real-world Ethereum. Experiments prove that our method improves the detection accuracy and recall rate on the detection tasks of reentrancy vulnerability and timestamp dependence vulnerability, the accuracy rate reaches 78% and 89%, and the recall rate reaches 77% and 91% respectively, which is better than state-of-the-art methods.