Intelligent recognition of events along the fiber has become an increasingly important issue for many safety monitoring to achieve fault diagnosis and anomaly warnings. The fiber signal acquisition environment is complex, there is a lot of nonstationary noise in the signal. The traditional denoising methods are usually based on the assumption of a constant noise level, which makes it difficult to adapt to different noise situations. Besides, to address the issue of feature extraction and recognition of vibration signals, many methods have been proposed, e.g., convolution neural networks and short-time Fourier transform. However, temporal order information is ignored, which leads to poor classification of certain events that are only clearly distinguishable in the temporal domain. Therefore, this paper designs an Adaptive Shrinkage Denoising and Sequence-state Learning Vibration Detector model (ASVD) that adaptively learns soft thresholds to eliminate noise-related information and capture the temporal sequence state evolution. Specifically, we embed the adaptive soft-threshold learning module into the deep framework to enhance the discriminative feature learning ability from noised signals. Then, we incorporate the sequential state encoder into the framework to capture temporal sequence state evolution. We conducted experiments on an open dataset and achieved an accuracy of 96.5%, a false positive rate of 0.7%, and a false negative rate of 3.53% averaged across the six events.