An efficient and effective unsupervised single moving object detection framework is presented in this paper. Given the sparsely labelled trajectory points, we adopt a coarse-to-fine strategy to detect and segment the foreground from the background. The superpixel level coarse segmentation reduces the complexity of subsequent processing, and the pixel level refinement improves the segmentation accuracy. A distance measurement is devised in the coarse segmentation stage to measure the similarities between generated superpixels, which can then be used for clustering. Moreover, a Quadmap is introduced to facilitate the refinement in the fine segmentation stage. According to the experiments, our algorithm is effective and efficient, and favorable results can be achieved compared with state-of-the-art methods.