Automatic microorganism detection, segmentation, and identification are essential to speeding up research in parasitology, biological treatment processes, and environment quality evaluation. Traditional methods based on microscopic images aim to find the differences in morphological features among the target species, such as outer shape and local features. This research focuses on species that share similar round shapes and seriously affect human health. Previous works show that segmentation is an essential step in improving detection accuracy. However, preparing segmentation ground truth is labor-intensive, especially in large datasets. For the dataset with no segmentation ground truths, we first generate pseudo-segmentation ground truths for training by applying a pre-trained segmentation network on a smaller dataset. We propose Attention-driven RetinaNet to detect, segment, and identify microorganisms in the microscopic images even when the training datasets have no annotated segmentation. The attention mechanism is applied to refine the incorrect pseudo-segmentation ground truths via Guided-attention. Self-attention is applied to select the essential part of the microorganism to improve detection performance. Experiments on the IEEE Parasitic Egg Detection and Classification in Microscopic Images Competition dataset show that the proposed method achieves 0.82 in mAP and outperforms other object detection methods.