As a novel remote sensing technique, GNSS reflectometry (GNSS-R) opens a new era of retrieving Earth surface parameters. Several studies employ the combination of deep learning and GNSS-R observable delay-Doppler maps (DDMs) to generate ocean wind speed estimation. Unlike these methods that often use convolutional neural networks (CNNs) with inductive bias, we proposed a Transformer-based model, named DDM-Former, to exploit fine-grained delay-Doppler correlation independently. Our model is evaluated on the Cyclone GNSS (CYGNSS) version 3.0 dataset and shown to outperform the other retrieval methods.