Abstract In this paper, the authors apply the feature pyramid network (FPN) to the single‐stage anchor‐free object detection algorithm CenterNet, and the effectiveness of the multi‐level feature fusion of FPN for the object detection algorithm is proved by experiments. However, multi‐level feature fusion leads to an increase in computational cost. In this regard, this paper proposes an object detection algorithm, called DDE‐Net, that does not use multi‐level feature fusion and only uses single‐level feature for optimization. The key component in it: the dense dilated encoder, which encourages dense information exchange of features between different spatial scales. This paper presents extensive experiments, and DDE‐Net shows strong performance compared to that of other popular models on the PASCAL VOC and on the COCO2017 dataset. On the COCO2017 dataset, the authors’ DDE‐Net achieves comparable results with its feature pyramids counterpart RetinaNet, while applying the same backbone with smaller params and GFLOPs than RetinaNet. With an image size of 512 × 512, DDE‐Net achieves 37.3 AP running at 81 fps on 2080 Ti.