Various industries use towers as part of their daily operations, such as transmission towers (aka. electricity pylons), telecommunications towers and water towers. These towers re- quire regular maintenance, and before the maintenance work can be done, a preliminary survey must be conducted to determine where to work. More and more, such surveys are being conducted via drones. This work develops a detection model to help locate tower issues from the video frames of drones. However, it does not provide satisfactory performance to directly train such an object detection model with the annotated problem locations from domain experts. Therefore, we propose to improve the quality of the extracted image features with the help of another separate task which detects the various parts that are involved in the tower issues, such as bolts, nuts, washers and pins, the annotations of which can be done without the need of domain expertise. Through this multi-task learning scheme, we improved the problem detection recall from 59.6% to 71.5%, providing much more effective recommendations of potential issues for inspectors to examine further. Also, the average number of problem detections in each image is merely 5.54 so inspectors are not overwhelmed by the recommended locations.