This paper proposes a multi-task detection system for garbage sorting based on high-order fusion of convolutional feature hierarchical representation. This system achieves the identification and localization of small target objects in complex background before using manipulator to automatically grab the sorting objects. The object recognition in complex background is a key problem which the system is trying to solve. The existing convolutional neural networks based object detection algorithms are usually designed for large target objects, lack of positioning ability for small targets, and cannot estimate the target pose changes at the same time. Aiming at the above problems, this paper, from the perspective of Hyper-Column, introduces the high order feature, which has been used in image classification into object detection. For the sake of retaining the position information, this paper employs a second order feature schemes in detection. The results of bottle sorting experiment in the garbage show that the algorithm and manipulator control method of the proposed system can efficiently achieve the garbage sorting.