Trajectory planning for autonomous driving is challenging in unstructured scenarios such as mining sites. Existing studies mainly resort to a heuristic search-based planner to find a feasible trajectory. In the case of the narrow area, the heuristic function of the planner suffers from over-expansion problems, which may result in heavy computation burden and memory usage as well as potential failure. Therefore, a novel trajectory planning approach, namely learning and optimization-based trajectory planning (LOTP), is proposed, which is featured by a hierarchical structure consisting of two modules: (1) path searching, (2) speed profile generation. Firstly, a path searching method based on deep learning and Monte-Carlo tree search is proposed to generate a coarse path connecting starting and terminal points. Then, the path is smoothed using path optimization and provided to the speed-planning module as the reference. Next, a speed planning method based on quadratic optimization is developed, which allows to seek maximum driving comfort and energy saving. Last, extensive simulation experiments were conducted in the real environment of mining sites. The results verified that LOTP enhances the computational efficiency and success rate of path planning and helps generate an optimal speed profile. Furthermore, LOTP exhibits desirable potential for the practical application of autonomous driving at mining sites. Source implementation will be released as an open-source code.