In LiDAR SLAM, data association of sparse LiDAR scan and high computational complexity of point-to-model metrics limit real-time applications of joint optimization module like Bundle Adjustment in Visual SLAM, which leads to cumulative errors deteriorating performance. To address this drawback, in this paper, we propose LPL-SLAM with plane and line optimization for the structural environment. Since plane and line primitives are ubiquitous in man-made environments, we treat the primitives as landmarks to process scans. We introduce line and plane factors relying on line-to-line and plane-to-plane metrics to optimize keyframe poses, planes and lines. With the line and plane factors, our system avoids a large-scale optimization problem and yields accurate and consistent tracking poses. Experimental results demonstrate that LPL-SLAM outperforms the state-of-the-art algorithms and achieves real-time performance.