In recent years, as the rise of edge intelligence hand tracking applications have emerged in various IoT systems and applications, such as human-computer interaction, sign language translation, and motion rehabilitation, etc. Due to the hardware and energy limitation of embedded IoT devices, hand tracking is difficult to achieve real-time continuous operation. To achieve this goal, in this paper we propose a set of optimization methods such as space-based error loss and Gaussian process constraints to design a lightweight hand tracking model with low storage overhead and high recognition efficiency. We design and implement the system, and the effectiveness of our proposed methods is verified in the evaluation comparing with the state-of-the-art.