We design a lightweight CNN, a practical and inexpensive scheme to reconstruct the geometry of the medical target from a 2D medical sequence. We make full use of natural images with rich texture features to compensate for the low-quality problem of fewer texture conditions in medical images. Besides, we design a Non-rigid Structure from Motion(NRSfM) scheme to estimate the camera structure and pose of consecutive frames with the view synthesis as the supervisory signal. Specifically, for achieving 3D reconstruction with a better time performance, we compress the CNN and medical image 3D prediction can be achieved in the lightweight CNN with the model size of 18M. The experiment results from LITFL of the open clinical library, as well as the Synapse multi-organ segmentation dataset, show that the proposed method can reconstruct the accurate geometry with better time performance.