Bone cutting tasks in spinal surgery require advanced skills to operate a handheld drill device. Since there is a risk to injure the spinal cord, both of support for surgeons and enhancement of surgeon’s skills are important for safer surgery. Therefore, orthopedic surgical robots and devices have been studied and developed. In this study, a handheld haptic drill simulator using a visual servoing system is developed. A stereo camera is used to acquire the position and posture of a handheld drill device used as a haptic interface. Voxel representation is applied to virtual objects in environment simulation. Axial cutting force is fed back to the operator via the haptic interface, while virtual environment images are presented by a two-dimensional display. A small virtual object composed of voxels is cut by the drill device in the experiment. The feasibility of the handheld haptic drill simulator and the applicability of the handheld haptic drill devices are confirmed by the experimental results.