Identifying potential ligand binding sites on proteins is an essential step in structure-based drug design. The computational detection of binding cavities can be improved by combining previous knowledge on protein-ligand binding and deep learning algorithms. In this paper, we present VoxelProt, an efficient deep learning approach for ligand binding site detection. VoxelProt uses the proteins 3-dimensional structure to create a surface point cloud. This point cloud is combined with octree-based sparse voxels and a feature vector containing chemical and geometric information to enable prediction of binding sites via binary classification. In this work, the model systems used to test the performance of VoxelProt are proteins bound to Adenosine Di-Phosphate (ADP), a byproduct of ATP hydrolysis, the primary energy-generating process in living cells. A 5-fold cross-validation was performed on a small size dataset. Training, testing, and validation are performed on voxel data based on protein surface concavity with good prediction accuracy. We test the performance of VoxelProt against multiple metrics, including accuracy, precision, sensitivity, F-score, and AUC-ROC. In addition, we explored modifications of VoxelProt from three perspectives to find the appropriate architecture. Statistical tests are employed to compare the performance of our original VoxelProt architecture to these 3 modifications. We find that balancing input datasets to keep similar ratios of positive to negative instances does not significantly affect performance. However, the addition of atom type features improves performance, and adding more fully connected layers to the model reduces prediction performance. VoxelProt exhibits good predictive ability while requiring less memory due to sparse voxelization, showing promise as an efficient approach to identifying ligand binding sites in more general and complex problems such as in the presence of multiple ligands or during protein dynamics.