The orbital angular momentum (OAM) of vortex beams offers a new degree for information encoding, which has been applied to optical communications. OAM measurement is essential for these applications, and has been realized in free space by several methods. However, these methods are inapplicable to estimate the OAM of vortex beams directly from the speckle patterns in the exit end of a multimode fiber (MMF). To tackle this issue, we design a convolutional neural network (CNN) to realize 100% accuracy recognition of two orthogonally polarized OAM modes from speckle patterns. Moreover, we demonstrate that even when the speckle patterns are cropped to only 1/64 of the original patterns, the recognition accuracy of the designed neural network is still higher than 98%. We also study the recognition accuracy of cropped speckles in different areas of speckle patterns to verify the feasibility of OAM recognition after cropping. The results demonstrate that recognizing the OAMs of two orthogonally polarized vortex beams from only a portion of speckle patterns in the exit end of an MMF is feasible, offering the potential to construct a 1 × N data transmission scheme.