We report a phase instability in oxygen-plasma-assisted molecular beam epitaxy of Fe3O4 films on sapphire (0001) substrates. Under a wide range of growth conditions, Fe3O4(111) films phase separate, on a nanometer length scale, into Fe3O4, FeO and metallic Fe, which is attributed to formation of the thermodynamically unstable phase FeO in the initial stages of (111) growth. In contrast, Fe3O4(001) films, grown simultaneously on MgO(001) substrates, do not exhibit this phase instability. We specify growth conditions for which single-phase, epitaxial Fe3O4(111) films can be grown by plasma-assisted molecular beam epitaxy or by reactive evaporation of Fe in molecular oxygen. Film orientation and phase separation strongly influence magnetic properties. Single-phase Fe3O4(111) films are much more difficult to magnetize than Fe3O4(001) films and phase separation makes the films even more difficult to magnetize.