While differences in learning performance among online instructional video formats have been extensively investigated, few studies focused on the effect of instructional video formats on learners’ mental fatigue during learning. Traditional mental fatigue measurements are conducted using questionnaires, which cannot reflect changes during the online learning process. This study examined changes in oxygenated haemoglobin concentrations in the prefrontal cortex of university students during the learning process using functional near-infrared spectroscopy and constructed brain functional networks of different learning stages to explore changes and difference in mental fatigue with different instructional video formats. Moreover, questionnaires were used to explore the difference of social presence, satisfaction, and learning performance after viewing two instructional videos. Learners had higher mental fatigue with picture-in-picture than lecture capture videos. Mental fatigue significantly increased during the learning process with picture-in-picture videos and was no change with lecture capture videos. This study provides a valuable reference for choosing instructional video formats for learning.