Streaming made a lasting effect on the way our society consumes media in the last decade. While due to streaming the way we listen to music and podcasts has changed drastically, there are very few studies about its Quality of Experience (QoE) and possible influence factors. From video QoE studies, we know that, for example, undesirable stops of the stream (stalling events) have a significant impact on QoE. However, the way in which music and video streaming is consumed differs significantly, as music is often played in the background, and thus, the influence of stalling could be significantly different. Thus, this work evaluates the impact of stalling on music streaming QoE. Therefore, we conduct two crowdsourced user studies: In the first study, users have to rate four songs with different stalling patterns and evaluate the degree of impairments. Afterwards, we compare the ratings to the results of a lab study and show that they are highly correlated, and that crowdsourcing is a suitable way of measuring music streaming QoE. In addition, we conduct a second crowdsourcing study to investigate the influence of the user's attentiveness on QoE. Here, participants have to listen to one song with two stalling events, while one half of them had to transcribe a handwritten text with music playing in the background. The attentiveness shows no influence on the perceived streaming quality, but it shows a significant influence on the perceived quality degradation due to stalling events. Furthermore, considerably more stalling events were missed for workers who focused on the transcription. These results are an important step towards establishing new methods for investigating QoE in multimedia.