Handling textiles is not only a semantic experience, but also an emotional one. Whilst handling a textile is crucial for its appreciation and understanding, this channel is still little explored in the digital realm, where focus has been given to the haptic feedback aspect of handling. In this paper, we discuss the importance of touch behaviour in interactive digital handling to allow people to explore, emotionally engage with and understand textile properties. We build on our findings from previous studies, where we investigated how people handle fabrics in real-life situations and more generally how their touch behaviour may affect the experience, relating it to literature from textile, HCI, embodied cognition and embodied affect to discuss how current technology should develop to provide a more realistic touch experience. Additionally, we consider how crowd sourcing of the textile experience could be extended by taking into account non-verbal expressions of textile-handling experience. We show that further knowledge is needed to design interactive technology that supports active and unconstrained touch, as well as the affective aspects of experience.