Weaving is a highly automated industrial process. Due to small inaccuracies during the production process, different types of weave defects can occur, by which the quality of the produced fabric is heavily impaired. The defects can diminish the selling price by up to 50%. Current automated visual defect detection systems need to be adjusted by a trained operator to every new fabric, making them impractical for industrial use. We present a novel automated visual defect detection framework which localizes and tracks yarns in new and unseen fabrics without the need for tedious settings, and which consecutively detects anomalies. The detection of weave defects is based on three consecutive steps, (1) the identification of single weft and warp float-points with fully convolutional networks, (2) the tracking of single yarns based on a set of rules, and finally (3) the recognition of defects using statistical analysis.