While the discriminative performance and classification benchmarks capture most of the attention, the vast amount of data required to train state-of-the-art models is often overlooked. Inductive transfer learning suggests shifting a model between different data domains and hence introduces versatility. In this work, we (1) thoroughly analyse the ability of text classification models to adapt to transfer learning tasks, whether they are specifically designed for it or not. We directly compare the Transformer Model with an attention-based bi-directional LSTM and naive Logistic Regression as a baseline. (2) Exploiting semantic embeddings, we quantify the domain shift between different classes and predict the expected model transferability and performance. (3) Drawing from our extensive analysis, we experimentally modify the vocabulary size, layer freezing and learning rates, pursing the goal to improve eligibility for transfer learning. Our study reveals that simplistic models may be advantageous in easy transfer learning tasks due to faster convergence. An inter-active, online colaboratory notebook that allows reproducing all results, is available here 1 1 https://colab.research.google.com/drive/18CxphAH1Ym4T5dySUIvjjl-b2gGiti4d.