In this work, a wideband system with flexible test sample sizes is proposed for retrieving the complex permittivity values of materials. It includes a well-designed testing holder and a customized machine learning-based reconstruction algorithm for the inversion. The holder is implemented on a quasi-coaxial structure consisting of three parts, i.e., a quasi-coaxial structure and two coaxial connectors (inner conductors joined together). The testing procedure consists of a simple calibration without additional assembling, a measurement without dielectric under testing (DUT), placement of the DUT, and another measurement with the DUT. This procedure suppresses system complexity and mitigates systematic errors. The reconstruction algorithm is developed based on a Gaussian process regression (GPR) machine learning technique to determine the complex permittivity of the DUT in the partially filled holder. The reconstruction results from numerical simulations and measured data with several nonmagnetic dielectric materials demonstrate the effectiveness of the proposed method.