The increasing softwarization of network infrastructures introduces an important challenge for network configuration. The growth of the network configuration space as a result of new device types and the expanding inter-dependence of network service components increases the network configuration complexity. Compounding this issue, new service deployment architectures lack mechanisms to validate the impact of service configuration on network resilience. Network operators need to adopt new mechanisms to validate and verify network configuration changes, taking inspiration from popular Continuous Integration/Continuous Development (CI/CD) mechanisms. This paper introduces Network Emulation-based Automated Testing (NEAT), an automated testing framework for network configuration. NEAT allows network managers to define network topologies and tests through YAML files, to then be later executed by NEAT within realistic network topologies. Furthermore, network managers can control the fidelity of their network tests and bound the execution time of testing suites, as well as exploit parallelization of modern servers to speedup test execution.