Designing efficient testing for Autonomous Driving Systems (ADS) presents a crucial challenge in ensuring their quality. Thorough tests are the keys to their successful deployment in real-world scenarios. However, real-world testing for ADS is usually expensive and hard to scale. Meanwhile, simulation-based testing has gained popularity due to its flexibility and cost-effectiveness. Unfortunately, there exist certain limitations in simulation-based testing. The inherent hardware uncertainty typically impacts the robustness of ADS in real-world contexts. Nevertheless, current approaches often neglect this crucial factor and assume the hardware is perfect. To bridge this gap, we initiate an early step and design a test pipeline for simulation-based testing with possible natural deviations. We validate the effectiveness of our testing pipeline on a multi-model ADS in 11 typical scenarios, each containing more than 60 deviations. Through experiments, we identify that both AI models and traditional software systems of ADS are not robust against natural deviation.