Automobiles continue to become more autonomous and connected as increasingly integrating with information technology. Meanwhile, this advance also comes with a higher risk of various security violations on vehicles. In this paper, we study how to detect attacks on autonomous vehicles, and specially focus on physical invariant-based attack detection. A physical invariant (PI) is defined as a property that a physical system always holds, i.e., the evolution of system states (usually measured by sensors) follows immutable physical laws. We first discuss existing research efforts of PI-based attack detection and classify them according to the knowledge of physical invariants and sensor redundancy. Then, we point out several critical challenges on attack detection research efforts including data sets, benchmark and testbeds, and evaluation metrics. Finally, we highlight open problems that offer promising research opportunities.