Although the Gaussian-noise assumption is widely adopted in the study of qubit decoherence, non-Gaussian noise sources, especially the strong discrete fluctuators, have been detected in many qubits. It remains an important task to further understand and mitigate the distinctive decoherence effect of the non-Gaussian noise. Here, we study the qubit dephasing caused by the non-Gaussian fluctuators, and predict a symmetry-breaking effect that is unique to the non-Gaussian noise. This broken symmetry results in an experimentally measurable mismatch between the extremum points of the dephasing rate and qubit frequency, which demands extra carefulness in characterizing the noise and locating the optimal working point. To further enhance the coherence time at the sweet spot, we propose to suppress the second-order derivative of the qubit frequency by the Floquet engineering. Our simulation with a heavy fluxonium shows an order of magnitude improvement of the dephasing time, even after including the noise introduced by the drive.