Abstract: Planetary gearboxes have one or several planet gears rotating around the sun gear while revolving along their axle. This unique gear structure results in the simultaneous meshing of the planet gear with both sun and ring gears. Because of the high transmission ratio and large bearing capacity of its compact structure, planetary gearboxes have been extensively used in a variety of industrial applications. Therefore, planetary gearbox fault diagnosis is essential to ensure safe and efficient industrial manufacturing. Acoustic signal analysis provides an effective and noninvasive method for detecting potential faults in the planetary gearbox. However, the theoretical foundation of planetary gearbox fault signatures in acoustic signals is ambiguous. In this work, the planetary gearbox acoustic signal model of the resonance frequency region under the nonstationary state is structured by amplitude and frequency modulation, and the gear fault characteristics of the acoustic signals are explicitly derived. Given that resonance frequency is independent of rotational speed, the resonance frequency can be distinguished from speed-related frequency components. This lays the foundation for extracting the gear fault characteristics of the resonance frequency region. Moreover, the planetary gearbox often runs under time-varying speed conditions, and the fault frequency components are time-varying. To overcome the limitations of the traditional time–frequency analysis method in limited time–frequency resolution or cross-term interferences, the appropriate time–frequency analysis method is essential. In this work, the high-order synchrosqueezing transform is exploited to identify the time-varying fault characteristics of the planetary gearbox acoustic signal. Owing to the step of squeezing the energy distributed along instantaneous frequency in frequency direction, time–frequency representation by synchrosqueezing transform achieves a high time–frequency resolution. The high-order interpretation of instantaneous frequency further improves the capability to capture the time–frequency details. The acoustic signal model and corresponding fault characteristics of the planetary gearbox in the resonance frequency region are verified by both numerical simulations and laboratory experiments. The gear defect within the planetary gearbox is successfully diagnosed via the high-order synchrosqueezing transform.