A planetary gearbox fault feature extraction method based on improved adaptive noise complete set empirical mode decomposition (ICEEMDAN) and variational mode decomposition (VMD) methods is proposed. The signal was decomposed using ICEEMDAN, and the signal was filtered and reconstructed based on the kurtosis of the component envelope. Based on the maximum envelope spectral kurtosis as the fitness function, the sparrow search algorithm is used to adaptively optimize the parameters of VMD, and the reconstructed signal is decomposed into multiple modal components. Based on the kurtosis of the envelope spectrum of the components, select the optimal component for envelope demodulation analysis to achieve feature extraction of planetary gearbox faults. Finally, the consistency correlation coefficient of the method proposed in this paper was found to be between 0.472 3 and 0.793 6 through experiments, which is much higher than the EEMD-WTD method’s 0.088 1 to 0.286 3 and the envelope spectrum selection index’s 0.142 7 to 0.286 4.