Speech and music have structured rhythms. We characterized spoken and musical rhythms using the modulation spectrum. The speech modulation spectrum is highly consistent across languages. A different, but similarly consistent modulation spectrum is seen for Western music. Peaks in the speech and music modulation spectra are well separated, reflecting distinct time scales in speech and music rhythms.