PAPER / ARXIV:2609.12945
Bin Lin , Bo Zhao , Boyang Wang , Boyang Zhang , Boyong Wu , Chao Yan , Chen Geng , Chen Wu , Cheng Yi , Chengli Feng , Chenglin Zhu , DanNi Wan , Daxin Jiang , Dongqing Pang , Fei Tian , Feng Tian , Future Li , Gang Yu , Guanglong Yang , Jia Peng , Jiahao Song , Jiamin Fan , Jiangjie Zhen , Jianzheng Gao , Jun Chen , Li Xie , Lifang Zhang , Lingli Ji , Liying Shi , Lun Cai , Min Xu , Na Wang , Peilin Li , Peng Yang , Pengfei Tan , Qingjian Lin , Ruijie Xiong , Runze Li , Shenghua Hu , Shi Qiu , Siqi Tu , Siyi Zhou , Tianjiao Deng , Wanying Lu , Weiming Niu , Wen Sun , WenWen Qu , Xiangyu Zhang , Xianwei Zhang , XiaoSu Su , Xing Chen , Xinyu Liu , Xuerui Yang , Yang Li , Yang Yang , Yechang Huang , Yibo Zhu , Yifan Zhang , Yiyang Xu , Yu Fu , Yu Luo , Yu Zhou , Yumang Wang , Yunzhou Ju , Yuxiang Yang , Zekai Liu , Zengwei Yao , Zhenwei Mou , Zheqi Dai , Zhiyue Wu , Zichao Zhou
RESUMO
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared $16 \times 2048$ residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at this https URL .
NO MESMO MAPA