Reviser: A New Paradigm for Non-Autoregressive Text Generation via Cursor Editing Actions

Reviser proposes a revolutionary text generation method that abandons traditional left-to-right token generation. Instead, it uses cursor actions (insert, move, stop) on an editable canvas to build text. This approach outperforms diffusion baselines like SEDD and MDLM at the 300M parameter scale, addressing the error accumulation issue of autoregressive models while balancing quality and flexibility.

Mainstream large language models (e.g., GPT, Claude) use autoregressive generation, leading to early error accumulation in long texts. Non-autoregressive methods like diffusion models and masked language models allow multiple revisions but struggle to balance quality and efficiency.

Reviser defines three basic edit actions: Insert (add tokens at cursor), Move (shift cursor position), Stop (end editing). Though the final text is non-autoregressive, the edit actions are generated autoregressively—combining the stability of autoregressive models with the flexibility of non-autoregressive ones.

Reviser uses a decoder-only Transformer architecture. Inputs include current canvas state and cursor position; outputs are next edit actions. Training involves deriving reasonable edit trajectories from target texts (via shortest edit distance) and maximizing the likelihood of these trajectories.

At 300M parameters, Reviser outperforms diffusion baselines SEDD and MDLM. It is competitive with same-size autoregressive models on C4续写. Evaluation metrics include EvalPPL (perplexity), MAUVE (distribution matching), and trajectory statistics (length, modification frequency).

Reviser is suitable for interactive writing assistants, code generation/refactoring, and multi-round dialogue. Open resources: code structure (paper, src, configs, scripts, results, visualizations), 100M/300M pre-trained models on Hugging Face (sean-diab/reviser-checkpoints), and quick start commands for inference.

Current limitations: complex edit trajectories may reduce efficiency, training is more complex than traditional tasks, and models are small (max 300M). Future directions: learn optimal edit strategies, mix with autoregressive models, and extend to multi-modal tasks.