Speculative Decoding

Reject Only Critical Tokens: Pivot-Aware Speculative Decoding
Speculative decoding enforces exact target-distribution matching, which can unnecessarily reduce token acceptance and limit speedups. …
Amir Ziashahabi, Yavuz Faruk Bakman, Duygu Nur Yaldiz, Mostafa El-Khamy, Sai Praneeth Karimireddy, Salman Avestimehr
PDF Code Source Document