Provably Distilling from Modern Language Models via Low Logit RankResearch Paper
Modern language models compute, at every position, a vector of logits and sample the next token from its softmax. Commercial interfaces often expose these logits: a user submits a prefix and receives the next-token logit vector. Distillation asks how much query access of this kind is needed to build a small model that samples almost the same distribution as…