Any plans to train the MTP draft model as well?
My initial testing of this finetune is really impressive, it is a lot more diverse. Unfortunately because it's choosing different tokens, MTP draft acceptance rate tanks, which of course is expected.
Any plans to finetune the draft model as well?
I'm looking into this, hope to have something soon!
So after a ton of investigation it turns out you can't just...finetune a draft model and call it a day - The way Google trained them was during the pretraining phase, and I gave it several shots to try and reproduce their methods but I simply can't create anything that's better then the default MTP model right now.
Worse then the default? Yeah, that totally wasn't an issue, lol. A very educational lesson, at least!
Closing this for now.
Bummer. Thanks for trying, that's fascinating information.