MTP doubles generation speed, but saves only ~3% total time at 64k context

When an AI generates a response, it works in two stages. First, it reads and processes all the text you gave it — this is called the prefill stage. Then it generates the reply word by word (or, with MTP, several words at a time). MTP targets only the second stage, so any speedup there is limited by how much of the total time that stage actually takes.

At a 64,000-token context — the kind you get with long documents or an AI agent that has accumulated a lot of conversation history — the prefill stage takes so long that doubling generation speed barely moves the needle on total wait time. The author measured this directly on an RTX 3090 GPU. The practical takeaway: MTP is most valuable for short-context tasks. For agents or pipelines handling large contexts, reducing the prefill cost (e.g., caching, shorter prompts) matters far more than faster generation.