Variable-Length Prompts on Chimera GPNPU

If you are sizing a language model for an edge product, the prompts it will see span a wide range. A spoken command is a dozen tokens, a pasted document is several thousand, and the same device serves both. Somewhere in the software stack, a decision is made about the gap between the longest prompt you provisioned for and the prompt that actually arrives. That decision sets how many binaries you build and qualify. It also sets how much of every inference is spent computing padding.

Most compiled inference stacks do not make that decision at all. A graph compiled for 1024 tokens processes 1024 positions every time it runs. If you send it 50 tokens, it computes the other 974 as padding. To serve short prompts efficiently as well as long ones, you have to build a separate binary for each length range and qualify all of them. Quadric Chimera GPNPU does not work this way. It gives you one guarantee and one optimization.

Read more here.

Fig.1: One graph serves every length up to the configured capacity. Source: Quadric

The post Variable-Length Prompts on Chimera GPNPU appeared first on Semiconductor Engineering.