How would you cut LLM serving cost by 10x? Enumerate every lever and rank them.

Common Questions Asked Across CompaniesInference, Serving and GPU PerformanceTopic: Cost optimisation

Reported at:MicrosoftAmazonNVIDIACursor

Model answer

A concise answer you could give in the room. Try your own version first, then compare.

What the interviewer is probing

Whether you can systematically enumerate the cost levers, estimate their impact and sequence them by effort and risk.

How to structure your answer

1Measure first: cost per request broken down by input and output tokens, model, cache hits and GPU utilisation.
2Product levers: shorter prompts, output limits, caching, routing easy queries to small models.
3Model levers: distillation, quantisation, fine-tuning a smaller model.
4Systems levers: batching, speculative decoding, right-sized hardware, spot capacity. Rank the levers by ROI.

Key points to hit

0/3 revealed

Try to answer each one out loud before revealing it.

The usual largest single lever?
Why look at output tokens?
How do you protect quality?

Common pitfalls

Optimising GPU kernels before measuring where the cost goes.
Shipping a cheaper model without an eval gate.
Ignoring engineering cost when ranking the levers.

Likely follow-ups

You have one week: what do you do first?
How do you price an LLM feature?
When is self-hosting cheaper than an API?

Source: Outcome School · ai-engineering-interview-questions-company-wise. Question compiled by Outcome School from publicly reported interview experiences (Apache-2.0). The guidance on this page is original.