The Expensive Reality
AI is expensive. Frontier model training runs cost tens to hundreds of millions of dollars. API calls to production models cost real money per token. GPU cloud instances cost $2-8/hour per GPU. A company building AI products needs to think seriously about costs — not as an afterthought, but as a core architectural concern. This article provides concrete numbers and decision frameworks for managing AI costs in production.
Training Cost: The Big Numbers
Estimated training costs for recent major models (based on public information and inference from available compute pricing and FLOPs estimates):
- GPT-3 (175B, 2020): ~$5M
- PaLM (540B, 2022): ~$10-20M
- Llama 3 70B (2024): ~$2-5M (estimated; Meta hasn't disclosed)
- DeepSeek V3 (2024): ~$5.5M (disclosed by DeepSeek)
For most applications, you're not training frontier models — you're using them. But even fine-tuning has significant costs. LoRA fine-tuning of a 7B model on a consumer A100 runs 1-4 hours; on 8×H100s, it's minutes. Cloud H100 costs: $2.50-6/GPU-hour depending on provider and commitment level.
Inference API Cost: Per-Token Economics
API costs (approximate, mid-2026 pricing — subject to rapid change as models become more efficient):
- Claude 3.5 Sonnet: ~$3/M input tokens, ~$15/M output tokens
- GPT-4o: ~$5/M input, ~$15/M output
- Claude 3 Haiku: ~$0.25/M input, ~$1.25/M output
- GPT-4o mini: ~$0.15/M input, ~$0.60/M output
- Llama 3.1 70B (via Groq): ~$0.59/M input, ~$0.79/M output
Key insight: the cost difference between frontier and commodity models is 10-50×. For high-volume applications, this difference is critical. A feature that makes 10M API calls/month at $5/M = $50K/month. The same feature at $0.15/M = $1.5K/month.
The Model Selection Cost Tradeoff
The key engineering decision: which model is good enough for this task? "Good enough" is the operative phrase. Using a frontier model for tasks that a cheaper model handles adequately wastes money. Using a cheap model for tasks that require frontier capabilities wastes user experience. Build a quality evaluation for each use case and measure whether cheaper models meet your quality bar.
Cost Optimization Strategies
- Caching: Cache LLM responses for repeated queries. Effective for search (same queries recur) and any application with repeated inputs. Scolta caches AI overviews for 30 days — most search queries recur frequently enough for this to significantly reduce per-query cost.
- Model routing: Use a cheap model for simple requests; route complex requests to expensive models. Requires a classifier to determine request difficulty.
- Context optimization: Input tokens cost money. Compress prompts, summarize conversation history, use RAG to provide targeted context rather than stuffing everything.
- Batching: Group requests for batch processing where latency allows. Many providers offer batch APIs at 50% discount.
- Fine-tuning for efficiency: A fine-tuned smaller model may match a prompted larger model at 10× lower inference cost.