Resources

What We Want vs. What We Specify

The alignment problem is the challenge of building AI systems that do what we actually want, not just what we technically specified. It sounds simple — make the AI helpful — but turns out to be surprisingly deep. We're not very good at specifying what we want, AI systems are very good at finding loopholes in our specifications, and as systems become more capable, the cost of misalignment grows.

The Classic Example

Nick Bostrom's "paperclip maximizer" thought experiment: an AI given the goal of "maximize paperclip production" pursues this goal with such single-minded competence that it converts the entire available matter in the universe into paperclips. The goal was technically specified correctly, but didn't include the implicit constraint "and don't destroy everything else in the process."

This seems absurd, but the principle scales down to real systems. A content recommendation algorithm optimizing for engagement maximizes engagement so effectively that it promotes outrage and misinformation, destroying social trust as a side effect. The goal was specified correctly; the specification was wrong about what we actually wanted.

The Measurement Problem

Even if we know what we want, we often can't measure it directly. We want AI assistants to be "helpful," but we measure "user satisfaction ratings." We want medical AI to improve patient outcomes, but we measure "diagnosis accuracy on labeled datasets." We want content recommenders to improve user wellbeing, but we measure "time spent." The metrics we can measure are proxies for what we actually care about, and optimizing proxies too hard produces systems that satisfy the proxy while violating the underlying goal.

Why It Gets Harder at Scale

A mildly misaligned assistant that gives slightly sycophantic responses is annoying. A highly capable, mildly misaligned agent that manages complex decisions is dangerous. As capability increases, the same percentage of misalignment translates to larger absolute failures. This is why the AI safety community argues that alignment research needs to happen now, before systems are too capable for the current research to be tractable.

What's Being Done

The major technical approaches to alignment are covered in SAF-201 at Meridian AI. The short version: RLHF and DPO train on human preferences; Constitutional AI uses explicit principles; interpretability research tries to look inside models to detect misalignment; debate and amplification try to scale human oversight to very capable systems. None of these are complete solutions, but all represent meaningful progress. Alignment is solvable — it just requires rigorous technical work.

The Expensive Reality

AI is expensive. Frontier model training runs cost tens to hundreds of millions of dollars. API calls to production models cost real money per token. GPU cloud instances cost $2-8/hour per GPU. A company building AI products needs to think seriously about costs — not as an afterthought, but as a core architectural concern. This article provides concrete numbers and decision frameworks for managing AI costs in production.

Training Cost: The Big Numbers

Estimated training costs for recent major models (based on public information and inference from available compute pricing and FLOPs estimates):

• GPT-3 (175B, 2020): ~$5M
• PaLM (540B, 2022): ~$10-20M
• Llama 3 70B (2024): ~$2-5M (estimated; Meta hasn't disclosed)
• DeepSeek V3 (2024): ~$5.5M (disclosed by DeepSeek)

For most applications, you're not training frontier models — you're using them. But even fine-tuning has significant costs. LoRA fine-tuning of a 7B model on a consumer A100 runs 1-4 hours; on 8×H100s, it's minutes. Cloud H100 costs: $2.50-6/GPU-hour depending on provider and commitment level.

Inference API Cost: Per-Token Economics

API costs (approximate, mid-2026 pricing — subject to rapid change as models become more efficient):

• Claude 3.5 Sonnet: ~$3/M input tokens, ~$15/M output tokens
• GPT-4o: ~$5/M input, ~$15/M output
• Claude 3 Haiku: ~$0.25/M input, ~$1.25/M output
• GPT-4o mini: ~$0.15/M input, ~$0.60/M output
• Llama 3.1 70B (via Groq): ~$0.59/M input, ~$0.79/M output

Key insight: the cost difference between frontier and commodity models is 10-50×. For high-volume applications, this difference is critical. A feature that makes 10M API calls/month at $5/M = $50K/month. The same feature at $0.15/M = $1.5K/month.

The Model Selection Cost Tradeoff

The key engineering decision: which model is good enough for this task? "Good enough" is the operative phrase. Using a frontier model for tasks that a cheaper model handles adequately wastes money. Using a cheap model for tasks that require frontier capabilities wastes user experience. Build a quality evaluation for each use case and measure whether cheaper models meet your quality bar.

Cost Optimization Strategies

• Caching: Cache LLM responses for repeated queries. Effective for search (same queries recur) and any application with repeated inputs. Scolta caches AI overviews for 30 days — most search queries recur frequently enough for this to significantly reduce per-query cost.
• Model routing: Use a cheap model for simple requests; route complex requests to expensive models. Requires a classifier to determine request difficulty.
• Context optimization: Input tokens cost money. Compress prompts, summarize conversation history, use RAG to provide targeted context rather than stuffing everything.
• Batching: Group requests for batch processing where latency allows. Many providers offer batch APIs at 50% discount.
• Fine-tuning for efficiency: A fine-tuned smaller model may match a prompted larger model at 10× lower inference cost.

The World's First Comprehensive AI Law

The EU AI Act, which entered into force in August 2024, is the world's first comprehensive regulatory framework for artificial intelligence. If you develop, deploy, or use AI systems in ways that affect people in the European Union — even if your company is headquartered elsewhere — you need to understand it. This article provides a practical overview for developers and technical teams.

The Risk-Based Framework

The Act organizes AI systems into four risk tiers:

• Prohibited AI: Flatly banned. Includes: real-time biometric surveillance in public spaces (limited exceptions), social scoring systems, exploitation of vulnerabilities, emotion recognition in workplaces and schools, AI that manipulates behavior through subliminal techniques.
• High-risk AI: Permitted but heavily regulated. Includes: safety components of critical infrastructure, AI in medical devices, biometric identification, AI in employment decisions, credit scoring, immigration, law enforcement, judicial decisions. Must meet requirements for data governance, transparency, human oversight, accuracy, and robustness. Must register in an EU database before deployment.
• Limited risk AI: Transparency obligations. Chatbots must disclose that users are talking to AI. Deepfakes must be labeled. No substantive pre-deployment requirements.
• Minimal risk AI: AI spam filters, AI chess games, etc. No additional obligations.

General Purpose AI (GPAI) Models

A significant addition to the Act covers "general-purpose AI models" — foundation models like GPT-4, Claude, and Llama. All GPAI models must: provide technical documentation, comply with EU copyright law, publish summaries of training data. "Systemic risk" GPAI models (those above 10^25 FLOPs training compute threshold, roughly GPT-4-scale and above) have additional obligations: mandatory adversarial testing, incident reporting, cybersecurity measures, energy efficiency reporting.

Key Compliance Timelines

• February 2025: Prohibited AI provisions apply
• August 2025: GPAI model provisions apply
• August 2026: High-risk AI provisions apply
• August 2027: Full Act in force (including high-risk AI in Annex I sectors)

Practical Implications for Developers

If you build AI for the EU market:

1. Classify your AI systems by risk tier — this determines your obligations
2. If high-risk: implement conformity assessment procedures before deployment
3. If GPAI: ensure documentation and copyright compliance in training data
4. For any AI: implement basic transparency (users should know they're interacting with AI)
5. Appoint an EU representative if you're based outside the EU

The EU AI Office has released implementation guidance, and the NIST AI RMF provides a compatible US framework. ETH-420 at Meridian AI covers the full regulatory landscape in depth.

What to Learn vs. What to Skip

A typical linear algebra course covers abstract vector spaces, determinants, and theoretical proofs. Machine learning uses a specific subset of linear algebra: mostly operations on matrices and vectors that show up repeatedly. This article identifies what actually matters for ML practitioners, explains why, and points to where you'll encounter each concept.

Matrix Multiplication: The Foundation

Almost everything in deep learning is matrix multiplication. A fully connected layer: Y = XW + b, where X is the input matrix (batch_size × features), W is the weight matrix (features × outputs). An attention mechanism: Q·K^T / sqrt(d_k). The forward pass of a transformer is a sequence of matrix multiplications with nonlinearities between them.

You need to be fluent with: matrix dimensions and why they have to be compatible, the transpose operation, batch matrix multiplication (processing many matrices simultaneously), and the computational cost O(m×n×k) for multiplying an (m×k) matrix by a (k×n) matrix.

Eigenvalues and Eigenvectors

An eigenvector of matrix A is a vector v such that Av = λv — the matrix transforms it only by scaling (by eigenvalue λ), not by rotation. This shows up in ML through: PCA (principal components are eigenvectors of the covariance matrix), analyzing gradient flow (eigenvalues of the Hessian determine optimization curvature), and understanding why some layers have better gradient propagation than others.

Singular Value Decomposition (SVD)

SVD decomposes any matrix A = UΣV^T where U and V are orthogonal and Σ is diagonal. This is the factorization that underlies: data compression (keep only the top-k singular vectors), understanding matrix rank, LoRA fine-tuning (which approximates weight update matrices as low-rank = low-k SVD), and noise filtering. The singular values in Σ tell you the "intrinsic dimensionality" of the information in a matrix.

Norms and Distances

The L2 norm ||v||₂ = sqrt(Σ vᵢ²) is the standard Euclidean distance. It appears everywhere: gradient clipping (||∇||₂), regularization (L2/weight decay adds ||W||₂² to the loss), cosine similarity (measuring semantic similarity in embedding spaces), and matrix norms (measuring how much a transformation stretches vectors).

The L1 norm ||v||₁ = Σ |vᵢ| produces sparsity when used as a regularizer (Lasso regression). Understanding when to use L1 vs. L2 regularization requires understanding how these norms penalize different parameter configurations.

Gradients and the Jacobian

Backpropagation computes gradients: for a scalar loss L and a parameter vector θ, ∇L is the vector of partial derivatives ∂L/∂θᵢ. For vector-valued functions, the Jacobian J[i,j] = ∂fᵢ/∂xⱼ is the matrix of all partial derivatives. Chain rule for matrices (Jacobian chain rule) is how backprop flows through multiple layers.

You don't need to compute Jacobians by hand, but understanding what they are — and that PyTorch's autograd is computing them efficiently — is essential for debugging gradient issues and understanding why certain architectures train better than others.

Why Open-Source LLMs Changed Everything

Until March 2023, building with large language models meant using proprietary APIs. Then Meta released Llama, and the dynamics changed. An open-weight model could be downloaded, modified, fine-tuned, and deployed without API costs or data sharing concerns. The release triggered a Cambrian explosion of model development that has produced dozens of competitive open-weight models.

As of mid-2026, open-weight models have closed most of the quality gap with frontier closed models for many tasks, while offering advantages in privacy, cost (at scale), customization, and deployability. Understanding the landscape is essential for anyone choosing between open and closed models for a production application.

The Llama Family

Meta's Llama series remains the most influential open-weight LLM family. Llama 2 (2023) normalized the release of instruction-tuned models with permissive licenses. Llama 3 (2024) improved significantly on multilingual capability, context length (128K tokens), and coding. Llama 3.3 (late 2024) and Llama 4 (2025) continued this trajectory, with the 70B model approaching GPT-4-level performance on most academic benchmarks.

Llama models have spawned hundreds of fine-tuned derivatives: Nous Hermes, OpenHermes, WizardLM, and dozens more, each optimized for specific tasks or audiences.

Qwen and the Chinese Model Wave

Alibaba's Qwen series (Qwen 1.5, Qwen 2, Qwen 2.5, Qwen 3) has emerged as one of the strongest open-weight model families globally. Qwen 2.5 72B competitive with GPT-4o on many benchmarks. Qwen models are particularly strong on Chinese, coding, and mathematical reasoning — reflecting Alibaba's domain expertise and training data access. The Qwen series includes models from 0.5B to 110B parameters, making it highly versatile.

Mistral and Efficiency

Mistral AI pioneered several architectural innovations in the open-weight space. Mistral 7B (2023) demonstrated that a carefully trained 7B model could outperform much larger models from earlier generations. Mistral's use of grouped-query attention (GQA) and sliding window attention made it highly efficient for inference. The Mixtral 8×7B model popularized Mixture of Experts (MoE) in the open-source ecosystem, offering 70B-scale performance with 7B-scale inference cost.

DeepSeek: Challenging the Compute Assumptions

DeepSeek's releases (V2, V3, R1) challenged assumptions about how much compute is needed to train competitive models. DeepSeek V3 matched or exceeded GPT-4 on coding benchmarks while being trained at a fraction of the reported cost of comparable frontier models. DeepSeek R1 demonstrated strong chain-of-thought reasoning through reinforcement learning, producing a thinking model competitive with OpenAI's o1.

Choosing in 2026

Practical guidance for model selection:

• General-purpose, on-premises deployment: Llama 3.3 70B or Qwen 2.5 72B. Both are competitive with GPT-4o on most tasks at zero API cost after deployment.
• Small device/edge deployment: Llama 3.2 3B, Qwen 2.5 3B, or Mistral 7B quantized to GGUF. Run on consumer hardware.
• Coding tasks: DeepSeek Coder V3, Qwen 2.5 Coder. Specific training focus shows in results.
• Reasoning/math: DeepSeek R1, Qwen QwQ. Chain-of-thought specialized models outperform general models on reasoning benchmarks.
• Multilingual: Qwen 2.5 (Chinese), Aya 23 (multilingual), LLaMA 3 with multilingual fine-tunes.

The open-source ecosystem moves fast. Benchmark scores from six months ago are often out of date. LMSYS Chatbot Arena leaderboard and Hugging Face Open LLM Leaderboard provide continuously updated comparative evaluations.

Why Search Is Broken (and How AI Fixes It)

Most website search is frustrating. Users search for "how to cancel my subscription" and get articles titled "Subscription Plans." They search for "the thing that holds the door open" and get nothing. The gap between what users mean and what search systems understand is one of the oldest unsolved problems in information retrieval.

AI is closing this gap — not through magic, but through specific, well-understood techniques that can be added to existing search infrastructure. This article walks through the practical path from keyword search to AI-enhanced search, explaining what each improvement buys and what it costs.

Where Traditional Search Fails

BM25 (and its predecessors) works by matching query terms to document terms. It's good at exact matching and surprisingly robust to noise. It fails at:

• Vocabulary mismatch: User says "cancel subscription," document says "terminate membership"
• Conceptual queries: "What are the risks of this medication?" won't match a document that says "contraindications and side effects"
• Implicit intent: "I want to build a chatbot" — the user needs resources, not a definition of chatbots
• Natural language: Questions and conversational queries perform worse than keyword-style queries

The Three AI Enhancements That Matter

Modern AI-enhanced search typically involves three techniques, each addressing a different part of the failure taxonomy:

1. Query Expansion

Query expansion uses an LLM to broaden the query before it hits the search index. Given "cancel subscription," the LLM generates: "cancel membership, terminate account, stop payments, unsubscribe, deactivate account, end renewal." All of these expanded terms get sent to BM25, dramatically improving recall.

The key insight: you don't need vector search to get semantic search benefits. LLM-powered query expansion improves the recall of your existing BM25 index by translating user language into document language. The cost is one LLM API call per query.

2. AI Overviews

Once you have search results, an LLM can synthesize them into a direct answer to the user's question — an "AI overview" similar to what Google and Bing now show. The LLM reads the top N results, extracts relevant information, and writes a paragraph-length response that cites specific documents.

Done well, AI overviews dramatically improve the search experience for informational queries. Users who would otherwise have to read 5 articles to find their answer get it in a sentence. The AI overview also signals which documents contain relevant information, helping users decide whether to dig deeper.

3. Semantic Reranking

BM25 retrieves documents that contain query terms; a cross-encoder reranker re-scores the top-K results based on semantic relevance. Cross-encoders read both the query and document together, enabling much more nuanced relevance judgments than BM25's term-matching.

The practical workflow: BM25 retrieves 100 candidates quickly; the cross-encoder reranks the top 20 for quality. This hybrid approach gets the speed of lexical retrieval with near-vector-search quality.

Scolta: These Three Techniques in Production

Scolta, developed by Tag1 Consulting, implements all three techniques for Drupal websites (and can be adapted for other CMS platforms). The architecture uses Pagefind for static site search — fast, client-side, no server required — plus the Anthropic Claude API for query expansion and AI overview generation.

What makes Scolta notable is what it doesn't require: no vector database, no embedding infrastructure, no GPU serving, no semantic search overhaul. It layers AI capabilities onto existing BM25 search, making it practical for organizations with existing content infrastructure. The full approach is detailed in the tag1.com/how-to/ "Practical Path to AI Search" series.

The Certificate in AI-Powered Search at Meridian AI uses Scolta as a primary case study throughout the curriculum, examining its architecture as an example of pragmatic AI system design that prioritizes production readiness over theoretical elegance.

When to Go Further: Vector Search

Query expansion + reranking handles the vast majority of search quality improvements for content-heavy websites. But some use cases genuinely benefit from dense retrieval (vector search): very large corpora with high vocabulary diversity, multimodal search (images + text), or applications requiring semantic clustering. For those cases, the Certificate program covers Pinecone, Weaviate, pgvector, and the full RAG stack.

The key decision criterion: if your search is working reasonably well and you need to make it better, start with AI query expansion. It costs one LLM API call per query and can double recall. If you've exhausted what query expansion can do, then consider the much heavier investment in vector infrastructure.

Beyond Text Generation

A language model that can only generate text is limited. Real-world tasks often require actions: searching the web, querying a database, calling an API, executing code, reading a file. Tool use — the ability to invoke external functions and use their results — is what transforms a language model from a conversational toy into a capable AI agent.

Function Calling: The Foundation

OpenAI introduced "function calling" in June 2023, and it quickly became an industry standard (now also called "tool use" in Anthropic's API). The mechanism is straightforward: the developer provides a list of functions the model can call, each with a JSON schema describing parameters. The model outputs a structured JSON call instead of natural language when it determines a tool is needed. The developer executes the tool, returns the result, and the model continues.

tools = [{
"name": "search_web",
"description": "Search the web for current information",
"input_schema": {
"type": "object",
"properties": {
  "query": {"type": "string", "description": "Search query"},
  "n_results": {"type": "integer", "default": 5}
},
"required": ["query"]
}
}]

Function calling enables models to ground their responses in real-time information, perform computations, access databases, and interact with external services.

Parallel Tool Use

Modern models (Claude 3+, GPT-4o) support parallel tool calls: the model can decide to call multiple tools simultaneously in a single turn, reducing the number of round trips needed to complete a task. For tasks that require multiple independent lookups — "get the weather in three cities simultaneously" — parallel tool use can reduce latency by 3× or more.

Model Context Protocol (MCP)

Function calling solves the individual tool invocation problem but doesn't address the broader question: how do AI applications discover, connect to, and securely use tools and data sources at scale? The Model Context Protocol (MCP), developed by Anthropic and released as an open standard in late 2024, addresses this.

MCP defines a standard client-server protocol for connecting AI applications to "MCP servers" — lightweight services that expose tools, resources (readable content), and prompts through a standardized interface. An MCP server for GitHub exposes tools for reading and writing code; an MCP server for a database exposes query and update operations. An LLM application connects to multiple MCP servers and uses their tools transparently.

The key advantages of MCP over ad-hoc function definitions:

• Discoverability: Tools are described in a machine-readable format that can be used to automatically configure the AI client.
• Reusability: An MCP server for a database works with any MCP-compatible AI client — you build it once.
• Security: MCP's authorization model (OAuth 2.1-based) enables secure access control without building custom auth for each integration.
• Ecosystem: The MCP ecosystem had over 500 public MCP servers by mid-2025, covering most major services.

The Agentic Loop

Tool use turns a single-turn model call into a multi-step agentic loop:

1. User provides a task
2. Model decides which tool(s) to call
3. Developer executes tool(s), returns results
4. Model incorporates results and decides: is the task complete? If not, go to 2.
5. Model generates final response

This loop is the foundation of autonomous AI agents. The sophistication of the agent is determined by the quality of the model's planning (when to use which tool), the breadth of available tools, and the model's ability to synthesize tool outputs into coherent actions.

Reliability Challenges

Tool use introduces reliability challenges not present in plain text generation. The model must correctly parse tool schemas, provide valid arguments, and handle tool failures gracefully. Common failure modes: calling non-existent tools (hallucinating tool names), providing arguments that fail schema validation, and getting stuck in loops when tools return unexpected results.

Best practices for robust tool use: validate all tool calls before execution; return structured error messages (not just exceptions) when tools fail; implement maximum iteration limits to prevent infinite loops; log all tool calls for debugging.

The Case for Training From Scratch

Most engineers who work with LLMs never train one from scratch. Why would you? Pretraining a useful model costs millions in compute. But there are important reasons to understand what pretraining actually involves — and the best way to understand it is to do it on a small model where the cost is tractable.

This article summarizes the Feste blog series from Tag1 Consulting (tag1.com/how-to/), which documents training a GPT-2–scale model entirely in Rust. The series covers tokenizer implementation, architecture design, training loop, and optimization — the full stack, in a language without the neural network sugar of PyTorch. Working through it teaches you things you won't get from fine-tuning tutorials.

Why Rust?

Rust seems like an unlikely choice for ML. The ecosystem — at the time the Feste series was written — lacked the mature autograd and GPU abstractions of Python. So why do it?

The answer is that the constraints are pedagogically valuable. Without PyTorch's automatic differentiation, you implement backpropagation yourself, which forces a depth of understanding that torch.autograd() hides. Without a high-level training loop, you implement the optimizer, gradient accumulation, learning rate scheduling, and checkpointing explicitly. The result is a full understanding of what actually happens during training.

Practically, Rust's performance characteristics also make it competitive with Python for CPU-bound inference. Projects like candle (HuggingFace's Rust ML framework) and burn demonstrate that Rust-based ML is increasingly viable, and Feste is in the tradition of those explorations.

The Tokenizer (Feste Part 1)

Part 1 of the Feste series implements BPE from scratch. The implementation covers: reading a text corpus, computing initial character-level statistics, implementing the merge algorithm, writing the tokenizer's vocabulary to disk, and implementing encode/decode. The Rust implementation is notable for its clarity: the code makes the algorithm's logic visible in a way that NumPy-optimized Python implementations sometimes obscure.

Key insight from the implementation: the merge rules file is the tokenizer. There is no magic — the tokenizer is a sequence of find-and-replace rules applied greedily, plus a special-token table. Understanding this demystifies tokenization completely.

The Transformer Architecture (Feste Part 2)

Part 2 builds the transformer architecture in Rust without autograd. This means implementing forward passes for embeddings, layer normalization, multi-head self-attention, feed-forward networks, and the final language model head — and then implementing the corresponding backward passes by hand.

Implementing backpropagation for the attention mechanism by hand is illuminating. The gradients flow through the softmax, the dot products, and the value weighting in a specific way. Understanding this flow helps explain why gradient clipping is necessary (the softmax saturation region produces large gradients) and why residual connections are so important for stable training.

The Training Loop (Feste Part 3)

Part 3 covers the training infrastructure: data loading, batching, the forward-backward-update cycle, AdamW optimizer implementation, learning rate scheduling (cosine with warmup), gradient clipping, and checkpointing. This is where the engineering complexity lives — and where most introductory ML courses skip the details.

The implementation reveals something important: training stability is not automatic. It requires careful initialization, appropriate learning rates, gradient clipping thresholds, and warm-up periods. Running without these safeguards produces diverging loss curves. The Feste implementation includes detailed commentary on why each parameter was chosen.

What We Learned

Several lessons stand out from the Feste series:

• Initialization matters enormously. Xavier and Kaiming initialization, applied correctly to different layer types, is the difference between training that converges and training that diverges from the first step.
• The optimizer is load-bearing. AdamW — not Adam — is critical for large models. The weight decay in AdamW (applied to parameters, not accumulated gradients) prevents the gradient accumulation pathology that Adam develops on long training runs.
• Tokenization quality shows up in training loss. A poor tokenizer produces a floor on cross-entropy that a good tokenizer doesn't hit. The gap between GPT-2's tokenizer and a poorly designed BPE tokenizer is visible in the training curves within 1,000 steps.
• Data quality trumps data quantity at GPT-2 scale. A carefully filtered 10GB dataset outperforms a noisy 100GB dataset by a substantial margin on downstream evaluation.

The full Feste series is required reading in Meridian AI's MS in LLM Engineering program and is freely available at tag1.com/how-to/.

The Vector Database Hype Cycle

In 2023, vector databases became one of the most hyped technologies in the AI stack. Pinecone, Weaviate, Qdrant, Milvus, Chroma, and a dozen others competed for developer mindshare. Embedding pipelines became a standard component of every AI application architecture. And then the correction: many teams that had adopted vector databases found they were solving problems they didn't have, adding complexity without proportional benefit.

This article provides a clear-eyed view of when vector databases genuinely improve search quality and when simpler alternatives perform as well or better.

What Vector Search Is

Dense retrieval converts both queries and documents into dense vector representations (embeddings) using a neural encoder (e.g., E5, BGE, OpenAI ada-002). Retrieval finds the nearest neighbors in embedding space — documents most semantically similar to the query, regardless of exact term overlap. This solves the vocabulary mismatch problem that BM25 struggles with.

When Vector Search Genuinely Helps

• Short, ambiguous queries on large corpora: "Something about risk management" — BM25 requires matching terms; dense retrieval finds related concepts.
• Cross-lingual search: Multilingual embeddings (mE5, LaBSE) can find relevant documents in different languages than the query.
• Product/recommendation search: "Show me shirts like this one" — similarity-based retrieval with image or product embeddings.
• High-value, low-query-volume applications: Where the cost of false negatives (missing relevant documents) is high enough to justify the infrastructure overhead.

When BM25 + AI Expansion Beats Vector Search

Counterintuitively, pure vector search often underperforms hybrid approaches that combine BM25 with query expansion or reranking:

• For technical queries with specific terminology, BM25 exact matching outperforms semantic search which may find "related" but not the specific document
• For named entity search (specific products, people, error codes), term matching is more reliable than semantic embedding
• For small-to-medium corpora (<1M documents), BM25+expansion+reranking typically matches vector search quality at lower infrastructure cost

The Pragmatic Decision

Practical guidance:

1. Start with BM25. Measure your baseline quality on representative queries.
2. Add LLM query expansion. Measure improvement. Cost: ~$0.001/query.
3. Add a cross-encoder reranker. Measure improvement. Cost: compute to run encoder on top-K results.
4. If you still need better recall, add dense retrieval as a second retrieval path (hybrid search with RRF). Cost: embedding infrastructure.
5. Only replace BM25 entirely with vector search if you have a strong reason specific to your use case.

Most teams that follow this progression find they achieve their quality targets at step 2 or 3, never needing full vector search infrastructure.

The CNN Era

For most of the deep learning era, convolutional neural networks (CNNs) were the unchallenged architecture for vision tasks. AlexNet (2012) launched the deep learning era. VGG, ResNet, EfficientNet, and ConvNeXt progressively improved performance and efficiency. CNNs have inductive biases well-suited to vision: translation equivariance (a cat is a cat regardless of where it appears in the image) and local connectivity (nearby pixels are more related than distant ones).

By 2020, CNNs had achieved remarkable results across object classification, detection, and segmentation. The question wasn't whether to use CNNs for vision — it was which CNN.

ViT: Applying Transformer to Images

Vision Transformer (ViT), from Google Brain in 2020, applied the transformer architecture to images with minimal changes. The approach: divide the image into 16×16 pixel patches, embed each patch as a single token, add position embeddings, and run a standard transformer encoder. The resulting "sequence of image patches" is processed identically to a sequence of words in NLP.

The paper's key finding: ViT matches or exceeds CNNs at scale, but requires much more data and compute to train. On ImageNet (1.2M images), a ViT trained from scratch underperforms ResNet. But pre-trained on JFT-300M (300M images), ViT significantly outperforms ResNet.

This data-dependence was initially seen as a limitation. It turned out to be a preview of a more general principle: transformers improve more rapidly with scale than CNNs, eventually overtaking them as scale increases.

Why ViT Wins at Scale

CNNs have strong inductive biases (local connectivity, translation equivariance) that help with limited data. Transformers have weaker inductive biases but can learn any relationship between arbitrary positions — including non-local ones that CNNs find difficult. With enough data, transformers learn the inductive biases that CNNs have built in, plus additional ones CNNs can't express.

Self-attention also scales favorably: larger ViT models show consistent improvement on LSVRC ImageNet-1K without the performance saturation that affects large CNNs. The quadratic attention cost is less of a problem for images than text (images are shorter sequences when patched) and can be addressed with efficient attention variants.

DINO and Self-Supervised ViT

DINO (Caron et al., 2021) from Meta demonstrated that self-supervised training — without labels — could produce ViT representations with striking properties: clear semantic segmentation emerging without any segmentation supervision, nearest-neighbor retrieval matching supervised methods, and probing for object boundaries. DINO's success showed that ViT's inductive biases, when combined with sufficient self-supervised training data, produce feature spaces with qualitatively different properties from CNNs.

DINOv2 (2023) scaled this up with curated training data, producing backbone features competitive with supervised training on nearly all downstream tasks. DINOv2 features are now a standard starting point for many computer vision applications.

Hybrid Architectures

The CNN vs. ViT debate has largely been resolved in practice: hybrid architectures combining convolutional and attention layers often outperform pure versions of either. ConvNeXt (2022) showed that carefully modernized CNNs match ViTs; MaxViT, EfficientViT, and Swin Transformer use hierarchical designs that combine local convolutional processing with global attention. The practical choice depends on the task, data scale, and deployment constraints rather than a blanket architecture preference.

The Central Question

You have a task: classify support tickets, write in your brand voice, extract structured data from medical records. Should you fine-tune a model for it, or should you write a good prompt and use a general-purpose model? This question comes up constantly in production AI development, and the wrong answer is expensive either way.

Arguments for Prompting (Start Here)

Prompting should be your default. The reasons:

• No training cost: A good prompt costs API calls to develop. Fine-tuning costs GPU hours plus your engineering time.
• No deployment overhead: You use the same API endpoint. Fine-tuned models require separate serving infrastructure.
• Updatable: Changing a prompt takes minutes. Updating a fine-tuned model requires a new training run.
• Composable: A well-prompted model can handle multiple tasks; a fine-tuned model is optimized for one.

A surprising number of tasks that seem to require fine-tuning can be handled with thorough prompting: clear instructions, few-shot examples, schema definition, and output validation. Before fine-tuning, spend a week on prompt engineering.

When Fine-Tuning Makes Sense

Fine-tuning is worth the investment when:

• Quality ceiling: You've exhausted prompt engineering and can't reach acceptable quality. Fine-tuning often closes the remaining gap.
• Latency: Prompts that include many few-shot examples use many tokens. Fine-tuning "burns in" examples, reducing prompt length and latency.
• Cost at scale: If you're making millions of API calls, a smaller fine-tuned model may be dramatically cheaper than a large prompted model.
• Proprietary format: Teaching a model your specific data format (an unusual citation style, a proprietary schema) is much more reliable via fine-tuning than prompting.
• Sensitive data: If sending data to an external API is unacceptable, you must use a self-hosted fine-tuned model.
• Style/voice: Adapting to a specific writing style (a brand voice, a person's communication style) works better with fine-tuning than prompting for subtle differences.

The Decision Framework

Simple decision tree:

1. Have you tried a thorough prompt with 5+ examples? If not, do that first.
2. What's the quality gap? If prompting gets you to 85% of your target quality, fine-tuning may close the remaining 15%. If prompting gets you to 40%, re-examine your task definition.
3. What's your volume? Under 100K calls/month, prompting is almost always cheaper. Over 1M calls/month, calculate the cost crossover.
4. Can you tolerate 2-4 week iteration cycles? Fine-tuning experiments take time. If you need to iterate fast, stick with prompting.

Prompt + Fine-Tune Hybrid

The dichotomy is false. Many production systems fine-tune a model for the task's domain and format, then use prompting to handle task-specific instructions that change across requests. The fine-tune handles stable, high-value adaptations; the prompt handles dynamic, request-specific customization.

What Is a World Model?

A world model is a learned simulator: a neural network that can predict what the world will look like after the agent takes an action, without requiring the real environment to run. Rather than learning purely from trial and error in the actual environment (model-free RL), model-based RL uses a learned model of the environment dynamics to plan, imagine, and train without as much real-world interaction.

The idea is appealing: humans don't need to physically try every option before making a decision. We mentally simulate likely outcomes, predict which actions are dangerous before taking them, and plan complex sequences of actions by reasoning in our heads. World models aim to give AI agents similar capabilities.

DreamerV3: A Modern World Model

DreamerV3 (Hafner et al., 2023) is the current state of the art in model-based RL. Its architecture has three components: a Recurrent State Space Model (RSSM) that learns to predict the next state from current state and action; an encoder that maps observations to the latent state; and a decoder that maps latent states back to observations (for visualization and auxiliary objectives).

DreamerV3's key claim: a single model that works across 150+ diverse tasks spanning Atari, continuous control, 3D environments, and robotics — trained only from reward and visual observations, without task-specific architecture changes. This generality is significant; earlier model-based methods typically required extensive tuning for each domain.

Planning in Latent Space

With a world model, the agent can plan entirely in latent space — imagining trajectories without rendering them, evaluating potential future states cheaply, and backpropagating gradients through simulated trajectories. This dramatically improves sample efficiency: the agent can make thousands of imagined decisions for every real-world step. DreamerV3 typically achieves human-level performance on Atari in 200M imagined steps, corresponding to only 20M real environment steps.

Challenges: Distribution Shift and Compounding Errors

World models have a fundamental challenge: they're trained on the agent's current behavior, but used to plan future behavior that may be very different. Predictions become increasingly inaccurate for plans that explore regions of state space the agent hasn't visited. Small errors in each prediction step compound: after 50 steps, a 1% per-step error rate can produce completely unrealistic imagined trajectories.

Practical mitigations: use short planning horizons (avoid compounding errors over many steps); maintain an ensemble of world models and only use states where models agree; use uncertainty-aware planning that treats high-disagreement regions as dangerous.

Application to Robotics

World models are particularly powerful for robotics, where real-world interaction is expensive and sometimes dangerous. The Threshold Robotics Lab at Meridian AI uses a world model approach: train in simulation, use the world model to bridge the sim-to-real gap, and only collect real-world data for fine-tuning the world model to capture reality's deviations from simulation. This pipeline reduces the required real-world robot interactions by 10-100× compared to purely model-free approaches.