The State of Generative AI

Pricing Comparison (as of July 2026)

API pricing has compressed dramatically. The cost of intelligence has dropped roughly 10× per year since GPT-4's launch. What cost $60/M tokens in early 2024 now costs $3–15/M for equivalent quality. The race to the bottom is real — but hidden costs in reliability, rate limits, and feature gaps make raw $/token comparisons misleading.

The Anthropic API

Anthropic's API has matured into arguably the most developer-friendly in the space. Key differentiators as of July 2026:

• Prompt Caching: Up to 90% discount on cached input tokens. System prompts, few-shot examples, and large documents can be cached across requests. This is a game-changer for applications with stable system prompts — at Coursera-scale, this alone could cut API costs 40–60%.
• Extended Thinking: Claude can show its reasoning chain via a dedicated thinking block in the response. You can set a budget_tokens to control how much "thinking" the model does — critical for cost management on reasoning-heavy tasks.
• Tool Use (Function Calling): Native support for structured tool definitions. Claude excels at multi-step tool chains — the agentic loop pattern where it decides which tool to call, processes the result, and decides next steps.
• Batches API: 50% cost reduction for async workloads. Submit up to 100K requests, results in ~24 hours. Perfect for assessment grading, content analysis, bulk translations.
• Citations: Built-in source attribution when working with provided documents — crucial for education use cases where provenance matters.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-5",
    max_tokens=16000,
    thinking={
        "type": "enabled",
        "budget_tokens": 10000  # Max tokens for reasoning
    },
    messages=[{
        "role": "user",
        "content": "Analyze this student's essay for critical thinking depth..."
    }]
)

# Response contains both thinking and text blocks
for block in response.content:
    if block.type == "thinking":
        print("Reasoning:", block.thinking)
    elif block.type == "text":
        print("Answer:", block.text)

The OpenAI API

OpenAI remains the default choice for many teams purely through inertia and ecosystem breadth. Their API surface area is enormous — chat completions, embeddings, image generation (DALL-E 3), audio (Whisper, TTS), Realtime API for voice, file search, assistants, and more. The "everything under one roof" pitch is compelling, but the sprawl means individual features often lag behind focused competitors.

Key considerations: OpenAI's Structured Outputs feature (JSON schema enforcement) is excellent and more battle-tested than competitors. Their Assistants API provides server-side state management — useful if you don't want to manage conversation history yourself, but it adds vendor lock-in. The Realtime API for voice is genuinely best-in-class for conversational AI, though Gemini's native audio support is closing fast.

from openai import OpenAI
from pydantic import BaseModel

class GradeResult(BaseModel):
    score: int
    feedback: str
    strengths: list[str]
    areas_for_improvement: list[str]

client = OpenAI()
completion = client.beta.chat.completions.parse(
    model="gpt-5",
    messages=[
        {"role": "system", "content": "Grade this assignment 0-100."},
        {"role": "user", "content": essay_text}
    ],
    response_format=GradeResult
)
result = completion.choices[0].message.parsed

The Google API (Vertex AI / Gemini API)

Google's play is differentiated on two axes: context window and cost. Gemini 3.6 Flash (shipped July 21, 2026) at $1.50/$7.50 per Mtok with a 1M context window remains the value pick for document-heavy workflows, and it uses roughly 17% fewer output tokens than 3.5 Flash while scoring higher on coding, long-context, and computer-use benchmarks — so the effective cost gap is wider than the sticker price suggests. Google also ships a cheaper Flash-Lite tier for high-volume work. If you need to ingest an entire textbook and answer questions about it, Gemini is still the economic winner. Update (August 13, 2026): Gemini 3.7 Flash superseded 3.6 as the default workhorse just three weeks later, with introductory pricing of $0.75/$3.75 per Mtok through December 31, 2026 — half of 3.6 Flash — before returning to $1.50/$7.50 on January 1, 2027. Google built it through algorithmic improvement rather than a from-scratch training run, and it shipped while the frontier-tier Gemini 3.5 Pro remained delayed. Budget off the post-January rate, not the promotional one.

The strategic caveat is bigger than pricing: Google is the only major frontier lab without a shipped 2026 flagship. Gemini 3.5 Pro has now missed multiple targets since Pichai promised it at I/O on May 19 — DeepMind abandoned the base iteration over reasoning and coding ceilings, and reporting indicates the team has moved on to pretraining Gemini 4. Plan around the Flash tier, not around a Pro model that may not arrive on your timeline.

The caveat: Google's API reliability and documentation quality still lag. Enterprise customers report intermittent 500 errors during peak usage, and the dual API surface (Gemini API vs. Vertex AI) creates confusion about which features are available where. The Vertex AI path adds GCP lock-in. If your infrastructure is already on GCP, this is a non-issue. If it's on AWS, the friction is real.

Platform Lock-in Analysis

✅ Knowledge Check

🃏 Flashcards

The Agent Landscape in 2026

2025 was the year agents went from demos to production. 2026 is the year they became boring infrastructure — which is exactly what you want from a technology you're betting your platform on. The hype cycle has settled, and what remains is a clear pattern: agents are just LLMs in a loop with tool access and memory. The magic is in the orchestration, not the individual model call.

Three frameworks dominate production deployments:

Claude Agent SDK (Anthropic)

Anthropic's Agent SDK, released alongside Claude Code, is the most opinionated of the three. Its core insight: most agent failures come from over-engineering the orchestration. The SDK favors a "put the model in a loop" approach — give it tools, let it decide what to do, and intervene only when it goes off track.

Key concepts:

• Agent loop: The SDK manages the retry/tool-call/response cycle automatically. You define tools, the agent decides when and how to use them.
• Hooks: Lifecycle callbacks (before/after tool calls, on error) for logging, guardrails, and human-in-the-loop checkpoints.
• MCP Integration: Native support for the Model Context Protocol, meaning your agent can connect to any MCP-compatible data source or tool.
• Subagents: Spawn child agents for parallel subtasks, each with their own tool set and context.

import anthropic
from anthropic.agent import Agent, Tool

# Define tools the agent can use
tools = [
    Tool(
        name="search_courses",
        description="Search Coursera course catalog",
        input_schema={
            "type": "object",
            "properties": {
                "query": {"type": "string"},
                "level": {"type": "string",
                         "enum": ["beginner", "intermediate", "advanced"]}
            }
        },
        handler=search_courses_fn
    ),
    Tool(
        name="generate_quiz",
        description="Generate assessment questions for a topic",
        input_schema={...},
        handler=generate_quiz_fn
    )
]

# Create and run agent
agent = Agent(
    model="claude-sonnet-5",
    tools=tools,
    system="You are a course design assistant for Coursera...",
    max_turns=20,
    hooks={
        "before_tool_call": log_tool_usage,
        "on_error": handle_agent_error
    }
)

result = await agent.run("Design a 4-week ML fundamentals course...")

OpenAI Agents SDK

OpenAI's Agents SDK takes a more structured approach with explicit agent handoffs. You define specialized agents (researcher, writer, reviewer) and the SDK manages routing between them. This works well for well-defined workflows but can be rigid for open-ended exploration.

Key differentiator: the Responses API — a stateless alternative to the Assistants API that's become the recommended path. It supports tool calling, structured output, and web search in a single call without server-side state management.

Model Context Protocol (MCP)

MCP is arguably the most important infrastructure standard to emerge in 2025. Created by Anthropic but designed as an open protocol, MCP standardizes how LLMs connect to external tools and data sources. Think of it as "USB-C for AI" — a universal interface that lets any model talk to any tool.

Why MCP Matters

Before MCP, every integration was bespoke. Want your agent to query a database? Write a custom tool. Want it to read Slack messages? Another custom tool. Want it to search your docs? Yet another. MCP replaces this N×M integration problem with a standard protocol:

• MCP Servers expose resources (data) and tools (actions) through a standardized JSON-RPC interface.
• MCP Clients (the AI application) discover and invoke these tools without knowing implementation details.
• Transport-agnostic: Works over stdio, HTTP/SSE, or WebSockets.

The ecosystem has exploded: there are now MCP servers for PostgreSQL, Slack, GitHub, Google Drive, Jira, Salesforce, and hundreds more. For Coursera, you could build MCP servers that expose your course catalog, learner data, and assessment engine — then any MCP-compatible agent can use them.

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";

const server = new McpServer({
  name: "coursera-catalog",
  version: "1.0.0"
});

// Expose a tool
server.tool(
  "search_courses",
  "Search the Coursera course catalog",
  { query: { type: "string" }, limit: { type: "number" } },
  async ({ query, limit }) => {
    const results = await searchCatalog(query, limit);
    return { content: [{ type: "text", text: JSON.stringify(results) }] };
  }
);

// Expose a resource
server.resource(
  "course://{courseId}",
  "Get full course details",
  async (uri) => {
    const course = await getCourse(uri.pathname);
    return { contents: [{ uri: uri.href, text: JSON.stringify(course) }] };
  }
);

Multi-Agent Architectures

The multi-agent pattern has settled into three reliable architectures:

✅ Knowledge Check

🃏 Flashcards

The Reasoning Revolution

The biggest capability jump in 2024–2025 wasn't a bigger model — it was teaching models to think before they answer. OpenAI's o1 (September 2024) proved that inference-time compute — letting the model spend more tokens reasoning internally — could dramatically improve performance on hard problems. By mid-2026, every frontier lab has a reasoning mode, and the architecture of how this works is now well understood.

How Extended Thinking Works

The core insight is simple: instead of generating an answer in a single forward pass, the model generates a chain of intermediate reasoning steps. These steps are generated autoregressively (each token depending on the previous), which means the model can "correct course" mid-thought — something that's impossible in a single forward pass.

There are two implementation patterns:

• Hidden reasoning (OpenAI o-series): The model generates internal reasoning tokens that are not shown to the user. You pay for these tokens but never see them. The API returns only a summary. This is a black box — you can't inspect the reasoning chain.
• Visible reasoning (Claude Extended Thinking, DeepSeek-R1): The reasoning chain is returned in a separate block. You can inspect, log, and use it for debugging. Claude returns a thinking block alongside the text block. This is dramatically more useful for production systems where you need to understand why the model reached a conclusion.

The Reasoning Model Landscape

Training Reasoning: GRPO and Beyond

How do you train a model to reason? This is where Group Relative Policy Optimization (GRPO) — pioneered by DeepSeek — has become the dominant technique. GRPO is an evolution of reinforcement learning from human feedback, but with a key twist: instead of using a separate reward model, it generates multiple responses to the same prompt and uses the relative ranking within the group as the training signal.

GRPO vs. Traditional RL

In standard RLHF (PPO-based), you train a reward model on human preferences, then use it to guide policy optimization. This requires a separate reward model, a reference model for KL-divergence penalties, and careful hyperparameter tuning. GRPO simplifies this: generate K responses, score them (using a verifier or outcome-based metric), and optimize the policy to increase the probability of higher-ranked responses relative to lower-ranked ones.

The implications for practitioners: GRPO makes it feasible to add reasoning capabilities to smaller, cheaper models. You can take a Llama 4 Scout, run GRPO with math/code verifiers, and get meaningfully better reasoning without the infrastructure complexity of full RLHF. DeepSeek-R1 was trained almost entirely with GRPO, and its reasoning quality rivals o3 on many benchmarks.

When to Use Reasoning Models

Reasoning models are not universally better. They're slower and more expensive per token (because you're paying for thinking tokens). The decision framework:

• Use reasoning mode for: complex multi-step problems, code debugging, mathematical proofs, nuanced assessment rubrics, any task where "thinking step by step" would help a human expert.
• Skip reasoning mode for: classification, extraction, simple Q&A, content generation, any task where the first intuitive answer is usually correct. A standard Sonnet 5 call at 5× lower cost will perform identically on these tasks.

✅ Knowledge Check

🃏 Flashcards

Where We Are

Multimodal AI has gone from "impressive demo" to "table stakes" in 18 months. Every frontier model now accepts images and most accept audio. Video understanding is maturing rapidly. Image and video generation have reached commercial quality. The convergence of all modalities into single models is the defining architectural trend of 2026.

Vision (Image Understanding)

All frontier models now have strong vision capabilities. Claude Opus 5, GPT-5.6, and Gemini 3.6 can all analyze charts, read handwritten text, understand screenshots, and reason about complex visual content. The quality gap between them has narrowed significantly.

Gemini has a structural advantage here: it was designed as "natively multimodal" from the ground up, meaning images are processed through the same transformer weights as text. Claude and GPT-5.6 use a vision encoder that feeds into the language model, which works well but can miss subtle spatial relationships.

For education: vision capabilities are a game-changer for STEM assessment. Students can photograph handwritten math solutions, circuit diagrams, or lab results, and the model can evaluate them. The accuracy on handwritten math recognition has reached ~95% for well-lit photos — good enough for formative assessment, not yet reliable enough for high-stakes exams.

Audio & Voice

OpenAI's Realtime API (now in its second generation) remains the leader for real-time conversational voice AI. It supports speech-to-speech with ~300ms latency, interruption handling, and function calling mid-conversation. The voice quality is remarkably natural — prosody, pacing, and emotional tone are all well-handled.

Gemini's Live API offers similar capabilities with native audio understanding (the model processes audio directly, not through a speech-to-text intermediary), which enables better handling of tone, hesitation, and non-verbal audio cues.

Claude currently processes audio through transcription rather than native audio understanding, which means it's not competitive for real-time voice applications. However, for async audio processing (analyzing recorded lectures, transcribing student submissions), the transcription-based approach works well.

Video

Video understanding is the active frontier. Gemini leads here — it can process up to an hour of video natively, understanding temporal relationships, identifying key moments, and answering questions about content that requires watching the video. Claude and GPT-5.6 handle video through frame extraction (sampling keyframes and analyzing them as images), which works for many use cases but misses temporal dynamics.

Image Generation

The image generation landscape has consolidated:

Video Generation (Sora and Competitors)

OpenAI's Sora launched as a consumer product in late 2024 and has improved significantly through 2025. It generates 5–20 second clips that are visually impressive but still struggle with physics consistency (hands, object permanence) and temporal coherence beyond ~10 seconds. Google's Veo 2 is competitive on quality. Runway Gen-3 and Pika have carved niches in professional video editing workflows.

The honest assessment: video generation is not yet reliable enough for educational content production at scale. You'll spend more time fixing artifacts than you would shooting the video. Where it shines: concept visualizations, placeholder content, and marketing materials where perfect accuracy isn't required.

Code Generation as a Modality

This deserves its own section because code generation has become qualitatively different from text generation. Claude Opus 5 and GPT-5.6 can now write, debug, and refactor production-quality code across complex codebases. The key advance: agentic coding — models that can navigate a codebase, understand project structure, run tests, read error messages, and iterate until the code works.

Claude Code (Anthropic's CLI tool) and GitHub Copilot Workspace represent the state of the art. These are not autocomplete tools — they're engineering agents that can take a GitHub issue description and produce a working PR. SWE-bench Verified scores have gone from ~13% (GPT-4, early 2024) to ~96% (Claude Opus 5, July 2026) — meaning the model now resolves nearly all of the real GitHub issues in this benchmark autonomously, and the benchmark itself is effectively saturated.

✅ Knowledge Check

🃏 Flashcards

The Decision Framework

The most expensive mistake in applied AI is fine-tuning when you shouldn't. The second most expensive is not fine-tuning when you should. Here's the framework that has held up across hundreds of production deployments:

Supervised Fine-Tuning (SFT)

SFT is the bread-and-butter technique: you provide (input, desired_output) pairs, and the model learns to produce outputs that look like your examples. With LoRA (Low-Rank Adaptation), you only train a small number of adapter parameters — typically 0.1–1% of the full model — making it feasible to fine-tune even large models on a single GPU.

Practical SFT Guidelines

• Data quality > data quantity. 500 high-quality examples beat 5,000 mediocre ones. Every example should be something you'd be proud to ship.
• Diverse examples. Cover edge cases, different input lengths, various difficulty levels. If you only train on easy examples, the model will struggle on hard ones.
• LoRA rank matters. Rank 8–16 for behavior changes (format, tone). Rank 32–64 for domain-specific knowledge integration. Higher rank = more parameters = more data needed.
• Validation set: Hold out 10–20% of your data. If training loss drops but validation loss increases, you're overfitting.

Alignment Techniques Compared

LoRA: The Practitioner's Guide

Low-Rank Adaptation (LoRA) has become the default fine-tuning method for production teams. Instead of updating all model parameters, LoRA decomposes the weight updates into low-rank matrices, reducing trainable parameters by 100–1000×.

Key practical details:

• QLoRA (quantized LoRA): Fine-tune a 4-bit quantized model with LoRA adapters in full precision. This lets you fine-tune a 70B model on a single 48GB GPU. Quality loss from quantization is minimal for most tasks.
• LoRA merging: After training, merge the adapter back into the base model for zero-overhead inference. No latency penalty compared to the base model.
• Multi-LoRA serving: Frameworks like vLLM and SGLang support serving multiple LoRA adapters on the same base model — route to different adapters based on task type. One base Llama 4 model with 10 LoRA adapters for 10 different use cases.

Fine-Tuning with Provider APIs

If you don't want to manage GPUs, both OpenAI and Google offer fine-tuning through their APIs. Anthropic doesn't offer public fine-tuning for Claude (as of July 2026) — they provide fine-tuning as a managed service for enterprise customers. OpenAI's fine-tuning API supports the smaller GPT-5.6 tiers and is well-documented. Google's Vertex AI supports fine-tuning Gemini models with standard SFT and RL-based approaches.

✅ Knowledge Check

🃏 Flashcards

RAG in 2026: Mature but Misunderstood

Retrieval-Augmented Generation (RAG) is the most deployed AI pattern in enterprise — and also the most poorly implemented. The concept is simple: retrieve relevant documents, stuff them into the prompt, let the model answer based on them. The execution is full of traps.

The landscape has evolved significantly. Naive RAG (embed → retrieve top-K → prompt) has given way to sophisticated pipelines that look more like traditional search engineering than pure ML. The best RAG systems in 2026 combine semantic search, keyword search, re-ranking, query expansion, and intelligent chunking.

The RAG Stack

The Chunking Problem (Solved?)

Chunking remains the most impactful and least glamorous part of RAG. The right chunking strategy can improve retrieval quality more than switching embedding models. The state of the art in 2026:

• Contextual chunking: Anthropic published a technique called "Contextual Retrieval" — before embedding each chunk, prepend a short context description (generated by a cheap model) that explains where this chunk sits in the overall document. This dramatically improves retrieval for chunks that are only meaningful in context.
• Late chunking: Embed the entire document first, then split embeddings at chunk boundaries. This preserves cross-chunk relationships that are lost in traditional chunk-then-embed approaches.
• Hierarchical chunking: Maintain chunks at multiple granularities (paragraph, section, document). Retrieve at the paragraph level, then expand to include surrounding context. This gives the best of both precision and recall.

GraphRAG

Microsoft's GraphRAG introduced a genuinely new pattern: instead of retrieving raw text chunks, build a knowledge graph from your documents (entities, relationships, summaries at multiple levels), then retrieve from the graph. This excels at global questions ("What are the main themes across our course catalog?") that traditional RAG handles poorly because the answer spans many documents.

The trade-off: GraphRAG requires significant upfront processing (building the knowledge graph costs ~$X per document in LLM calls) and doesn't handle rapidly changing data well. For a Coursera course catalog that changes weekly, you'd need incremental graph updates — which the tooling is still maturing on.

MCP as an Alternative to RAG

Here's the provocative question: do you even need RAG? For many use cases, MCP (Model Context Protocol) offers a simpler alternative. Instead of pre-computing embeddings and maintaining a vector database, you give the agent a tool that can query your data directly.

The pattern: define an MCP tool called search_courses that takes a query string and returns relevant results from your existing search infrastructure (Elasticsearch, Algolia, whatever you already have). The LLM decides when to search, what to search for, and how to combine results. No embedding pipeline, no vector DB, no chunking strategy.

When to prefer MCP over RAG:

• Your data already has good search infrastructure
• Data changes frequently (daily or more)
• The model needs to make targeted queries rather than passively having context
• You're building an agentic system anyway

When to prefer RAG:

• You need to process the entire knowledge base against every query (e.g., finding all relevant courses)
• Latency matters — pre-computed embeddings are faster than real-time search
• You need semantic similarity that keyword search can't provide
• The model needs extensive context (multiple long documents) to answer well

The Long-Context Escape Hatch

With Gemini offering 1M+ token context windows and Claude at 200K, a legitimate question is whether you need RAG at all for moderate-sized corpora. If your entire knowledge base fits in 200K tokens (~150K words), you can just stuff it all into the prompt.

This is not as absurd as it sounds. For a course catalog of 200 courses with short descriptions, the entire catalog might be 50K tokens. Prompt caching means you pay full price once and then 90% less on subsequent queries. The downside: latency increases with context length (roughly linearly), and very long contexts can dilute the model's attention on the most relevant sections.

✅ Knowledge Check

🃏 Flashcards

The Safety Landscape Has Matured

AI safety has evolved from an abstract concern to an engineering discipline with concrete tools, standards, and regulations. The conversation has shifted from "will AI destroy humanity?" to "how do we prevent a student from tricking the AI tutor into writing their essay?" Both matter, but the latter is what you'll deal with daily.

Constitutional AI (Anthropic)

Anthropic's Constitutional AI (CAI) is the most influential alignment technique to emerge from industry research. The core idea: instead of training the model on human feedback for every possible scenario, give it a set of principles (a "constitution") and train it to self-critique and revise its outputs to comply with those principles.

In practice, CAI training has two phases:

• SL-CAI: Generate responses, have the model critique them against the constitution, revise, and fine-tune on the revisions.
• RL-CAI: Use the constitution-trained model as its own reward model for RLHF, reducing reliance on human annotators.

For application developers, the takeaway is that Claude's safety behavior is principled and predictable — it follows rules rather than pattern-matching on "dangerous-sounding" words. This means fewer false positives on legitimate educational content (a chemistry course discussing reactions won't trigger safety filters) and more robust protection against actual misuse.

Prompt Injection: The Unsolved Problem

Prompt injection remains the most serious security vulnerability in LLM applications in 2026. Despite significant research, there is no complete solution. The attack surface is simple: user-provided content (or content retrieved from external sources) contains instructions that override the system prompt.

Attack Categories

• Direct injection: User writes "Ignore previous instructions and..." in their input. Basic, easily caught.
• Indirect injection: Malicious instructions embedded in documents, web pages, or databases that the model retrieves. Much harder to catch because the model doesn't know which content is trusted.
• Multi-turn escalation: Gradually shifting the model's behavior over multiple conversation turns. Each individual message looks benign; the cumulative effect is a jailbreak.

Defense Strategies (State of the Art)

The EU AI Act

The EU AI Act is now partially in force (as of August 2025 for prohibited practices, with the full regime phasing in through 2027). For an education platform like Coursera operating in the EU, the key provisions:

• High-risk classification: AI systems used for educational assessment and access to education are classified as high-risk. This means mandatory conformity assessments, risk management systems, data governance requirements, and human oversight obligations.
• Transparency requirements: Users must be informed when they're interacting with an AI system. If AI is used for grading, this must be disclosed.
• General-Purpose AI (GPAI): Frontier model providers (Anthropic, OpenAI, Google) have obligations around documentation, copyright compliance, and systemic risk assessment. This is their problem, not yours — but it affects model availability and features in the EU.
• Prohibited practices: Social scoring, real-time biometric identification, and emotion recognition in educational settings are prohibited. If you were considering sentiment analysis of student videos for engagement metrics, reconsider.

Practical Safety for Education AI

The safety concerns specific to education:

• Academic integrity: Students using AI to complete assignments. Detection tools (GPTZero, Turnitin AI detection) have ~85% accuracy with a 10–15% false positive rate — unacceptable for high-stakes decisions. Better approach: redesign assessments to be AI-resistant (oral exams, process portfolios, collaborative projects) or AI-embracing (explicitly allow AI use and grade the student's ability to direct it).
• Bias in assessment: AI grading systems can exhibit bias correlated with writing style, dialect, or cultural references. Regular bias audits across demographic groups are essential, not optional.
• Hallucination in tutoring: An AI tutor that confidently teaches wrong information is worse than no AI tutor. For factual domains, always ground the model's responses in verified course materials (RAG), and display confidence levels to students.
• Data privacy: Student data is protected under FERPA (US) and GDPR (EU). Ensure no student PII is sent to model providers, or use providers with BAA/DPA agreements and data processing addendums.

✅ Knowledge Check

🃏 Flashcards

The Capital Stack

The AI industry has absorbed more capital in 2024–2026 than any technology sector in history. The numbers are staggering and worth understanding because they shape what's available to you as a buyer.

Strategic Analysis by Company

OpenAI: The Platform Play

OpenAI's strategy has shifted from "build the best model" to "build the platform." ChatGPT has 400M+ weekly users. The enterprise product (ChatGPT Enterprise/Team) is growing rapidly. The API business, while large in absolute terms, is increasingly a means to attract developers who then upsell their customers to ChatGPT Plus.

The risk: OpenAI is trying to be everything — model lab, consumer product, enterprise platform, and app store (GPTs). This works while they're the default choice, but as model quality converges across providers, the "everything" strategy means they're fighting on many fronts simultaneously. Their transition from nonprofit to for-profit has also created governance concerns that enterprise buyers (especially in regulated industries like education) should monitor.

Anthropic: The Enterprise Trust Play

Anthropic has positioned itself as the "serious" AI company — safety-focused, enterprise-oriented, and developer-friendly. Claude Code has become a genuine hit among developers, and Claude's performance on coding and agentic tasks has made Anthropic the preferred API for many tech companies.

The strategic insight: by investing heavily in safety research and Constitutional AI, Anthropic has built trust as a competitive moat. In regulated industries (finance, healthcare, education), trust in your AI provider is not a nice-to-have — it's a procurement requirement. Anthropic's safety credentials are a genuine differentiator, not just marketing.

Google: The Infrastructure Advantage

Google's AI strategy is harder to parse because it's embedded in a $2T+ company with distribution advantages no startup can match. Gemini is integrated into Google Search, Gmail, Docs, Android, and Cloud. They don't need to win on model quality — they need to be "good enough" while leveraging distribution.

For buyers: Google is the infrastructure play. If you're on GCP, Gemini integration is nearly frictionless. The 1M+ token context window is a genuine differentiator for document-heavy workloads. But the API reliability and developer experience still lag Anthropic and OpenAI.

Meta: The Open-Source Kingmaker

Meta's open-weight strategy with Llama is not altruism — it's a strategic play to commoditize the model layer (where Meta doesn't make money) and drive value to the application layer (where Meta's products live). Llama 4 has fragmented the commercial model market and given enterprises a credible self-hosting option.

The result: Meta has more influence over the AI ecosystem than its revenue from AI would suggest. Every company that self-hosts Llama is a company that isn't paying OpenAI or Anthropic. This competitive pressure keeps API prices falling — which benefits everyone, including Meta's own AI-powered products.

The Compute Bottleneck

The single biggest constraint on AI progress in 2026 is not algorithmic — it's compute. NVIDIA's H100/H200/B200 GPUs remain in short supply, and the pricing reflects it. Training frontier models requires $100M–$1B+ in compute. Serving them at scale requires thousands of GPUs.

This has created a "GPU-rich vs. GPU-poor" divide. Google, Meta, and Microsoft (via OpenAI) have massive compute arsenals. Anthropic has secured significant capacity through Amazon (AWS) partnership. Startups without GPU access are increasingly unable to train competitive frontier models — which is why the number of frontier labs has not grown despite the billions flowing into the sector.

The Application Layer

The real money is moving to the application layer. Notable patterns:

• Vertical AI SaaS: Companies building domain-specific AI products (Harvey for legal, Abridge for medical, Cursor/Windsurf for coding) are reaching $100M+ ARR faster than any previous SaaS cohort. The model is a commodity input; the value is in the domain-specific data, workflows, and UX.
• AI-native education: Companies like Khanmigo (Khan Academy + GPT), Duolingo Max (with Gemini), and various startups are building AI-first learning experiences. This is Coursera's competitive landscape — and the window to build a defensible AI-native product is 12–18 months.
• Developer tools: The "AI for developers" category (Cursor, GitHub Copilot, Claude Code, Windsurf) has matured from experimental to essential. Most professional developers now use AI assistance daily. The market is large (~$10B annually) and growing.

✅ Knowledge Check

🃏 Flashcards

The Scaling Laws Debate

The "scaling laws" hypothesis — that model capabilities improve predictably with more compute, data, and parameters — drove the industry from 2020 to 2024. The hypothesis was largely correct: GPT-2 → GPT-3 → GPT-4 showed smooth, predictable improvements on benchmarks as scale increased.

But in 2025, the narrative cracked. Several data points suggest we're hitting diminishing returns on pre-training scale alone:

• Data wall: We've approximately exhausted the public internet for high-quality training text. Synthetic data (model-generated) is being used to supplement, but it introduces quality ceilings and potential model collapse.
• Benchmark saturation: Frontier models score 90%+ on most standard benchmarks, making it hard to demonstrate improvement from additional scale.
• Cost constraints: Training runs have gone from $10M (GPT-4) to $100M+ (GPT-5), with diminishing quality improvements per dollar.

However, the response has been clever: instead of abandoning scale, labs have shifted to test-time compute scaling. The o-series models, Claude's Extended Thinking, and DeepSeek-R1 show that you can get significant quality improvements by spending more compute at inference time (reasoning) rather than training time. This is a fundamental paradigm shift — and it's why reasoning models are the most important development of 2025–2026.

The AGI Question

Every frontier lab has published timelines suggesting AGI (however they define it) is 2–5 years away. These claims are worth neither dismissing nor taking at face value.

What's actually happening: capabilities are expanding in breadth faster than in depth. Models can now do a wider range of tasks competently, but they still struggle with:

• Novel reasoning: Tasks requiring genuine insight that wasn't in the training distribution. Models are excellent at recombining existing knowledge but rarely generate truly novel solutions.
• Long-horizon planning: Tasks requiring consistent execution over hours or days. Agentic loops help, but error accumulates over many steps.
• Self-knowledge: Models still hallucinate about their own capabilities and confidently attempt tasks they cannot do. Calibration has improved but remains imperfect.
• Robust generalization: Performance on distribution-shifted versions of "solved" tasks reveals that some capabilities are more brittle than benchmarks suggest.

The pragmatic view: whether AGI arrives in 2028 or 2035 doesn't change what you should build today. The current generation of models is transformatively useful — good enough to automate significant portions of knowledge work, including education. Focus on capturing value with today's capabilities while building architecture flexible enough to absorb capability jumps.

The Future of Education AI

AI will transform education more profoundly than any technology since the printing press. This is not hype — the capabilities are already here for most of the following, and the remaining gaps will close within 12–24 months:

Near-Term (2026–2027)

• AI tutoring at scale: One-on-one tutoring is the gold standard of education (Bloom's 2-sigma problem). AI tutors powered by current models can provide 80% of the benefit at 0.1% of the cost. The key is grounding them in verified course content (RAG) and designing interactions that promote active learning rather than passive answer-seeking.
• Automated assessment: Beyond multiple choice. AI can now grade essays, code submissions, design projects, and mathematical proofs with human-level accuracy when given clear rubrics and reference materials. The constraint is not capability but trust — instructors need to see and verify AI grading before they'll adopt it.
• Adaptive learning paths: Models that understand a student's knowledge state and dynamically adjust content difficulty, pacing, and modality. The technology is ready; the content pipeline (generating variations of explanations, practice problems, and examples) is the bottleneck.
• Content generation: AI-generated practice problems, case studies, and worked examples. Not replacement of expert-authored core content, but multiplication of supplementary materials.

Medium-Term (2027–2029)

• AI teaching assistants: Always-available TAs that handle 80% of student questions, escalating to human instructors for complex or sensitive issues. This is already being piloted at several universities.
• Personalized credentialing: AI-powered skill assessment that goes beyond course completion to verified competency demonstration. The model observes you solving problems, evaluates your approach, and certifies specific skills.
• Multimodal learning: Courses that seamlessly combine text, interactive simulations, voice tutoring, and video — all orchestrated by AI based on the student's learning preferences and progress.

Long-Term (2029+)

• AI-designed curricula: AI systems that analyze labor market data, employer needs, and learner outcomes to design optimal learning paths. The human role shifts from content creator to content curator and quality assurer.
• Continuous assessment: Assessment becomes indistinguishable from learning. Every interaction with the AI tutor is an assessment opportunity — no more "study then test" cycles.

What to Build Now

For a VP of Engineering at an education company, the action items are clear:

1. Build the AI infrastructure layer: Multi-model routing, prompt management, eval frameworks, MCP-based tool integration. This is the foundation everything else depends on.
2. Start with tutoring and assessment: These have the highest impact and are most feasible with current technology. Use RAG to ground the AI in your course content. Use extended thinking for rubric-based grading.
3. Invest in eval: Build task-specific eval suites for every AI-powered feature. Without evals, you're flying blind. Plan to spend 20–30% of your AI engineering effort on evaluation and monitoring.
4. Hire for the right skills: You need ML engineers who understand both traditional software engineering and model behavior. The hardest skill to find: people who can design good prompts, build robust eval suites, and debug when the model does unexpected things.
5. Plan for regulation: EU AI Act compliance for educational AI is non-trivial. Start the conformity assessment process now — it takes 6–12 months and will only get more demanding.

✅ Knowledge Check

🃏 Flashcards

The Open-Weight Landscape Has Fundamentally Shifted

Let's dispense with the polite framing: the narrative that open-weight models are perpetually six months behind frontier closed models is dead. Not dying. Dead. By mid-2026, open-weight models are leading on multiple benchmarks that matter to production engineering teams, and the gap on the rest has collapsed to noise.

The numbers tell the story clearly. GLM-5.2 from Z.ai scored 62.1 on SWE-bench Pro — a benchmark that measures real-world software engineering capability — versus GPT-5.5's 58.6. DeepSeek V4 Pro, a 1.6 trillion parameter MoE model with 49B active parameters, ships under an MIT license with a 1 million token context window. Meta's Llama 4 Scout claims 10 million tokens of context. These aren't research previews or limited betas. They're production-ready, commercially licensable, and you can download the weights right now.

The strategic context matters here. US export controls on advanced AI chips (the October 2022 and subsequent rounds of restrictions) were intended to slow China's AI progress. The unintended consequence has been to accelerate China's open-source AI strategy. If you can't buy the best chips, you optimize your software ruthlessly and give it away — building ecosystem lock-in through ubiquity rather than API margins. Z.ai (formerly Zhipu AI), DeepSeek, and Alibaba's Qwen team have all embraced this playbook. The result is that the most permissively licensed, most competitive open-weight models are now disproportionately coming from Chinese labs.

For engineering leaders, this shift demands a strategic reassessment. If your 2025 AI strategy was "use the best API, fine-tune open models for non-critical paths," your 2026 strategy needs to account for the possibility that the best model for your use case is open-weight, self-hostable, and costs a fraction of the API alternative.

GLM-5.2: The Breakout Model

GLM-5.2 was released on June 13, 2026, by Zhipu AI (rebranded as Z.ai) under the MIT license. It is, by several important measures, the most capable open-weight model ever released — and its architecture contains genuinely novel ideas worth understanding.

Architecture & Specifications

• Total parameters: 753 billion (sparse Mixture of Experts)
• Active parameters per token: ~40 billion
• Context window: 1 million tokens (quadrupled from GLM-5.1's 200K)
• Maximum single-turn output: 128K tokens
• License: MIT — no usage restrictions, no MAU caps, fully commercial
• Availability: Full weights on Hugging Face, no regional restrictions

The headline architectural innovation is IndexShare sparse attention. The core problem with long-context models is that standard attention scales quadratically with sequence length. GLM-5.2's approach: run a full indexer pass once every 4 layers to identify the most relevant token indices, then reuse those selected indices for the following 3 layers before re-indexing. This yields a 2.9× FLOP reduction at 1M context length compared to standard sparse attention, without the quality degradation you'd see from simpler fixed-pattern sparsity. It's an elegant solution — you pay the full attention cost periodically to maintain quality, but amortize it across multiple layers.

Benchmark Performance

These aren't cherry-picked numbers. GLM-5.2 is leading on coding, agentic, and design benchmarks — the categories that matter most for engineering teams evaluating models for developer tooling, code generation, and autonomous agent workflows.

Pricing & Economics

Even if you choose to use GLM-5.2 via API rather than self-hosting, the economics are striking:

At roughly one-sixth the cost of frontier closed-model APIs — while outperforming them on coding benchmarks — GLM-5.2 fundamentally changes the cost-benefit analysis for any team doing significant LLM inference.

Strategic Significance

GLM-5.2 is China's answer to US AI export controls, and it's a remarkably effective one. The strategy is straightforward: if you can't buy the best chips, make the best software and give it away. An MIT-licensed model with no regional restrictions, no MAU caps, and no competitive-use clauses is a direct play for global ecosystem adoption. Every developer who builds on GLM-5.2 is a developer who isn't locked into an American API provider. The geopolitical implications are significant, but for engineering teams, the practical implication is simpler: you now have a world-class model you can run anywhere, modify freely, and deploy without legal uncertainty.

The Full Open-Weight Roster: Mid-2026

GLM-5.2 is the headline, but the open-weight ecosystem in mid-2026 is remarkably deep. Here's the complete competitive landscape:

Qwen 3.x: The Steady Climb

Alibaba's Qwen team has executed one of the most consistent improvement trajectories in open-weight AI. The Qwen 3 series introduced "thinking mode" — a hybrid reasoning approach where the model can toggle between fast generation and extended chain-of-thought within a single conversation. The 235B MoE variant (22B active) delivers strong multilingual performance across 100+ languages, making it the go-to choice for teams with significant non-English workloads. The evolution through Qwen 3.5 and into 3.6 has steadily improved instruction following, tool use, and coding quality. The 27B dense model in Qwen 3.6 is particularly noteworthy: it's small enough to run on a single consumer GPU while delivering performance competitive with models 5–10× its size. Apache 2.0 licensing throughout means no surprises for commercial deployment.

DeepSeek V4: Hybrid Attention at Scale

DeepSeek V4 Pro's 1.6 trillion parameters make it the largest open-weight model available, but the interesting engineering is in the attention mechanism. V4 uses a hybrid approach: traditional multi-head attention for layers that need maximum expressiveness, combined with multi-latent attention (MLA) — DeepSeek's innovation from V2 — for layers where compressed KV representations suffice. This hybrid design allows V4 Pro to maintain quality at 1M context without the memory explosion that would make pure MHA impractical at this scale. The Flash variant at 284B total / 13B active is the sleeper hit: it delivers 80–90% of V4 Pro's quality at a fraction of the compute cost, making it ideal for high-throughput production workloads where you're doing millions of inference calls per day.

Gemma 4: Google's On-Device Play

Gemma 4 is Google's clearest statement yet on the on-device AI future. At 26B total parameters with only ~4B active (MoE architecture), it's designed to run on mobile SoCs and edge hardware. The Apache 2.0 license is strategic — Google wants Gemma embedded in as many Android apps and IoT devices as possible. Performance per active parameter is exceptional: Gemma 4 at 4B active competes with dense models at 7–10B on standard benchmarks. For teams building mobile-first AI features or deploying to resource-constrained environments (retail kiosks, automotive, industrial IoT), Gemma 4 is currently the strongest option.

Mistral: The EU Sovereignty Card

Mistral's positioning is increasingly about European digital sovereignty. Mistral Large 3 at 675B parameters is competitive with the best open-weight models, but its real value proposition is regulatory: it's a European company, subject to EU data protection law, with weights you can host on European infrastructure. For organizations navigating GDPR, the EU AI Act, and data residency requirements, Mistral offers a compliance story that Chinese and American models can't match. Mistral Small 4 (119B MoE, ~6B active) is their unified model play — combining text, code, vision, and function calling in a single model small enough for cost-effective deployment, with 256K context that exceeds most competitors in its class.

Phi-4: Small Model, Outsized Performance

Microsoft's Phi-4 family continues to push the "small but mighty" thesis. Phi-4 14B is a dense model that punches absurdly above its weight on math and logic benchmarks — it consistently outperforms models 5–10× its size on tasks requiring structured reasoning. The secret is aggressive data curation: Microsoft uses synthetic data generation pipelines that produce training examples specifically designed to teach mathematical reasoning patterns. Phi-4 Mini at 3.8B is purpose-built for edge deployment scenarios where even 7B models are too large. Both ship under MIT license, making them ideal for embedding in commercial products. If your use case is primarily math-heavy (financial modeling, scientific computing, engineering simulation) and you're constrained on compute, Phi-4 14B should be your first evaluation target.

Open Weight vs Open Source: The Distinction That Matters

The industry has been sloppy with terminology, and it's causing real confusion. "Open source" and "open weight" are not the same thing, and the difference has material implications for your engineering and legal teams.

The OSI Definition

In October 2024, the Open Source Initiative (OSI) published the Open Source AI Definition (OSAID) 1.0, establishing formal criteria for what qualifies as "open source AI." The requirements are stringent:

1. Model weights released under a permissive license
2. Training code (the full training pipeline) made available
3. Training data — either the actual dataset or a sufficiently detailed description to reproduce it — made available
4. All components under licenses that permit free use, modification, and redistribution

By this definition, almost none of the models discussed in this module qualify as open source. Llama 4, Qwen 3, DeepSeek V4, GLM-5.2 — they all release weights but not their training data. What we have is overwhelmingly open weight: you get the trained model, but not the recipe to reproduce it from scratch.

License Comparison

Practical Implications

For production deployments, focus on these concrete questions rather than the "open source" label:

• Can you fine-tune? All models listed above: yes.
• Can you redistribute? MIT and Apache 2.0: yes, with attribution. Llama Community: yes, but derivative models inherit the license terms including the MAU cap.
• Can you use commercially without restriction? MIT: yes. Apache 2.0: yes (with patent grant provisions). Llama Community: yes below 700M MAU — above that, you need a separate agreement with Meta.
• What's the attribution requirement? MIT: include the license. Apache 2.0: include the license, state changes, include NOTICE file if present. Llama Community: include Meta's license text.
• Can you use it to train competing models? MIT and Apache 2.0: yes, no restrictions. Llama 4 removed the explicit anti-competition clause from Llama 3, but the custom license still gives Meta's legal team more latitude than standard OSS licenses.

Running Open Models Locally: The 2026 Toolkit

The inference runtime landscape has matured significantly. In 2024, running a model locally meant fighting with dependencies and reading GitHub issues. In 2026, it's a solved problem — the question is which runtime matches your deployment scenario.

Ollama (v0.30+)

Ollama remains the lowest-friction path to running models locally. A single-command install (curl -fsSL https://ollama.com/install.sh | sh) gets you running, and since v0.19 it automatically detects Apple Silicon and uses the MLX backend for optimal performance. The v0.30 release added native agentic tool integration, meaning tools like Claude Code and Codex CLI can use local Ollama models as backends. Model management is trivial: ollama pull glm5.2 downloads the quantized weights and handles configuration automatically.

Best for: Developer prototyping, single-user workloads, getting started with any model in under 5 minutes. Not ideal for: Multi-user production serving — Ollama's request handling is sequential by default, and while it supports concurrent requests in newer versions, it lacks the sophisticated batching and scheduling of production runtimes.

vLLM

vLLM is the production serving standard for GPU-based deployments. Its PagedAttention implementation delivers 16–20× higher concurrent throughput compared to naive serving (and 3–5× vs Ollama in multi-user scenarios) by treating KV cache memory like virtual memory pages — allocating on demand, sharing across requests, and eliminating the fragmentation that kills throughput in other servers. Continuous batching means new requests are absorbed into running batches without waiting, keeping GPU utilization consistently above 80%.

The caveat: vLLM is not production-ready on Apple Silicon as of mid-2026. It's designed for NVIDIA (CUDA) and AMD (ROCm) datacenter GPUs. If you're deploying on Mac hardware, look elsewhere.

Best for: Multi-user production serving on NVIDIA/AMD GPUs, high-throughput API endpoints, teams that need OpenAI-compatible API interfaces. Not ideal for: Apple Silicon deployments, edge/embedded scenarios, teams without GPU ops expertise.

llama.cpp

The C/C++ reference implementation remains the most portable inference engine available. It runs on essentially everything: CUDA, ROCm, Metal, Vulkan, SYCL, and CPU-only. If your target hardware exists, llama.cpp probably supports it. The project pioneered practical quantization formats (GGUF) and provides the most granular control over quantization levels, memory layout, and inference parameters.

Best for: Edge deployment, custom quantization experiments, maximum hardware compatibility, scenarios where you need to run on unusual or constrained hardware. Not ideal for: Teams that want a batteries-included experience — llama.cpp gives you maximum control at the cost of more configuration.

MLX (Apple)

Apple's MLX framework has matured from research curiosity to production-grade runtime since 2025. On M5 silicon, MLX is 30–60% faster than llama.cpp's Metal backend for token generation. The standout feature is Neural Accelerator integration: prompt processing (the "prefill" phase) is 3–4× faster when MLX offloads to Apple's dedicated neural engine, which is otherwise idle during LLM inference. This matters enormously for long-context workloads where prefill dominates latency.

Best for: Apple Silicon production workloads, teams already in the Apple ecosystem, applications where prefill latency matters (long documents, RAG with large retrieval sets). Not ideal for: Cross-platform deployments, teams targeting NVIDIA/AMD GPUs, scenarios requiring maximum community support and tooling breadth.

Runtime Quick Reference

Hardware Guidance

What can you actually run on what hardware? Here's a practical breakdown:

• M2/M3 Ultra with 192GB unified memory: Full 70B-class dense models (Qwen 3 72B, Llama 3.3 70B) at Q6 or higher quantization with room for large KV caches. Excellent for development and light production workloads.
• Single H100 (80GB VRAM): Llama 4 Scout (109B total, 17B active) at int4 quantization. Most 7–14B dense models (Phi-4 14B, Gemma 4, Qwen 3.6 27B with quantization) at full or near-full precision. The sweet spot for single-GPU production serving.
• 8×H100 cluster (640GB total VRAM): GLM-5.2 (753B MoE), DeepSeek V4 Flash (284B), full Llama 4 Maverick (400B) at reasonable quantization. This is the minimum hardware for serving the largest open-weight MoE models.
• Consumer: M3 Max 96GB unified memory: Qwen 3.6 27B dense at Q8, Phi-4 14B at full precision, Gemma 4 (26B MoE) comfortably. Surprisingly capable for individual developer use — you can run genuinely useful models on a laptop.
• Consumer: RTX 4090 (24GB VRAM): Phi-4 14B at Q4, smaller Qwen/Gemma models. Limited but functional for experimentation.

Practical Guidance: When to Self-Host vs API

The existence of excellent open-weight models doesn't mean you should self-host everything. The decision framework is more nuanced than "open = self-host, closed = API." Here are the factors that actually determine the right choice:

Cost Crossover Analysis

Self-hosting has fixed costs (hardware, ops, engineering time) and low marginal costs per token. APIs have zero fixed costs and constant marginal costs. The crossover point depends on your volume:

• Llama-class models (70B-400B) on 8×H100: Cost crossover at approximately 100M tokens/month. Below that, the API is cheaper when you factor in ops burden. Above that, self-hosting saves 40–70% depending on utilization.
• GLM-5.2 via API: At ~$1.40/$4.40 per million tokens, the crossover point is much higher because the API is already so cheap. You'd need to be doing 500M+ tokens/month before self-hosting on an 8×H100 cluster makes economic sense. For most teams, the GLM-5.2 API is the right answer.
• Small models (7B-14B) on single GPU: Crossover at approximately 20–30M tokens/month. Small models on modest hardware are cheap to self-host, so the crossover comes quickly.

Data Sovereignty

This is the strongest non-economic argument for self-hosting. If you're in healthcare (HIPAA), finance (SOC 2, PCI-DSS), or operating under GDPR with strict data residency requirements, self-hosting may be a requirement, not a choice. Key considerations:

• Can your data leave your infrastructure at all? If not, self-hosting is mandatory.
• Does the API provider offer data processing agreements (DPAs) and data residency guarantees? Many do, but the terms vary.
• What are the audit and logging requirements? Self-hosted gives you complete control over logging. API providers may not give you the granularity your compliance team needs.

Latency

Self-hosted inference can be faster than API-based inference for streaming workloads, because you eliminate the network round-trip and don't contend with other users for GPU time. If your application is latency-sensitive (real-time coding assistance, interactive chat, robotics), and you have adequate hardware, self-hosting can reduce time-to-first-token by 50–200ms compared to API calls.

Conversely, API providers maintain warm model caches and handle cold-start automatically. Self-hosted solutions require you to manage model loading, which on large MoE models can take 30–60 seconds.

Maintenance Cost

This is where teams consistently underestimate self-hosting costs:

• Model updates: When GLM-5.3 drops, someone needs to evaluate it, validate it against your test suite, handle the weight download (hundreds of GB), and manage the rollout. API providers do this for you.
• Infrastructure ops: GPU driver updates, CUDA version compatibility, cooling/power management in datacenters, monitoring and alerting. This requires dedicated ML infrastructure engineers.
• Reliability: What's your uptime target? 99.9% on self-hosted GPU infrastructure is hard. API providers (with SLAs) give you this for free.
• Scaling: Traffic spikes require either over-provisioned hardware (expensive) or auto-scaling infrastructure (complex). APIs handle this natively.

The Hybrid Approach

The most sophisticated teams in 2026 are running hybrid architectures:

• Self-hosted for steady-state: Run a right-sized GPU cluster for your baseline inference volume. Optimize for cost and latency on predictable workloads.
• API for burst and experimentation: Route overflow traffic to API providers during spikes. Use APIs to evaluate new models before committing to self-hosting them.
• Model routing: Use a lightweight router (based on task complexity, latency requirements, or cost constraints) to direct requests to the optimal backend. Simple classification tasks go to a self-hosted Phi-4 Mini. Complex coding tasks go to GLM-5.2 API. Long-context analysis goes to self-hosted DeepSeek V4 if you have the hardware, API if you don't.

This hybrid approach captures most of the cost savings of self-hosting while maintaining the flexibility and reliability of API access. The key enabler is that open-weight models give you the option to self-host — even if you don't exercise it today, having that option is strategically valuable as a hedge against API price changes, service disruptions, or policy shifts.

✅ Knowledge Check

🃏 Flashcards