AI Infrastructure at Scale

Why Quantization Matters

A 70B parameter model at FP16 requires ~140GB VRAM — that's 2x A100 80GB just to load. Quantize to INT4 and it fits on a single A100 with room for a 4K context KV cache. Quantization is the single biggest lever for reducing inference cost.

Quantization Formats Compared

Quality Benchmarks — Real Numbers

Internal benchmarks on Llama 3.1 70B across standard evals. Your mileage will vary by task — always benchmark on YOUR data.

When Quantization Is Worth It

Practical: Quantizing with AutoAWQ

# Quantize Llama 3.1 70B to AWQ INT4
# Requires ~160GB RAM + 80GB GPU for calibration

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Llama-3.1-70B-Instruct"
quant_path = "llama-3.1-70b-instruct-awq"

# Calibration config
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM"     # Use GEMM for vLLM compat
}

model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

# 128-512 samples of real production prompts
calib_data = load_calibration_data()

model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_data)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

The Caching Landscape

Caching is the highest-ROI optimization in AI infrastructure. Unlike traditional web caching, LLM caching operates at multiple layers — from exact response dedup to KV cache reuse for shared prefixes.

Anthropic Prompt Caching — The Biggest Win for API Users

If you're using Claude, prompt caching is the single most impactful cost optimization. Cached input tokens cost 90% less. For applications with long system prompts or repeated context (course materials, rubrics, knowledge bases), this is transformative.

# Python - Anthropic SDK with prompt caching
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_RUBRIC_TEXT,   # 4000+ tokens of grading rubric
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": student_essay}
    ]
)

# Check cache performance
usage = response.usage
print(f"Cache read: {usage.cache_read_input_tokens} tokens")
print(f"Cache miss: {usage.cache_creation_input_tokens} tokens")

Cost Math: Prompt Caching at Coursera Scale

Prefix Caching — Self-Hosted

vLLM's --enable-prefix-caching reuses KV cache entries when requests share the same prefix (system prompt). This is the self-hosted equivalent of Anthropic's prompt caching.

The Batch API — 50% Off for Non-Urgent Work

Anthropic's Batch API processes requests asynchronously at 50% cost. For offline tasks — grading backlogs, content generation pipelines, bulk classification — this is pure savings.

# Batch API usage - process 10K essays overnight
import anthropic

client = anthropic.Anthropic()

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"essay-{i}",
            "params": {
                "model": "claude-sonnet-4-20250514",
                "max_tokens": 1024,
                "messages": [{"role": "user", "content": essay}]
            }
        }
        for i, essay in enumerate(essays[:10000])
    ]
)
# Cost: 50% of standard pricing. Completes in 15min-24hr.

Semantic Caching — Application Layer

For high-volume applications with repetitive queries (FAQ bots, tutoring on popular topics), semantic caching at the application layer can eliminate LLM calls entirely for ~20-40% of queries.

# Simple semantic cache with Redis
import hashlib, json, redis

r = redis.Redis()

def cached_completion(prompt, model):
    cache_key = hashlib.sha256(
        json.dumps({"prompt": prompt, "model": model}).encode()
    ).hexdigest()

    cached = r.get(f"llm:{cache_key}")
    if cached:
        return json.loads(cached)  # Hit! Zero cost.

    result = call_llm(prompt, model)
    r.setex(f"llm:{cache_key}", 3600, json.dumps(result))
    return result

GPU Hardware Landscape (2025)

Cloud Provider Comparison

Spot vs Reserved — The Cost Decision

Multi-GPU Inference — Parallelism Strategies

# Terraform - Provision GPU cluster on AWS

resource "aws_instance" "inference_gpu" {
  count         = 4
  ami           = "ami-0abcdef1234567890"  # NVIDIA Deep Learning AMI
  instance_type = "p4d.24xlarge"           # 8x A100 80GB

  root_block_device {
    volume_size = 500
    volume_type = "gp3"
  }

  tags = {
    Name        = "inference-node-${count.index}"
    Environment = "production"
    Team        = "ai-platform"
  }

  # Placement group for low-latency NVLink/EFA
  placement_group = aws_placement_group.inference.id
}

resource "aws_placement_group" "inference" {
  name     = "inference-cluster"
  strategy = "cluster"
}

# Auto Scaling Group for spot-based burst capacity
resource "aws_autoscaling_group" "inference_spot" {
  desired_capacity = 0
  max_size         = 8
  min_size         = 0

  mixed_instances_policy {
    instances_distribution {
      on_demand_percentage_above_base_capacity = 0
      spot_allocation_strategy = "capacity-optimized"
    }
    launch_template {
      launch_template_specification {
        launch_template_id = aws_launch_template.gpu_spot.id
      }
      override { instance_type = "p4d.24xlarge" }
      override { instance_type = "p4de.24xlarge" }
    }
  }
}

Reference Architecture — Load Balancing AI Inference

Multi-Model Routing — The Smart Scaling Pattern

Not every request needs your most powerful (and expensive) model. A routing layer that classifies request complexity can cut costs 40-60% by sending simple tasks to cheap models.

# Model router - classify and route requests

from enum import Enum

class ModelTier(Enum):
    FAST   = "fast"     # Haiku / Llama 8B  - $0.25/MTok
    SMART  = "smart"    # Sonnet / Llama 70B - $3/MTok
    BEST   = "best"     # Opus - $15/MTok

def route_request(request) -> ModelTier:
    task = request.task_type

    # Classification, translation, autocomplete -> cheap model
    if task in ["classify", "translate", "autocomplete", "summarize_short"]:
        return ModelTier.FAST

    # Essay grading, content generation -> capable model
    if task in ["grade_essay", "generate_content", "explain_concept"]:
        return ModelTier.SMART

    # Curriculum design, complex reasoning -> best model
    if task in ["curriculum_design", "research_synthesis"]:
        return ModelTier.BEST

    return ModelTier.SMART  # default

Auto-Scaling Configuration

# Kubernetes HPA for vLLM inference pods

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-inference-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama-70b
  minReplicas: 2          # Always-on baseline
  maxReplicas: 8          # Peak capacity
  metrics:
    - type: Pods
      pods:
        metric:
          name: vllm_pending_requests   # Custom metric from vLLM
        target:
          type: AverageValue
          averageValue: 50             # Scale up when queue > 50
    - type: Pods
      pods:
        metric:
          name: gpu_utilization
        target:
          type: AverageValue
          averageValue: 80             # Scale at 80% GPU util
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 2
          periodSeconds: 120
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 300

Request Queuing & Rate Limiting

What to Measure

LLM observability is different from traditional service monitoring. Latency percentiles matter more than averages, cost is a first-class metric, and "quality" is something you need to track but can't directly measure from metrics alone.

Tracing LLM Calls

# Structured logging for LLM calls
import json, logging

logger = logging.getLogger("llm_trace")

class LLMTracer:
    def trace_call(self, request, response, timing):
        trace = {
            "trace_id":       request.trace_id,
            "model":          response.model,
            "task_type":      request.task_type,
            # Token counts
            "input_tokens":   response.usage.input_tokens,
            "output_tokens":  response.usage.output_tokens,
            "cached_tokens":  response.usage.cache_read_input_tokens,
            # Timing
            "queue_time_ms":  timing.queue_ms,
            "ttft_ms":        timing.ttft_ms,
            "generation_ms":  timing.generation_ms,
            "total_ms":       timing.total_ms,
            # Cost
            "cost_usd":       self.calc_cost(response),
            "cache_hit":      response.usage.cache_read_input_tokens > 0,
        }
        logger.info(json.dumps(trace))

    def calc_cost(self, response):
        # Claude Sonnet pricing
        P = {"input": 3.0/1e6, "output": 15.0/1e6, "cache": 0.3/1e6}
        u = response.usage
        return (
            (u.input_tokens - u.cache_read_input_tokens) * P["input"] +
            u.cache_read_input_tokens * P["cache"] +
            u.output_tokens * P["output"]
        )

Observability Stack

Error Budgets for LLM Services

The Real Math: Cost Per Query

The most important number in AI infrastructure is cost per query. Everything else (GPU utilization, throughput, model selection) is a lever to move this number. Let's build the cost model ground-up.

API-Based Cost Model (Claude Sonnet)

Self-Hosted Cost Model (vLLM + Llama 70B AWQ)

API vs Self-Hosted Breakeven Analysis

Cost Optimization Levers — Ranked by Impact

Cost Per Student — The Business Metric

Reference Architecture: AI-Powered EdTech Platform

Streaming Responses — Critical for UX

For interactive AI features (tutoring, Q&A), streaming is mandatory. Users perceive 200ms TTFT with streaming as "instant" but 3s wait-then-dump as "slow" — even if total time is identical.

# Server-Sent Events (SSE) streaming endpoint

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import anthropic, json

app = FastAPI()

async def stream_response(prompt: str):
    client = anthropic.Anthropic()
    with client.messages.stream(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    ) as stream:
        for text in stream.text_stream:
            yield f"data: {json.dumps({'text': text})}\n\n"
    yield "data: [DONE]\n\n"

@app.post("/api/ai/stream")
async def ai_stream(request: Request):
    body = await request.json()
    return StreamingResponse(
        stream_response(body["prompt"]),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",  # Disable nginx buffering
        }
    )

Failover Strategy

# Failover implementation with retries

class AIService:
    FAILOVER_CHAIN = [
        {"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
        {"provider": "self_hosted", "model": "llama-70b-awq"},
        {"provider": "anthropic", "model": "claude-haiku-4-5-20251001"},
    ]

    async def complete(self, request):
        for i, target in enumerate(self.FAILOVER_CHAIN):
            try:
                response = await self._call_model(target, request)
                if i > 0:  # Log failover event
                    self.metrics.increment("failover_count",
                        tags={"from": self.FAILOVER_CHAIN[0]["model"],
                              "to": target["model"]})
                return response
            except (RateLimitError, TimeoutError, ServiceUnavailable):
                continue

        # All providers failed
        return await self.graceful_degradation(request)

Multi-Region Deployment

Docker Compose — Full Local Dev Stack

# docker-compose.yml - Development AI platform stack

version: "3.8"
services:
  ai-router:
    build: ./services/ai-router
    ports: ["8080:8080"]
    environment:
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - VLLM_ENDPOINT=http://vllm:8000
      - REDIS_URL=redis://redis:6379
      - POSTGRES_URL=postgresql://ai:ai@postgres:5432/ai_platform
    depends_on: [redis, postgres]

  vllm:
    image: vllm/vllm-openai:v0.6.4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --max-model-len 4096
      --gpu-memory-utilization 0.90
      --enable-prefix-caching
    ports: ["8000:8000"]

  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    volumes: [redis-data:/data]

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: ai_platform
      POSTGRES_USER: ai
      POSTGRES_PASSWORD: ai
    ports: ["5432:5432"]
    volumes: [pg-data:/var/lib/postgresql/data]

  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./config/prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    volumes:
      - ./config/grafana/dashboards:/var/lib/grafana/dashboards

volumes:
  redis-data:
  pg-data:

Production Checklist