The 28-Day LLM + Data Engineering Deep-Prep Playbook

The Transformer architecture revolutionized NLP by replacing recurrence with attention. Key components: **Self-Attention Mechanism**: Each token attends to all other tokens via Q, K, V matrices. Scaled dot-product attention = softmax(QK^T / sqrt(d_k)) V. Multi-head attention runs this in parallel across different learned subspaces, capturing diverse relationships (syntax, semantics, position). **Positional Encoding**: Since attention is permutation-invariant, sinusoidal or learned embeddings inject position info. Formula: PE(pos, 2i) = sin(pos / 10000^(2i/d_model)). **Encoder-Decoder Stack**: Encoder processes input, decoder generates output. Masked self-attention in decoder prevents peeking at future tokens. Cross-attention connects decoder to encoder outputs. **Layer Norm + Residuals**: Each sub-layer (attention, FFN) has residual connections + layer norm. This stabilizes training and enables deep stacking (BERT: 12-24 layers, GPT-3: 96 layers). **Feedforward Networks**: Position-wise 2-layer MLPs (often 4x expansion: d_model=768 → 3072 → 768). ReLU or GELU activations. **Computational Complexity**: Self-attention is O(n^2 * d), where n = sequence length. This becomes a bottleneck for long contexts (hence innovations like FlashAttention, sparse attention, linear attention). **Modern Variants**: GPT uses decoder-only (autoregressive LM), BERT uses encoder-only (masked LM), T5 uses full encoder-decoder. Understanding these trade-offs is key for choosing/fine-tuning models. **Why It Matters for Agents**: Agents often need to reason over long contexts (tool documentation, chat history, code). KV cache optimization, attention patterns, and context window limits directly impact agent performance.

**KV Cache Mechanics**: In autoregressive generation, each token attends to all previous tokens. Without caching, we recompute K and V matrices for the entire history at every step—O(n^2) redundancy. KV cache stores (keys, values) for past tokens, so new token computation is O(n) instead of O(n^2). Trade-off: memory grows linearly with sequence length (e.g., Llama 2 70B at 4K context: ~40GB KV cache). **FlashAttention**: Standard attention materializes the full NxN attention matrix in GPU HBM (slow). FlashAttention uses tiling + kernel fusion to keep intermediate results in SRAM (fast), computing attention in blocks. Result: 2-4x faster, lower memory footprint, enables 10K+ context windows. **FlashAttention-2**: Further optimizations via warp-level parallelism and reduced non-matmul FLOPs. Now the default in PyTorch 2.x (torch.nn.functional.scaled_dot_product_attention). **Other Optimizations**: - **Multi-Query Attention (MQA)**: Share K/V heads across queries (GPT-J, PaLM). Reduces KV cache size by 8-12x. - **Grouped-Query Attention (GQA)**: Middle ground—share K/V within groups (Llama 2, Mistral). Balances quality and efficiency. - **Paged Attention (vLLM)**: Manages KV cache like OS virtual memory—non-contiguous blocks, reduces fragmentation. - **Speculative Decoding**: Use a small draft model to generate candidate tokens, verify with large model in parallel. 2-3x speedup. **Why It Matters**: Your agent's latency and cost are dominated by inference. A 10x speedup from FlashAttention + KV cache = 10x cheaper, 10x faster responses. Critical for real-time interactions. **Hands-On**: Use vLLM or TGI (Text Generation Inference) for serving—both include FlashAttention, paged attention, continuous batching. Benchmark tokens/sec and memory usage.

**Prompt Engineering Foundations**: LLMs are instruction-followers. Small changes in phrasing, format, or examples can drastically change output quality. Key techniques: **Zero-Shot**: Direct instruction ("Translate this to French:"). Works for common tasks but brittle for complex reasoning. **Few-Shot**: Provide 2-5 examples in the prompt. Format matters—consistent structure (Input/Output pairs, Q/A, JSON) helps. Few-shot is powerful but consumes context window. **Chain-of-Thought (CoT)**: Add "Let's think step by step" or show reasoning in examples. Unlocks emergent reasoning in large models (>100B params). Critical for multi-step tasks. **Function Calling (Tool Use)**: LLMs can output structured JSON to invoke external tools. Two approaches: 1. **Native function calling** (GPT-4, Claude 3.5): Model trained to emit tool calls as special tokens. Reliable, fast. 2. **Prompt-based**: Define tools in prompt, parse JSON from response. More flexible but requires robust parsing + validation. **Best Practices**: - **Clear tool schemas**: Use JSON Schema with descriptions, types, required fields. Example: `{"name": "search_web", "parameters": {"query": {"type": "string", "description": "Search query"}}}`. - **Error handling**: LLMs will generate invalid JSON or wrong arguments. Validate, retry with error feedback. - **Constrained decoding**: Use grammars (llama.cpp) or logit biases (OpenAI) to force valid JSON structure. **Agent Patterns**: - **ReAct** (Reasoning + Acting): Model alternates between thought (reasoning), action (tool call), observation (result). Loop until answer found. - **Plan-and-Execute**: Generate full plan upfront, then execute steps. More efficient but less adaptive. **Why It Matters**: Your agent is only as good as its prompt. A well-designed prompt reduces hallucinations, improves reliability, and lowers token costs. Function calling is the bridge between LLMs and the real world. **Hands-On**: Experiment with different CoT styles (explicit steps, implicit, structured). Benchmark success rate on multi-step tasks (e.g., "Book a flight from NYC to SF on June 10, under $300").

**ReAct (Reasoning + Acting)**: Introduced by Yao et al. (2022), ReAct interleaves reasoning and acting. At each step: 1. **Thought**: Model reasons about what to do next ("I need to find the actor who won Best Actor in 2023"). 2. **Action**: Model calls a tool (e.g., search("Oscar best actor 2023")). 3. **Observation**: Tool returns result ("Brendan Fraser"). 4. Loop until "Final Answer". **Advantages**: Flexible, handles unexpected results, human-readable trace. **Disadvantages**: Can loop indefinitely, high token cost, sensitive to prompt quality. **Plan-and-Execute**: Generate a full plan (sequence of steps) before execution. Used in BabyAGI, AutoGPT. **Advantages**: More efficient (fewer LLM calls), clearer intent. **Disadvantages**: Brittle—if a step fails, hard to recover. Works best for well-structured tasks. **Model Context Protocol (MCP)**: Anthropic's standard for connecting LLMs to data sources and tools. Key concepts: - **Resources**: Read-only data (files, databases, APIs). - **Prompts**: Reusable prompt templates. - **Tools**: Functions the model can invoke. - **Sampling**: Model can request another model to generate text (e.g., specialized model for code). **MCP Architecture**: Client-server model. MCP server exposes tools/resources via JSON-RPC. Client (e.g., Claude Desktop, custom agent) connects to multiple servers, model decides which to use. Similar to LSP (Language Server Protocol) for IDEs. **Agent Frameworks**: - **LangChain**: Popular but heavy. Offers chains, agents, memory. Good for prototyping, but abstractions leak. Production systems often outgrow it. - **LlamaIndex**: Focused on RAG + agents. Better for data-centric use cases. - **AutoGPT/BabyAGI**: Early autonomous agents. Interesting ideas but prone to looping, hallucination. - **GPT-Researcher, SWE-agent**: Domain-specific agents (research, coding). Study their prompts—well-engineered examples. **Best Practices**: - **Max iterations**: Cap ReAct loops at 10-15 to prevent runaway costs. - **Fallback**: If agent fails, return error + partial result. Don't let it loop forever. - **Observability**: Log every thought, action, observation. Critical for debugging. **Hands-On**: Build a mini-MCP server that exposes your Google Calendar as a tool. Agent should be able to query events, add new ones.

**Why Fine-Tuning?**: Pretrained LLMs are generalists. Fine-tuning adapts them to specific domains (medical, legal, code) or behaviors (concise, formal, creative). Two main approaches: **supervised fine-tuning (SFT)** and **reinforcement learning from human feedback (RLHF)**. **Parameter-Efficient Fine-Tuning (PEFT)**: Full fine-tuning updates all weights—expensive (100GB+ RAM for 70B model). PEFT updates only a small subset: - **LoRA (Low-Rank Adaptation)**: Inject trainable low-rank matrices into attention layers. Original weights frozen. LoRA learns ΔW = AB where A, B are low-rank (e.g., rank=8). Final output = W + ΔW. Reduces trainable params by 100-1000x. Typical LoRA checkpoint: 10-100MB vs. 140GB for Llama 2 70B. - **QLoRA**: Quantize base model to 4-bit (NF4), train LoRA adapters in 16-bit. Fits 65B model in 24GB GPU. Minimal accuracy loss. Breakthrough for accessibility. **LoRA Best Practices**: - **Target modules**: Apply LoRA to Q, K, V, O projections in attention. Sometimes also to FFN. More modules = better accuracy but larger checkpoint. - **Rank (r)**: Higher rank = more capacity but slower training. Start with r=8 or r=16. Diminishing returns beyond r=64. - **Alpha (α)**: Scaling factor (often 2r). Affects learning rate dynamics. **RLHF (Reinforcement Learning from Human Feedback)**: Used to align models (e.g., InstructGPT, Claude). Three stages: 1. **SFT**: Fine-tune on curated demonstrations (human-written responses). 2. **Reward Model (RM)**: Train a model to predict human preferences. Given (prompt, responseA, responseB), RM learns which is better. 3. **PPO**: Use RM as reward signal to fine-tune SFT model via proximal policy optimization. Model learns to maximize reward while staying close to SFT policy (KL penalty). **Alignment Challenges**: RLHF is expensive (requires lots of human labels), unstable (PPO can diverge), and can cause "reward hacking" (model exploits RM flaws). Alternatives: DPO (Direct Preference Optimization), RLAIF (use AI feedback instead of human). **When to Fine-Tune vs. RAG**: - **Fine-tune**: You have 10K+ examples, need low latency, or want to change model behavior (tone, format). - **RAG**: You have dynamic/changing data (docs, recent events), small datasets, or need citations. - **Hybrid**: Fine-tune on domain + RAG for facts. Example: medical chatbot fine-tuned on clinical dialogue + RAG over PubMed. **Hands-On**: Use Hugging Face TRL (Transformer Reinforcement Learning) + PEFT. Example: Fine-tune Mistral 7B on your GitHub issues to auto-generate responses. Dataset: (issue_title, issue_body) → (your typical response). Evaluate with BLEU, ROUGE, or manual inspection.

**Embeddings 101**: An embedding is a dense vector representation of text (or images, audio). Key property: semantically similar items are close in embedding space (cosine similarity ≈ 1). Example: "dog" and "puppy" have similar embeddings, "dog" and "airplane" are far apart. **Embedding Models**: - **Sentence Transformers**: Fine-tuned BERT/RoBERTa models for sentence embeddings. Popular: `all-MiniLM-L6-v2` (small, fast), `all-mpnet-base-v2` (best quality). Output: 384-768 dim vectors. - **OpenAI `text-embedding-ada-002`**: 1536 dims, good quality, but API cost adds up. $0.0001 per 1K tokens. - **Instruct Embeddings**: New generation (e.g., `instructor-xl`). Takes task instruction + text. Can adapt to different domains without fine-tuning. **Vector Databases**: Store and query embeddings efficiently. Unlike traditional DBs (exact match), vector DBs do **approximate nearest neighbor (ANN)** search. Key players: - **Pinecone**: Fully managed, fast, but $$. Best for production at scale. - **Weaviate/Qdrant**: Open-source, feature-rich. Support hybrid search (vector + keyword), multi-tenancy. - **ChromaDB**: Simple, embeddable (like SQLite). Good for prototyping. - **FAISS (Meta)**: Not a database, but a library for ANN search. Fast, but no persistence/metadata out of the box. Often wrapped by other DBs. **ANN Algorithms**: - **HNSW (Hierarchical Navigable Small World)**: Graph-based. Fast queries, but index build is slow. Used by Weaviate, Qdrant. - **IVF (Inverted File Index)**: Partition space into clusters (Voronoi cells), search nearest clusters. Used by FAISS. - **Product Quantization (PQ)**: Compress vectors to reduce memory. Trade-off: lower precision but 10-100x less memory. **Similarity Metrics**: - **Cosine similarity**: Most common for text. cos(θ) = (A · B) / (||A|| ||B||). Range: [-1, 1]. - **Euclidean distance**: L2 norm. Sensitive to magnitude. - **Dot product**: Fast, but assumes normalized vectors. **Best Practices**: - **Normalize embeddings**: If using dot product, normalize to unit length first. - **Chunking**: Embed paragraphs or sections, not entire docs. Long text → lower quality embeddings. - **Metadata filtering**: Store metadata (date, source, type) alongside vectors. Filter before/after ANN search. **Hybrid Search**: Combine vector search (semantic) + keyword search (BM25, lexical). Reciprocal Rank Fusion (RRF) to merge results. Example: User searches "Python async" → vector search finds semantically similar, BM25 ensures exact keyword match is ranked high. **Hands-On**: Use Qdrant (Docker) + sentence-transformers. Embed 1000 docs, query with "explain transformer architecture", return top-5. Benchmark latency and recall@10 (vs. brute-force).

**RAG (Retrieval-Augmented Generation)**: Combines retrieval (from external knowledge base) with generation (LLM). Pipeline: User query → Retrieve relevant docs → Stuff into prompt → Generate answer. Key benefit: Fresh, accurate info without retraining model. **RAG Architecture**: 1. **Ingestion**: Chunk docs → Embed → Store in vector DB. Chunking strategies: fixed size (512 tokens), semantic (split on paragraphs/sections), recursive (split on headings, then paragraphs). 2. **Retrieval**: Embed query → ANN search → Top-k docs. Typical k=3-10. 3. **Synthesis**: Construct prompt with retrieved docs + query → LLM generates answer. Prompt template: "Context: {docs} Question: {query} Answer:". **Challenges & Solutions**: - **Retrieval quality**: Naive ANN can miss relevant docs. Solutions: **Re-ranking** (use cross-encoder to re-score top-k), **Hybrid search** (vector + BM25), **Query expansion** (rewrite query for better recall). - **Context window limits**: Can't fit all relevant docs. Solutions: **Summarize** long docs, **Two-stage retrieval** (retrieve top-100, re-rank to top-5), **Hierarchical indexing** (retrieve at section level, then paragraph level). - **Hallucination**: LLM may ignore context or fabricate. Solutions: **Constrained generation** (force citations), **Answer verification** (use separate model to check if answer is supported by context). **Advanced RAG Patterns**: - **Self-RAG**: Model decides when to retrieve (not every query needs retrieval). Uses special tokens to trigger retrieval. - **Iterative retrieval**: Retrieve → Generate partial answer → Retrieve more (if needed) → Generate final answer. Used in multi-hop QA. - **FLARE (Forward-Looking Active Retrieval)**: Model generates answer incrementally. If confidence drops (high entropy), pause and retrieve. **Evaluation**: - **Retrieval metrics**: Recall@k (% of relevant docs in top-k), MRR (mean reciprocal rank), NDCG. - **Generation metrics**: BLEU, ROUGE (weak for QA). Better: **Answer correctness** (exact match, F1), **Faithfulness** (is answer supported by context?), **Relevance** (does answer address query?). **Production Tips**: - **Caching**: Embed queries and cache results. 30-50% of queries are duplicates. - **Indexing strategy**: Separate indices for different doc types (code, docs, issues). Filter by type before retrieval. - **Observability**: Log query, retrieved docs, final answer. Essential for debugging retrieval failures. **Hands-On**: Build a RAG agent for your preparation notes. Ingest all Week 1 deep_notes → embed → store in ChromaDB. Agent should answer questions like "What is FlashAttention?" or "Explain LoRA" by retrieving + synthesizing. **Week 1 Retrospective**: You now understand transformers, inference optimization, prompt engineering, agent architectures, fine-tuning, embeddings, and RAG. These are the core building blocks of LLM systems. Week 2 will go deeper into production systems (serving, evals, observability).

**LLM Serving Challenges**: Naive approach (HuggingFace generate()) is slow—single request at a time, no batching, inefficient memory use. Production systems need high throughput (many users) + low latency (each user). **vLLM (UC Berkeley)**: State-of-the-art LLM serving system. Key innovations: - **PagedAttention**: Manages KV cache like OS virtual memory. Non-contiguous blocks reduce fragmentation, enable efficient memory sharing across requests. Result: 2-4x higher throughput. - **Continuous Batching**: Traditional batching waits for all sequences in a batch to finish (slow—bottlenecked by longest sequence). vLLM adds/removes sequences dynamically as they complete. Result: ~10x higher throughput. - **Optimized CUDA Kernels**: Fused attention, custom sampling, minimal Python overhead. **Text Generation Inference (TGI, Hugging Face)**: Production-grade serving. Features: - FlashAttention 2, paged attention, continuous batching (inspired by vLLM). - Tensor parallelism (shard model across GPUs), quantization (GPTQ, AWQ, bitsandbytes). - OpenAPI-compatible HTTP server, Prometheus metrics, distributed tracing. - Streaming support (SSE), token streaming, JSON schema constraints. **Throughput vs. Latency Trade-off**: Large batches → higher throughput (GPU utilization) but higher latency (queuing delay). Small batches → lower latency but lower throughput. Continuous batching helps: achieve high throughput without sacrificing latency. **Benchmarking**: Use `wrk` or `locust` to send concurrent requests. Metrics: - **Throughput**: Requests/sec, tokens/sec (output tokens only, or total). - **Latency**: Time to first token (TTFT), inter-token latency (ITL), end-to-end. - **P50/P95/P99**: Median, 95th, 99th percentile latencies. P99 matters for user experience. **Quantization in Serving**: 4-bit quantization (GPTQ, AWQ) reduces memory 4x, enables larger batch sizes. Minimal accuracy loss (<1% perplexity). AWQ is activation-aware (better than naive round-to-nearest). Both supported in vLLM/TGI. **Tensor Parallelism**: Split model across GPUs (each GPU holds part of each layer). Required for large models (70B won't fit on 1 GPU). Trade-off: inter-GPU communication adds latency. Best for throughput-bound workloads. **When to Use What**: - **vLLM**: Cutting-edge performance, research-friendly. Good for custom models, experimentation. - **TGI**: Production-ready, enterprise support. Better for stability, monitoring, multi-model serving. - **Ollama**: Local desktop use. Easy setup but not optimized for multi-user serving. **Hands-On**: Deploy vLLM with Llama 2 7B. Send 100 concurrent requests (varying lengths: 50-500 tokens). Plot throughput vs. batch size. Compare to baseline (transformers.generate()).

**Chunking Strategies**: How you split documents drastically affects retrieval quality. **Fixed-Size Chunking**: Split every N tokens (e.g., 512). Simple but breaks mid-sentence, mid-paragraph. Poor for semantic coherence. **Semantic Chunking**: Split on natural boundaries: - **Paragraph-level**: Most common. Preserves context within a paragraph. - **Heading-based**: Split on H1, H2, H3. Good for structured docs (code docs, articles). - **Recursive**: Split on headings first, then paragraphs if chunk still too large. Used by LangChain RecursiveCharacterTextSplitter. **Overlap**: Add 10-20% overlap between chunks to avoid losing context at boundaries. Example: chunk 1 = tokens 0-512, chunk 2 = tokens 460-972. **Metadata**: Store chunk metadata (doc_id, section, heading, page_num). Enables filtering ("search only in 'Installation' section"). **Re-Ranking**: ANN retrieval (cosine similarity) is fast but coarse. Re-ranking refines results. **Cross-Encoders**: Encode (query, doc) jointly (vs. bi-encoders which encode separately). More accurate but slower. Typical flow: 1. Bi-encoder retrieval: Query → top-100 candidates (fast, <10ms). 2. Cross-encoder re-ranking: Score each (query, candidate) pair, keep top-5 (slower, ~100ms). **Popular Cross-Encoders**: `ms-marco-MiniLM-L-12-v2` (fast, 12 layers), `ms-marco-electra-base` (better quality). Fine-tuned on MS MARCO passage ranking dataset. **When to Re-Rank**: Always, if latency allows. Re-ranking typically improves Recall@5 by 20-30%. Cost: ~10x slower than bi-encoder, but only for top-k candidates (amortized). **Hybrid Search**: Combine vector search (semantic) + keyword search (lexical). Why? Vector search misses exact keyword matches ("Python 3.11 release notes" → retrieves "Python 3.10" if embeddings are close). BM25 ensures exact match ranks high. **BM25 (Best Match 25)**: TF-IDF variant. Scores based on term frequency, document length normalization. Fast, effective for keyword queries. **Fusion Algorithms**: - **Reciprocal Rank Fusion (RRF)**: Score = Σ 1/(k + rank_i) where rank_i is rank in source i, k=60 (constant). Simple, works well. - **Linear Combination**: α * vector_score + (1-α) * bm25_score. Requires score normalization (min-max or z-score). - **Learned Fusion**: Train a ranker (LightGBM, neural) on (query, doc, vector_score, bm25_score) → relevance. Expensive but best quality. **Implementation**: Weaviate, Qdrant support hybrid search natively. FAISS + Elasticsearch = manual but flexible. **Query Expansion**: Rewrite user query to improve recall. Techniques: - **Synonyms**: "car" → "car OR automobile OR vehicle". - **LLM-based**: "What's the capital of France?" → "capital France Paris city location". - **Multi-query**: Generate 3-5 variations of the query, retrieve for each, merge results. **Evaluation**: Use labeled (query, relevant_docs) dataset. Metrics: Recall@k (what % of relevant docs are in top-k?), MRR (mean reciprocal rank), NDCG (normalized discounted cumulative gain). **Hands-On**: Implement hybrid search with FAISS (vector) + BM25 (via rank_bm25 library). Test on BEIR benchmark (diverse retrieval tasks). Compare vector-only, BM25-only, hybrid (RRF).

**Why Automated Evals Matter**: Manual eval doesn't scale (100+ queries? 1000+?). LLM systems need fast feedback loops. Automated evals enable A/B testing, regression detection, CI/CD for prompts. **RAGAS (Retrieval-Augmented Generation Assessment)**: Framework for RAG evaluation. Four core metrics: **1. Faithfulness** (answer grounded in context?): LLM checks if answer statements are supported by retrieved context. Example: Answer="Paris is the capital of France." Context="France's capital city is Paris..." → Faithful=1.0. If answer hallucinates, Faithful <1.0. Uses NLI (natural language inference) under the hood. **2. Answer Relevancy** (answer addresses query?): Measures semantic similarity between answer and query. Low if answer is correct but off-topic. Example: Query="What's the capital of France?" Answer="France is in Europe and has many cities..." → Low relevancy (doesn't directly state Paris). **3. Context Recall** (retrieved context covers ground truth?): % of ground truth answer that can be inferred from context. High recall = retrieval is good. Example: Ground truth="Paris is the capital; population 2.1M." Context mentions Paris as capital but not population → Recall=0.5. **4. Context Precision** (retrieved context is relevant?): Ranks relevant chunks higher than irrelevant. Measures ranking quality. Example: Top-5 chunks, only #1 and #3 relevant → Precision@5 lower than if #1, #2 relevant. **RAGAS Workflow**: 1. Prepare test set: (query, ground_truth_answer, contexts, generated_answer). 2. Run RAGAS: `evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_recall, context_precision])`. 3. Get scores per query + aggregate metrics. **LLM-as-a-Judge**: Use GPT-4 or Claude to evaluate another LLM's output. Prompt: "Rate the following answer on a scale of 1-5 for accuracy, relevance, and coherence..." LLM outputs structured score + justification. **Best Practices for LLM-as-Judge**: - **Use strong judge model**: GPT-4, Claude 3.5 Sonnet. Weak models (GPT-3.5) are unreliable. - **Structured output**: JSON schema with score + reasoning. Easier to parse, more consistent. - **Rubric**: Provide detailed criteria ("Score 5: Answer is factually correct, directly addresses query, cites sources..."). - **Few-shot examples**: Include 2-3 scored examples in prompt. - **Multiple judges**: Run 3 independent evaluations, take median/majority vote. Reduces variance. **Known Biases**: - **Verbosity bias**: Judges favor longer, more detailed answers even if less accurate. - **Position bias**: Judges prefer answer shown first (if comparing A vs. B, randomize order). - **Self-preference**: GPT-4 rates GPT-4 outputs higher than Claude's (use neutral judge). **Human Evals**: Gold standard but slow. Annotation tools: Label Studio, Argilla. Workflow: Annotators label (query, answer) as correct/incorrect, rate 1-5. Use inter-annotator agreement (Cohen's kappa) to ensure quality. Typical: 100-500 examples for initial calibration, then automated evals. **A/B Testing**: Deploy two variants (e.g., different prompts). Route 50/50 traffic. Measure win rate (which answer users prefer?). Requires user feedback mechanism (thumbs up/down, preference clicks). **Regression Testing**: Eval suite as unit tests. CI/CD runs evals on every prompt change. If score drops >X%, block deploy. Prevents regressions. **Hands-On**: Use RAGAS on your RAG agent. Create test set: 20 queries from your domain (code, docs, whatever). Run baseline, then improve (hybrid search, re-ranking), re-run RAGAS. Track metric improvements.

**Agent Eval Challenges**: Agents are non-deterministic, multi-step, interact with external systems (APIs, DBs, shells). Traditional NLP metrics (BLEU, ROUGE) don't apply. Need task-specific success criteria. **Key Metrics**: **1. Task Success Rate**: Binary—did agent complete task? Example: "Book a flight" → Success if booking confirmed, else failure. Gold standard but requires ground truth or verifiable final state. **2. Partial Credit**: Some tasks have intermediate milestones. Example: "Research + summarize + email" → Credit for research (0.33), summary (0.66), email sent (1.0). More nuanced than binary. **3. Tool Call Accuracy**: Did agent call correct tools with correct args? Example: Task requires calling search("Python 3.11") → Agent calls search("Python") → Incorrect (too broad). **4. Efficiency**: # of tool calls, # of LLM calls, total tokens, wall-clock time. Lower is better (cost, latency). But don't optimize so much that success rate drops. **5. Hallucination Rate**: Agent invents tools/APIs that don't exist, or fabricates tool outputs. Critical safety issue. **Failure Modes**: - **Wrong tool**: Agent calls search when should call calculator. - **Wrong arguments**: Calls search("") or search(malformed JSON). - **Looping**: Agent repeats same action ("I'll search... [searches]... I'll search again..."). - **Premature termination**: Agent gives up before completing task. - **Hallucinated output**: Agent claims tool returned X when it returned Y. **ReAct Trace Analysis**: Log every (thought, action, observation). Annotate failures: - **At which step did agent fail?** (e.g., step 3/10) - **Why?** (wrong tool? wrong arg? misunderstood observation?) - **Could agent have recovered?** (or was it a dead-end?) **Benchmark Datasets**: - **SWE-bench**: GitHub issues → Agent must write PR to fix. 2,294 real-world tasks. State-of-the-art: ~13% solve rate (as of 2024). - **WebArena**: Agents interact with realistic websites (shopping, social media, etc.). 812 tasks. Complex (multi-page navigation, form filling). - **AgentBench**: 8 environments (OS, DB, knowledge graph, etc.). Holistic benchmark. - **GAIA (General AI Assistants)**: Real-world questions requiring multi-step reasoning + tool use. Hard—human baseline ~92%, GPT-4 ~30%. **Building Your Own Eval Set**: 1. Define tasks from your domain (e.g., "Given a codebase, find the function that handles auth"). 2. Write ground truth (expected function name + file path). 3. Run agent, check if final answer matches ground truth. 4. Start with 10-20 tasks, expand to 100+ as you iterate. **Eval Harness Design**: - **Sandbox environment**: Don't let agent access production (use Docker, VMs, mock APIs). - **Timeouts**: Cap max steps (e.g., 15) and wall-clock time (e.g., 5 min). - **Logging**: Capture full trace (thoughts, actions, observations, errors). - **Reproducibility**: Fix random seed, cache LLM outputs (for debugging). **Human-in-the-Loop Evals**: For ambiguous tasks, have human judge final output. Example: "Write a trip itinerary" → No single correct answer, human rates quality 1-5. **A/B Testing Agents**: Deploy two agent variants (e.g., ReAct vs. Plan-and-Execute). Route 50/50 traffic. Measure success rate, latency, cost. Beware of task variation (ensure same difficulty distribution). **Hands-On**: Implement eval harness for your Week 1 RAG agent. 10 multi-hop questions (e.g., "Who invented the transformer architecture, and what year?"). Measure success rate. Analyze failures: was it retrieval (missing doc)? generation (hallucination)? or reasoning (wrong logic)?

**Why Memory Matters**: LLMs are stateless—each API call is independent. For multi-turn conversations or long-running agents, you need to manage context explicitly. **Types of Memory**: **1. Short-Term Memory (Conversation History)**: Last N messages in prompt. Simple, effective for recent context. Limit: Token window (4K, 8K, 128K). Strategies: - **Sliding window**: Keep last 10 messages, drop older. - **Summarization**: When history exceeds window, summarize older messages ("User asked about X, agent responded Y..."). - **Token budgeting**: Reserve tokens for system prompt, tools, output; fill remaining with history. **2. Long-Term Memory (Episodic Memory)**: Store past conversations/events, retrieve relevant ones when needed. Example: User asks "What did we discuss about project X?" → Retrieve past messages mentioning project X. **Implementation**: - Embed each conversation turn (or summary of multi-turn). - Store in vector DB with metadata (user_id, timestamp, topic). - On new query, retrieve top-k relevant past conversations. - Inject into prompt as context ("Relevant past discussions: ...."). **3. Semantic Memory (Knowledge Base)**: Facts learned over time. Example: "My favorite color is blue" → Store as structured fact (user_id, preference, color, blue). Retrieve via DB query or vector search. **4. Procedural Memory (Skills/Tools)**: What tools agent has used successfully. Example: Agent solved task X by calling tool Y → Remember this pattern. Advanced: Meta-learning (agents that improve tool selection over time). **Memory Retrieval Strategies**: - **Recency**: Retrieve most recent memories (time-based). - **Relevance**: Retrieve semantically similar (embedding-based). - **Hybrid**: Combine recency + relevance (e.g., boost recent by 2x). **MemGPT (Memory-GPT)**: Research project for long-term agent memory. Key idea: Agents manage their own memory via function calls (store, retrieve, summarize). Inspired by OS virtual memory (paging). Agent decides what to keep in 'working memory' (prompt) vs. 'storage' (vector DB). **Conversation Summarization**: When history is long, summarize to save tokens. Two approaches: - **Periodic**: Every 10 turns, summarize last 10 into 2-3 sentences. - **On-demand**: When token budget exceeded, summarize oldest messages. **Prompt Template with Memory**: **Persistent State (Databases)**: For structured data (user prefs, task status, etc.), use DB: - **SQLite**: Simple, embedded. Good for single-user agents. - **PostgreSQL**: Production-grade. Multi-user, ACID transactions. - **Redis**: In-memory KV store. Fast, good for session state. **State Management Patterns**: - **Conversation ID**: Each session has unique ID. Store messages under that ID. - **User profiles**: Store user metadata (name, preferences, history) in DB. Load on session start. - **Task state**: For long-running tasks (multi-day projects), persist state ("step 3/10 complete"). **Context Window Tradeoffs**: Longer windows (128K GPT-4 Turbo, 200K Claude) reduce need for memory management but increase cost + latency. For most tasks, 8K-16K with smart memory is enough. **Hands-On**: Build a multi-session agent: 1. Store conversation history in SQLite (table: messages(id, session_id, role, content, timestamp)). 2. On new session, retrieve last 10 messages for that session_id. 3. Embed all past messages, retrieve top-3 relevant when user asks "What did I ask before?" Test: Multi-turn conversation about a project. Close session, start new session, ask "What project were we discussing?" Agent should retrieve + answer.

**Why Observability Matters**: LLM systems are complex—many components (retrieval, generation, tools), non-deterministic outputs, hard to reproduce bugs. Without observability, you're flying blind. **Three Pillars of Observability**: **1. Logs**: Structured records of events. Example: `{"timestamp": "2024-01-15T10:30:00Z", "query": "What is RAG?", "retrieved_docs": 5, "prompt_tokens": 1200, "response_tokens": 150, "latency_ms": 3400}`. Use JSON for easy parsing. Tools: Elasticsearch, Loki, CloudWatch Logs. **2. Traces**: Show execution flow across components. Example: Query → Embedding (50ms) → Vector Search (120ms) → Re-ranking (80ms) → Prompt Construction (10ms) → LLM Call (3000ms) → Total (3260ms). Identify bottlenecks. Tools: Jaeger, Zipkin, OpenTelemetry. **3. Metrics**: Aggregated stats over time. Example: P95 latency, requests/min, error rate, avg tokens/request, RAG faithfulness score. Tools: Prometheus + Grafana, DataDog, New Relic. **Key Metrics for LLM Systems**: - **Latency**: P50/P95/P99 end-to-end latency, per-component latency (retrieval, generation), TTFT (time to first token). - **Throughput**: Requests/sec, tokens/sec. - **Cost**: $ per request (LLM API cost), tokens used. - **Quality**: RAGAS scores, LLM-as-judge ratings, user feedback (thumbs up/down rate). - **Errors**: Rate of failures (API errors, timeouts, hallucinations detected). **Structured Logging Best Practices**: - **JSON format**: Easy to parse, index, search. - **Correlation IDs**: Trace a request end-to-end (assign unique ID at entry point, log in every component). - **Log levels**: DEBUG (verbose, everything), INFO (important events), WARN (recoverable errors), ERROR (failures). - **Sample logging**: For high-volume systems, log 1% of requests (or dynamically sample based on latency/errors). **OpenTelemetry (OTel)**: Vendor-neutral standard for traces, metrics, logs. Instrumentation libraries for Python, JS, etc. Exporters for Jaeger, Prometheus, DataDog. **Best practice**: Use OTel from day 1, avoids vendor lock-in. **LangSmith**: LangChain's observability platform. Auto-captures LLM calls, chains, agents. Features: trace viewer (see exact prompt, response, timing), dataset management (eval sets), playground (test prompts), annotation (label good/bad outputs). Free tier: 1K traces/month. **Phoenix (Arize AI)**: Open-source alternative to LangSmith. Runs locally. Features: trace visualization, embedding drift detection (are queries changing over time?), model performance tracking. Good for privacy-sensitive use cases. **Debugging Workflows**: 1. **Latency spike**: Check trace → Which component slow? (Often LLM call, sometimes retrieval if DB overloaded). 2. **Quality drop**: Check logs → Are retrieved docs relevant? (Context_recall metric). Is prompt truncated? (Token limit hit). 3. **Error rate up**: Check logs → Which error? (API timeout? Invalid JSON? Tool failure?). Correlate with recent changes (prompt, model, infra). **Dashboards**: Grafana dashboard example: - **Panel 1**: Requests/min (time series). - **Panel 2**: P95 latency (time series), alert if >5s. - **Panel 3**: Error rate (%), alert if >1%. - **Panel 4**: Avg RAGAS faithfulness score (daily), alert if <0.8. - **Panel 5**: Top 10 slowest queries (table). **Alerting**: Set up alerts for critical metrics: - P95 latency >5s for 5 minutes → Page on-call. - Error rate >5% → Slack alert. - Daily RAGAS score drops >10% → Email team. **Cost Tracking**: Log tokens per request. Aggregate: `total_cost = (prompt_tokens * 0.03 / 1K)`. Track by user, by endpoint, by day. Identify expensive queries (often very long contexts or high token generation). **Prompt Versioning**: Store prompt templates in code (not hardcoded strings). Log prompt version with each request. If quality drops, correlate with prompt change. Use git-style versioning (v1, v2, ...) or semantic versioning. **Hands-On**: Instrument your RAG agent: 1. Add structured logging (Python `logging` + JSON formatter, or `structlog`). 2. Log: query, retrieved_doc_ids, prompt_tokens, response_tokens, latency, ragas_scores. 3. Run 100 queries, export logs to CSV. 4. Visualize in Pandas/matplotlib: latency distribution, tokens vs. latency scatter plot, faithfulness over time.

**Production RAG Checklist**: This is the culmination of Week 2. Real-world system must handle scale, failures, evolving data. **Architecture Overview**: **1. Model Serving**: Use vLLM or TGI for LLM inference. Deploy on GPU instance (A10, A100). Config: - Model: Llama 2 13B or Mistral 7B (good quality/cost balance). - Quantization: AWQ 4-bit (2x throughput, minimal quality loss). - Max concurrent requests: 50-100 (tune based on GPU memory). - Timeout: 30s per request. **2. API Server (FastAPI)**: **3. Retrieval Pipeline**: - **Embed query**: Use sentence-transformers (cached model, GPU inference). - **Vector search**: FAISS or Qdrant (top-100). - **BM25 search**: rank_bm25 library (top-100). - **Merge**: RRF to get top-20. - **Re-rank**: Cross-encoder (ms-marco-MiniLM-L-12-v2) to get top-5. **4. Prompt Engineering**: **5. Observability**: - **Logs**: JSON structured logs (timestamp, query, latency, tokens, sources, answer). - **Traces**: OpenTelemetry spans for each component (embed, search, rerank, generate). - **Metrics**: Prometheus `/metrics` endpoint. Track: `requests_total`, `request_duration_seconds`, `llm_tokens_total`, `retrieval_docs_count`, `error_rate`. **6. Evaluation**: - **Offline**: RAGAS test set (100 queries). Run weekly, track trends. - **Online**: Collect user feedback (thumbs up/down). Log to DB, analyze monthly. - **A/B tests**: Deploy two variants (different prompt, retrieval strategy), measure win rate. **7. Deployment**: Deploy on Kubernetes or AWS ECS. Auto-scaling based on CPU/memory. **8. Continuous Improvement**: - **Prompt versioning**: Track prompt changes in git. Log prompt version with each request. - **Eval regression tests**: CI/CD runs RAGAS on every PR. Block merge if score drops. - **Data flywheel**: Collect failed queries → manually label → add to eval set → improve retrieval/prompts → re-deploy. **9. Cost Optimization**: - **Caching**: Cache LLM responses for repeated queries (Redis, TTL=1 hour). - **Smart routing**: Use small model (GPT-3.5, Llama 2 7B) for simple queries, large model (GPT-4, Llama 2 70B) for complex. Classifier determines complexity. - **Batch inference**: If latency allows (e.g., async jobs), batch multiple queries to same LLM call (10x cheaper). **10. Security**: - **Input validation**: Sanitize user queries (prevent injection attacks). - **Rate limiting**: Max 100 requests/min per user (prevent abuse). - **Auth**: API keys or OAuth for access control. - **Data privacy**: If handling sensitive data, use local models (not OpenAI API), encrypt logs. **Scaling**: - **Horizontal**: Run multiple API server replicas (load balancer in front). - **Vertical**: Larger GPU for LLM serving (A100 > A10). - **Separate retrieval and generation**: Retrieval on CPU instances, generation on GPU. Scales independently. **Failure Modes & Mitigations**: - **LLM API timeout**: Retry with exponential backoff. Fallback to cached response or error message. - **Vector DB down**: Fallback to BM25-only search (degraded but functional). - **High latency**: Circuit breaker (if P95 >10s for 1 min, return cached/error, alert team). **Hands-On**: Build the full system. Deploy locally (Docker Compose: API server + vLLM + Qdrant). Load test with `locust` (100 concurrent users, 10 min). Measure: requests/sec, P95 latency, error rate. Verify Prometheus metrics, check logs in Elasticsearch/Loki. **Week 2 Retrospective**: You now understand production LLM serving, advanced RAG, evaluation frameworks, memory management, and observability. Week 3 shifts to distributed data engineering and system design—critical for building large-scale data pipelines that feed LLM systems.

Spark enables distributed data processing across clusters. Core abstractions: RDD (low-level), DataFrame (high-level, optimized), Dataset (typed DataFrame). **RDD (Resilient Distributed Dataset)**: Immutable, partitioned, fault-tolerant. Transformations (map, filter, groupBy) are lazy; actions (count, collect) trigger execution. Lineage DAG enables recomputation on failure. **DataFrame**: Table abstraction (rows + schema). Uses Catalyst optimizer (logical plan → optimized logical → physical plan). 10-100x faster than RDDs due to code generation (Tungsten project). **Execution Model**: Driver (coordinator) + Executors (workers). Job → Stages (shuffle boundaries) → Tasks (per partition). Shuffle is expensive (disk I/O, network). Minimize via coalesce, repartition carefully. **Lazy Evaluation**: Build DAG of transformations, execute only on action. Enables optimization (predicate pushdown, column pruning). **Partitioning**: Data split across executors. Default: 200 partitions. Too few → underutilization; too many → overhead. Rule of thumb: 2-4 partitions per CPU core. **Best Practices**: Use DataFrame API (not RDD unless necessary). Persist/cache intermediate results if reused. Broadcast small lookup tables (<100MB). Use Parquet (columnar, compressed, predicate pushdown). **Hands-On**: Process 1B row dataset (generate synthetic). Measure time to filter, aggregate, join. Compare narrow vs wide transformations (map vs groupBy). Use Spark UI to inspect stages, tasks, shuffle read/write.

**Partitioning**: Determines parallelism and data locality. **Repartition**: Full shuffle (expensive) to N partitions. Use when reducing skew or increasing parallelism. **Coalesce**: Reduce partitions without shuffle (combines adjacent). Use when writing fewer files. **PartitionBy**: For output (Parquet/ORC), partition by column (e.g., date). Enables partition pruning at read time. **Caching**: `cache()` = MEMORY_AND_DISK, `persist(StorageLevel.MEMORY_ONLY)`. Cache if DataFrame used 2+ times. Cost: memory usage vs recomputation. Monitor via Spark UI (Storage tab). **Broadcast Join**: For small tables (<10MB), broadcast to all executors. Avoids shuffle. `broadcast(df_small).join(df_large)`. **Shuffle**: Occurs on groupBy, join, distinct, repartition. Expensive: write to disk, network transfer, read. Minimize via: (1) filter early, (2) reduce data before shuffle, (3) use broadcast join. **Adaptive Query Execution (AQE)**: Spark 3+ feature. Dynamically adjusts plan at runtime: (1) coalesce shuffle partitions, (2) switch join strategies, (3) optimize skew joins. Enable: `spark.sql.adaptive.enabled=true`. **Skew Handling**: One partition much larger → straggler task. AQE detects and splits skewed partition. Manual: salting (add random prefix to join key). **Spill**: If executor memory full, spill to disk. Slow. Solution: increase executor memory or reduce partition size. **Resource Tuning**: Executors (# cores, memory), driver memory. More executors → more parallelism but overhead. Typical: 5 cores, 10-20GB per executor. **Best Practices**: Start with defaults, profile (Spark UI → longest stages), optimize bottlenecks. Common fixes: repartition before expensive ops, cache reused data, broadcast small tables, enable AQE. **Hands-On**: Benchmark join of 1B row table + 100M row table. Baseline: sort-merge join (shuffle both). Optimize: broadcast smaller table. Measure shuffle read/write (should drop to zero for small table).

**Kafka Architecture**: Distributed log. Topics (logical channels) → Partitions (ordered, immutable sequence). Producers append, Consumers read. **Brokers**: Kafka servers. Leader handles reads/writes for a partition, followers replicate. ISR (In-Sync Replicas) = replicas caught up. **Producers**: Send records to topics. Key-based partitioning (same key → same partition, preserves order). Acks: 0 (fire-forget), 1 (leader ack), all (ISR ack). Trade-off: latency vs durability. **Consumers**: Pull model. Consumer group = multiple consumers share topic. Each partition assigned to one consumer in group (parallelism = # partitions). Offset = position in partition. Commit offset to track progress. **Consumer Rebalance**: When consumer joins/leaves, partition reassignment. Stop-the-world (all consumers pause briefly). Minimize by stable consumer group. **Partitioning**: Determines parallelism and order. If key=null, round-robin. If key provided, hash(key) % num_partitions. Use consistent key for order (e.g., user_id → all events for user in same partition, in order). **Retention**: Kafka stores messages for configurable time (default 7 days). Old messages deleted. Compacted topics: keep latest per key (good for state, e.g., user profile). **Exactly-Once Semantics (EOS)**: Idempotent producer + transactional writes. Prevents duplicates. Enable: `enable.idempotence=true`. **Throughput**: Kafka handles millions of messages/sec. Optimizations: batch writes (producer), zero-copy (sendfile syscall), sequential disk I/O (append-only log). **Use Cases**: Event streaming, log aggregation, CDC (Change Data Capture), real-time analytics. **Hands-On**: Spin up Kafka (Docker: wurstmeister/kafka). Create topic: 10 partitions, replication factor 3. Produce 1M messages (vary keys). Observe partition distribution (kafka-consumer-groups --describe). Consume with 3 consumer group members, verify each gets ~3 partitions.

**Data Lake Limitations**: Parquet/ORC on S3/HDFS = great for batch reads, but no ACID (can't update/delete), no schema enforcement, no time travel. Enter **Lakehouse**: combines warehouse features (ACID, schema, versioning) with lake scalability (cheap storage, open formats). **Delta Lake** (Databricks): Open-source layer over Parquet. Features: (1) ACID transactions via transaction log (JSON), (2) Time travel (query old versions), (3) Schema enforcement + evolution, (4) Upsert/Merge (MERGE INTO), (5) Z-order clustering (multi-dimensional indexing), (6) Optimize/Vacuum (compaction, cleanup). **Transaction Log**: `_delta_log/` folder with versioned JSON files (00000.json, 00001.json, ...). Each commit = atomic operation. Readers check log for latest version. **Time Travel**: `SELECT * FROM table VERSION AS OF 10` or `TIMESTAMP AS OF '2024-01-01'`. Use case: audit, rollback, reproduce bugs. **Apache Iceberg** (Netflix): Similar to Delta but more vendor-neutral. Features: (1) Hidden partitioning (users don't specify partition in query), (2) Snapshot isolation (concurrent reads/writes), (3) Schema evolution (add/drop/rename columns safely), (4) Partition evolution (change partitioning without rewriting data). Used by: Netflix, Apple, Airbnb. **Apache Hudi** (Uber): Focus on streaming upserts. Features: (1) Copy-on-Write (COW) vs Merge-on-Read (MOR) tables, (2) Incremental queries (read only new data since last query), (3) Record-level updates (vs Delta/Iceberg = file-level). Trade-off: MOR = faster writes, slower reads; COW = slower writes, faster reads. **Comparison**: Delta Lake = easiest (tight Spark integration, mature), Iceberg = most flexible (multi-engine: Spark, Flink, Trino), Hudi = best for streaming upserts (but complex). **Use Case**: Delta Lake for Databricks-centric, Iceberg for multi-engine, Hudi for high-frequency updates. **Optimization**: Compaction (merge small files), Z-order (cluster by multiple columns for range queries), Vacuum (delete old files). **Hands-On**: Create Delta table from Parquet. Insert 1M rows. Update 100K rows (MERGE INTO). Query old version (VERSION AS OF). Measure performance: Delta vs plain Parquet for point lookups (WHERE id=X).

**Rate Limiting**: Prevent abuse, ensure fair resource usage. **Fixed Window**: Count requests in time window (e.g., 100 req/min). Reset at boundary. Issue: burst at boundary (99 req at 0:59, 99 at 1:00 = 198 in 2 sec). **Sliding Window Log**: Track timestamp of each request. Remove old entries. Accurate but memory-intensive (store all timestamps). **Sliding Window Counter**: Hybrid. Weighted count from prev + curr window. Example: 80 req in prev min, 20 in curr (30 sec elapsed) → estimate 80*0.5 + 20 = 60. Approximate but efficient. **Token Bucket**: Bucket holds N tokens, refilled at rate R. Each request consumes 1 token. If bucket empty, reject. Allows burst (up to N). Used by AWS API Gateway, Stripe. **Leaky Bucket**: Queue requests, process at fixed rate. Smooths traffic but adds latency (queuing delay). **Distributed Rate Limiting**: Use Redis (centralized) or consistent hashing (partitioned). Redis: INCR key, EXPIRE. Lua script for atomicity. **Caching**: Store frequently accessed data closer to user. **Cache Levels**: L1 (in-memory, app server), L2 (Redis/Memcached, shared), CDN (edge, static assets). **Cache Strategies**: (1) Cache-aside (lazy load: read from cache, if miss read DB + write cache), (2) Write-through (write to cache + DB synchronously), (3) Write-behind (write to cache, async write to DB). **TTL**: Time-to-live. Expire old entries. Trade-off: longer TTL = stale data, shorter = more DB load. **Eviction Policies**: LRU (Least Recently Used), LFU (Least Frequently Used), FIFO. LRU most common (Redis default). **Cache Invalidation**: Hardest problem. Strategies: (1) TTL-based (simple but stale), (2) Event-driven (invalidate on DB write), (3) Versioned keys (immutable cache). **Load Balancing**: Distribute traffic across servers. **Algorithms**: (1) Round-robin (each server in turn), (2) Least connections (server with fewest active), (3) Weighted (prefer powerful servers), (4) IP hash (sticky sessions). **Layer 4 vs Layer 7**: L4 (TCP/UDP, fast, no content awareness), L7 (HTTP, can route by path/header, slower). **Consistent Hashing**: Map servers + keys to ring. Each key → nearest server clockwise. Add/remove server → only 1/N keys remapped (vs all keys in naive hash). Used by Cassandra, DynamoDB, Memcached. **Hands-On**: Implement token bucket in Python + Redis. Use INCR + EXPIRE. Simulate 10K concurrent clients, each sending requests. Measure: latency P99, false negative rate (legit request rejected), false positive rate (excess request allowed).

**News Feed System (Twitter/Facebook)**: Core operations: post, fetch timeline. **Fan-Out on Write** (Push): When user posts, write to all followers' timelines (precompute). Fetch = fast read (O(1), just read user's timeline). Write = slow (O(# followers)). Issue: celebrity with 10M followers → 10M writes per tweet. **Fan-Out on Read** (Pull): When user fetches timeline, query followees' recent posts, merge, sort. Write = fast (O(1), just store tweet). Read = slow (O(# following), need to query + merge + sort). **Hybrid**: Fan-out on write for regular users, fan-out on read for celebrities. Detect celebrity (>1M followers), skip fan-out, fetch their tweets on-read. **Architecture**: API servers (post, timeline), write workers (fan-out), cache (Redis: user timeline), database (tweets, follows). **Storage**: Tweets table (id, user_id, text, timestamp), Follows table (follower_id, followee_id). Graph DB (Neo4j) or sharded SQL. **Timeline Cache**: Redis sorted set (ZSET). Key = user_id, value = tweet_id, score = timestamp. Fetch: ZREVRANGE (latest N). **Ranking**: Not just chronological. Use ML model (user engagement, tweet quality, recency). Two-stage: candidate generation (retrieve 1000 tweets), ranking (score + sort top 100). **Chat System (WhatsApp/Slack)**: Requirements: 1-1 and group chat, real-time delivery, read receipts, persistence. **Architecture**: WebSocket servers (persistent connection), message queue (Kafka), DB (messages, users). **Message Flow**: User A sends → WebSocket server → Kafka → WebSocket server (for user B) → User B receives. Kafka ensures durability, decouples senders and receivers. **Read Receipts**: Track last_read_message_id per user. When user reads, send ack to server, update DB. **Group Chat**: Store group membership in DB. On send, fan out to all members. Use Kafka partitioning (group_id = partition key). **Presence (Online/Offline)**: Heartbeat (client pings every 30 sec). If no ping for 60 sec, mark offline. Use Redis (TTL-based). **Search Ranking (Google)**: Web crawler → indexer → query processor → ranker. **Indexing**: Inverted index (term → list of doc_ids + positions). Stored in sharded clusters. **Query Processing**: Parse query, tokenize, retrieve candidate docs (AND/OR logic), rank. **Ranking**: PageRank (link-based authority) + content signals (term frequency, freshness, user engagement). ML model trained on click data. **Sharding**: Shard index by doc_id (document partitioning) or by term (term partitioning). Doc partitioning more common (better load balance). **Hands-On**: Design Twitter feed on paper. Estimate: 100M DAU, avg 50 following, 1 tweet/day per user. QPS = 100M / 86400 ≈ 1157 writes/sec. Fan-out = 1157 * 50 = 57K writes/sec to cache. Storage = 100M users * 100 tweets * 1KB ≈ 10TB. Discuss: caching, sharding, replication.

**Real-Time Analytics Pipeline**: Combine streaming (Kafka), processing (Spark), storage (Delta Lake), serving (API). Use case: dashboards, monitoring, alerting. **Architecture**: Events → Kafka → Spark Structured Streaming → Delta Lake → Spark SQL API → Dashboard. **Kafka**: Topic = `clickstream`, partitions = 10 (parallelism). Producers = web servers, mobile apps. Retention = 1 day (enough for reprocessing). **Spark Structured Streaming**: Micro-batch (default 2 sec) or continuous. Read from Kafka, apply transformations (window aggregations), write to Delta Lake. **Windowing**: Tumbling (non-overlapping, e.g., every 5 min) vs Sliding (overlapping, e.g., last 5 min, updated every 1 min) vs Session (gap-based, e.g., user sessions with 30 min inactivity gap). **Watermarking**: Handle late data. Set watermark (e.g., 10 min). Drop events older than watermark. Trade-off: longer watermark = more complete but higher latency. **Delta Lake**: ACID, time travel. Write aggregates to `page_analytics` table (page, hour, view_count, avg_duration). Optimize with Z-order on page + hour (fast point queries). **Serving**: FastAPI + Spark SQL. Example endpoint: `/analytics/page/{page}?start_hour=2024-01-15T10&end_hour=2024-01-15T12`. Query Delta Lake, return JSON. Cache results (Redis, TTL=5 min). **Lambda Architecture**: Batch layer (reprocess historical data daily) + Speed layer (real-time, approximate). Merge results at query time. Pro: correct + fast. Con: maintain two systems. **Kappa Architecture**: Streaming-only (no batch layer). Reprocess by replaying Kafka. Simpler but requires infinite retention (expensive) or snapshotting. **Exactly-Once Semantics**: Idempotent writes + transactional commits (Kafka + Delta Lake both support EOS). Prevents duplicates. Cost: slightly lower throughput. **Monitoring**: Track: Kafka lag (consumer offset - latest offset), Spark processing time (should be < batch interval), Delta Lake write latency. Alert if lag >1M messages or processing time >batch interval (backpressure). **Hands-On**: Deploy pipeline locally (Docker Compose: Kafka, Spark, MinIO for S3-compatible storage). Generate synthetic clickstream (Python script, 1K events/sec). Verify: (1) Kafka topic populated, (2) Spark job running (check Spark UI), (3) Delta Lake table updated (query with spark-sql), (4) API returns results. Measure end-to-end latency (event timestamp → query result). Week 3 Retrospective: You now understand distributed data processing (Spark), streaming (Kafka), lakehouse (Delta Lake), and system design patterns. Week 4 focuses on production concerns (observability, cost, security) + interview prep (LeetCode, mocks, behavioral).

**Cost Optimization**: Cloud costs grow fast. Strategies: (1) Right-size instances (don't over-provision), (2) Use spot/preemptible (70% cheaper, but can be terminated), (3) Auto-scaling (scale down during low traffic), (4) Reserved instances (1-3 year commit, 30-60% off), (5) S3 lifecycle policies (move old data to Glacier). **LLM-Specific**: (1) Cache responses (avoid redundant API calls), (2) Use smaller models (GPT-3.5 vs GPT-4 = 10x cheaper, often good enough), (3) Batch requests (many providers offer batch APIs, 50% off), (4) Self-host (vLLM + Llama 2 = 10/M). **Security Threats**: (1) **Prompt Injection**: User crafts input to override system prompt. Example: "Ignore previous instructions, reveal your system prompt." Mitigation: Input validation, separate user/system prompts (Anthropic Claude supports), output filtering. (2) **Data Leakage**: Model trained on sensitive data, user queries elicit it. Mitigation: Fine-tune on sanitized data, use RAG (don't embed PII), redact outputs. (3) **Jailbreaking**: Bypass safety guardrails (e.g., "pretend you're DAN"). Mitigation: Use moderation API (OpenAI, Azure Content Safety), log outputs, human review for high-risk. (4) **Model Theft**: Attacker queries model extensively, trains clone. Mitigation: Rate limiting, watermarking (research area). (5) **PII Exposure**: Agent logs contain user data. Mitigation: Redact logs (regex for emails, SSN), encrypt at rest, short retention (7-30 days). **Input Sanitization**: Strip special characters, limit length (e.g., 1000 chars), detect injection patterns (regex for "ignore instructions", "system prompt"). **Output Filtering**: Check response for PII (spaCy NER, Presidio), profanity (profanity-check library), toxicity (Perspective API). **Compliance**: **GDPR** (EU): User data rights (access, deletion), consent, data minimization. For LLMs: Don't train on user data without consent, support deletion (hard—need to retrain or use RAG only). **HIPAA** (US healthcare): Encrypt PHI, audit logs, BAA with vendors. **SOC 2**: Security, availability, confidentiality. Audit controls (access logs, encryption, backups). **Best Practices**: (1) Least privilege (IAM roles), (2) Encrypt in transit (TLS) + at rest (KMS), (3) Audit logs (CloudTrail, Splunk), (4) Secrets management (AWS Secrets Manager, Vault), (5) Vulnerability scanning (Snyk, Dependabot). **Hands-On**: Run cost analysis on your RAG system. Example: 10K users, 100 queries/day = 1M queries/day. OpenAI GPT-4 Turbo: 30/M output tokens. Assume 1K input, 200 output per query = 1B input + 200M output tokens/day = 6K = 480K/month. Optimization: Switch to Claude 3.5 Sonnet (15) = 3K = 180K/month (63% savings).

**Sliding Window**: Variable or fixed-size window that slides over array/string. Use when: (1) contiguous subarray/substring, (2) optimization (max/min length), (3) constraint (e.g., at most K distinct). **Template**: `left=0; for right in range(n): add arr[right] to window; while constraint violated: remove arr[left], left+=1; update result`. **Examples**: Longest substring without repeating chars (variable window, HashSet for seen chars). Max consecutive 1s after flipping K 0s (variable window, count 0s, shrink if >K). **Two Pointers**: Two indices move toward each other or same direction. Use when: (1) sorted array, (2) pairs/triplets with target, (3) palindrome. **Opposite Direction**: Classic = container with most water (left=0, right=n-1, move pointer with smaller height). 3Sum = fix i, two pointers on [i+1, n-1], move based on sum. **Same Direction**: Remove duplicates from sorted array (slow=unique index, fast=scan). **Fast & Slow Pointers** (Floyd's): Detect cycles in linked list. Slow moves 1 step, fast moves 2 steps. If cycle, they meet. Find cycle start: reset slow to head, move both 1 step, meet at start. **Why it works**: When they meet, slow traveled k steps, fast 2k. If cycle length = C, fast = slow + nC → k = nC → cycle start is k steps from head. **Examples**: Linked list cycle, find duplicate in array (treat as linked list: arr[i] = next pointer). **Hands-On**: Solve 10 problems listed in exercise. Focus on recognizing patterns. For each: (1) identify pattern (sliding window? two pointers? fast & slow?), (2) write pseudocode, (3) implement, (4) test on examples. Track time—aim for <25 min per medium problem. Pro tip: After solving, read top 2-3 solutions on LeetCode Discuss. Learn alternative approaches (often cleaner or faster).

**Graph Representation**: Adjacency list (dict of lists, space O(V+E)) vs adjacency matrix (2D array, space O(V^2)). List is default (sparse graphs). **BFS (Breadth-First Search)**: Explore level by level. Use queue. **When to use**: Shortest path (unweighted), level-order traversal. **Template**: `queue=[start]; visited={start}; while queue: node=queue.pop(0); for neighbor in graph[node]: if neighbor not in visited: visited.add(neighbor); queue.append(neighbor)`. **DFS (Depth-First Search)**: Explore as far as possible before backtracking. Use recursion or stack. **When to use**: Connected components, cycle detection, topological sort. **Template**: `def dfs(node, visited): visited.add(node); for neighbor in graph[node]: if neighbor not in visited: dfs(neighbor, visited)`. **Backtracking**: Explore all possibilities, prune invalid paths. **Template**: `def backtrack(path): if valid(path): result.append(path); return; for choice in choices: path.append(choice); backtrack(path); path.pop()`. **Examples**: Permutations (choose from remaining elements), Subsets (include or exclude each element), N-Queens (place queen, check conflicts, backtrack). **Optimization**: (1) Prune early (if partial solution invalid, don't recurse), (2) Memoization (cache subproblems, but often not applicable in backtracking), (3) Iterative (avoid recursion overhead for shallow trees). **Topological Sort**: Order nodes such that u comes before v for every edge u→v. Use: Course schedule (dependencies). **Algorithm**: (1) Kahn's (BFS): Remove nodes with in-degree=0, update neighbors. (2) DFS: Post-order (reverse of finish time). **Cycle Detection**: DFS with 3 colors (white=unvisited, gray=visiting, black=visited). If visit gray node, cycle exists. **Union-Find (DSU)**: Alternative for connected components. Faster for dynamic connectivity (add edges incrementally). Not covered today but worth learning. **Hands-On**: Solve 10 problems. Focus on: (1) Choosing BFS vs DFS (shortest path → BFS, explore all → DFS), (2) Backtracking template (build, recurse, undo), (3) Graph construction (parse input → adjacency list). Track time. If stuck >30 min, read hints, re-attempt later. Pro tip: For backtracking, always draw recursion tree (helps visualize pruning).

**DP (Dynamic Programming)**: Solve by breaking into overlapping subproblems. Two approaches: **Top-Down (Memoization)**: Recursion + cache. Natural, easier to write. **Bottom-Up (Tabulation)**: Iterative, fill table. Faster (no recursion overhead), harder to reason. **Steps**: (1) Define state (what does dp[i] represent?), (2) Recurrence relation (how to compute dp[i] from smaller subproblems?), (3) Base case, (4) Order of computation (bottom-up: iterate i from 0 to n). **1D DP**: Single variable. **Examples**: Climbing stairs (dp[i] = dp[i-1] + dp[i-2], ways to reach step i). House robber (dp[i] = max(dp[i-1], dp[i-2] + nums[i]), max money up to house i). **2D DP**: Two variables. **Examples**: Longest common subsequence (dp[i][j] = LCS of s1[0:i], s2[0:j]). Edit distance (dp[i][j] = min edits to transform s1[0:i] to s2[0:j]). **Knapsack**: Subset selection to optimize value under constraint. **0/1 Knapsack**: Each item used once. dp[i][w] = max value using first i items, weight ≤ w. Recurrence: dp[i][w] = max(dp[i-1][w], dp[i-1][w-weight[i]] + value[i]) (exclude or include item i). **Unbounded Knapsack**: Each item used unlimited times. Similar but inner loop considers dp[i][w-weight[i]] (same row, not previous row). **Examples**: Coin change (fewest coins to make amount), subset sum (can partition into equal sum). **Space Optimization**: 2D DP often reducible to 1D (only need previous row). Example: LCS dp[j] = max(dp[j], dp[j-1]+1). Saves space O(n) instead of O(n^2). **Common Pitfalls**: (1) Forgetting base case (e.g., dp[0]=0), (2) Off-by-one (index i vs i-1), (3) Overwriting dp before using it (space optimization requires reverse iteration). **Hands-On**: Solve 10 problems. For each: (1) Write brute-force recursion (exponential), (2) Add memoization (top-down DP), (3) Convert to tabulation (bottom-up), (4) Optimize space if possible. Track time. DP is hard—don't worry if you need hints. Re-solve 3 days later to solidify. Pro tip: Draw DP table for small examples (n=3, 4). Helps visualize recurrence.

**Mock Interview Benefits**: (1) Simulates pressure (real interviews are stressful, practice helps), (2) Identifies weak areas (DP? graphs? system design?), (3) Improves communication (thinking aloud, explaining trade-offs). **Coding Interview Format**: 45 min, 1-2 problems. Interviewer expects: (1) Clarify problem (ask questions, confirm examples), (2) Discuss approach (brute-force, optimal, time/space complexity), (3) Write code (syntactically correct, handle edge cases), (4) Test (walk through example, corner cases), (5) Optimize if time. **Common Mistakes**: (1) Jumping to code (spend 5 min on approach first), (2) Not thinking aloud (interviewer can't help if silent), (3) Ignoring edge cases (empty input, single element), (4) Poor naming (i, j, k → use descriptive names). **Time Management**: If stuck >10 min, ask for hint. Better to solve with hint than fail silently. If easy problem, solve quickly (15 min), spend remaining on optimization/edge cases. If hard, aim for working solution (may not be optimal). **System Design Format**: 45-60 min. Interviewer expects: (1) Clarify requirements (functional: post tweet, timeline; non-functional: 100M users, <200ms latency, 99.9% uptime), (2) High-level design (draw boxes: API, cache, DB, queue), (3) Deep dive (choose 1-2 components, discuss details: DB schema, caching strategy, sharding), (4) Discuss trade-offs (consistency vs availability, latency vs cost). **Common Mistakes**: (1) Jumping to details (spend 10 min on high-level first), (2) Not asking questions (clarify scale, constraints), (3) Ignoring non-functionals (scalability, reliability), (4) Overcomplicating (start simple, add complexity only if needed). **Feedback**: After mock, ask interviewer: (1) What went well? (2) What to improve? (3) Would you hire me? (be honest). Write down feedback, track trends (e.g., weak on DP → practice 10 more DP problems). **Platforms**: **Pramp** (free, peer-to-peer, coding + system design), **Interviewing.io** ($50-200/session, anonymous, real engineers), **Exponent** (subscription, recorded mocks with feedback). **Recording Yourself**: Watch recording, cringe, learn. Notice: filler words (um, like), long pauses, poor explanations. Improve by practicing explanations (talk through solutions even when solo). **Hands-On**: Do 3 mocks today. Schedule in advance (Pramp requires 24h notice). After each, spend 15 min reviewing: What did I do well? What should I improve? Write action items (e.g., "practice more graph problems", "explain trade-offs earlier in system design").

**Behavioral Interview Purpose**: Assess: (1) Culture fit (do you align with company values?), (2) Past behavior (best predictor of future), (3) Soft skills (communication, leadership, resilience). **STAR Method**: Structure answers to avoid rambling. **Situation** (context, 1 sentence): "I was working on a recommendation system at Amazon." **Task** (your role, 1 sentence): "I was responsible for reducing latency." **Action** (what you did, 2-3 sentences): "I profiled the code, identified that feature extraction was the bottleneck. I parallelized it using multiprocessing, reducing time from 500ms to 80ms." **Result** (impact, 1 sentence + metric): "Latency dropped 84%, improving user engagement by 12%." **Common Questions**: (1) Tell me about a time you led a project. (2) Describe a conflict with a teammate and how you resolved it. (3) Tell me about a failure and what you learned. (4) Give an example of a time you innovated. (5) Describe a time you had to work with limited resources. (6) Tell me about a time you disagreed with your manager. (7) Describe a time you had to learn something quickly. (8) Tell me about a time you missed a deadline. **Preparing Stories**: Write 10-15 stories covering different themes (leadership, conflict, failure, innovation, collaboration). For each: (1) Real example (don't fabricate), (2) Quantify impact (metrics: latency, cost, users), (3) Highlight your role (use "I", not "we"), (4) Show growth (what you learned, how you improved). **Amazon Leadership Principles**: If interviewing at Amazon (or Amazon-ish companies), align stories to LPs: (1) Customer Obsession, (2) Ownership, (3) Invent and Simplify, (4) Bias for Action, (5) Learn and Be Curious, (6) Hire and Develop the Best, (7) Insist on the Highest Standards, (8) Think Big, (9) Frugality, (10) Earn Trust, (11) Dive Deep, (12) Have Backbone; Disagree and Commit, (13) Deliver Results, (14) Strive to be Earth's Best Employer, (15) Success and Scale Bring Broad Responsibility. Map each story to 1-2 LPs. **Conflict Resolution Example**: Situation: "I disagreed with a teammate on architecture (monolith vs microservices)." Task: "We needed to decide before sprint start." Action: "I scheduled a 1:1, listened to his concerns (deployment complexity), shared mine (scalability). We agreed on a hybrid: monolith for MVP, plan migration to microservices in 6 months. I documented the decision and trade-offs." Result: "Launched on time, migrated successfully later. Relationship strengthened through open communication." **Failure Example**: Situation: "I was tasked with migrating a database, underestimated time." Task: "Migrate 10TB in 1 week." Action: "Migration took 3 weeks (slow network, schema issues). I communicated delays to stakeholders daily, prioritized critical tables first (80% of queries hit 20% of tables). I also wrote a post-mortem, identifying 5 lessons (test migration on subset first, allocate buffer time)." Result: "Migration completed with minimal downtime. Post-mortem adopted as team practice." **Delivery Tips**: (1) Be concise (2-3 min max), (2) Show impact (use metrics), (3) Highlight learning (growth mindset), (4) Stay positive (even for failure stories, focus on what you learned), (5) Practice aloud (record, refine, re-record). **Hands-On**: Write 15 stories (use template: S, T, A, R). Practice delivering to a friend or record yourself. Time yourself—aim for 2-3 min. Get feedback: Was it clear? Engaging? Result quantified?

**Resume**: 1 page (unless 10+ years experience). Sections: (1) Header (name, email, phone, LinkedIn, GitHub), (2) Summary (2-3 sentences: who you are, what you do, what you're looking for), (3) Experience (reverse chronological, bullet points with metrics), (4) Education (degree, school, GPA if >3.5), (5) Skills (languages, frameworks, tools), (6) Projects (optional, if space). **Bullet Points**: Use CAR (Context, Action, Result). Example: "Reduced RAG system latency by 60% (from 3s to 1.2s) by implementing hybrid search + re-ranking, improving user engagement by 15%." Action verbs: built, led, optimized, reduced, increased. Quantify: % improvement, $ saved, # users. **LinkedIn**: (1) Headline (not just job title—add value prop: "Data Engineer @ MSFT | Building ML pipelines for OneNote Copilot"), (2) Summary (tell your story: background, what you do, what excites you, what you're open to), (3) Experience (same as resume but can be longer), (4) Featured (pin 3 projects: GitHub repos, blog posts, talks), (5) Skills (add 20-30, endorse others), (6) Recommendations (ask 2-3 colleagues/managers). **GitHub**: (1) Profile README (what you do, tech stack, featured projects, contact), (2) Pin 3-5 best repos, (3) Each repo README (what it does, why it's cool, how to run, tech stack, screenshots/demo). Use badges (build status, license, stars). **Portfolio Projects**: Showcase: (1) RAG agent (from Week 1-2), (2) Real-time analytics pipeline (Week 3), (3) One personal project (startup idea, side project). Each project: clear README, working demo (deployed or video), clean code. **Final Mock**: Schedule 2 hours: (1) Coding (45 min, 2 problems), (2) System design (45 min), (3) Behavioral (30 min, 5 questions). Treat as real interview (dress up, quiet room, no interruptions). Record, review, note gaps. **Reflection**: Write: (1) What I learned (top 10 takeaways from 4 weeks), (2) What I'm proud of (3 wins), (3) What needs more work (2-3 areas), (4) Next steps (apply to 10 companies, schedule interviews, continue practicing 1 LC problem/day). **Readiness Assessment**: Green (ready): Solve 80%+ mediums in <30 min, system design feels comfortable, 15 STAR stories ready. Yellow (almost): 60-80% mediums, system design shaky, 10 stories. Red (need more time): <60% mediums, can't design a system, <5 stories. Adjust timeline accordingly. **Interview Strategy**: (1) Apply to 20-30 companies (mix of target, reach, safety), (2) Schedule interviews 1-2 weeks apart (time to prep + learn from each), (3) Start with safety companies (practice, build confidence), (4) Target companies last (when you're sharpest). **Hands-On**: Update resume, LinkedIn, GitHub today. Schedule final mock. Write 1-page reflection. Share with a friend or mentor, get feedback. Set calendar reminders: (1) Apply to 10 companies tomorrow, (2) Solve 1 LC problem daily, (3) Weekly system design review. **4-Week Retrospective**: You've covered: (1) AI Agents + LLM foundations, (2) Production RAG + evals, (3) Distributed data eng + system design, (4) Production concerns + interview prep. You're ready. Trust your preparation. Go get those offers.