Daily Dose of DS 2026-09-01

Static vs. Dynamic vs. Continuous Batching in LLMs, clearly explained!

Transitioning a RAG application from a local environment to a production environment with multiple replicas introduces challenges related to state management. Local setups often store vector indices, conversation history, and documents in memory or on local disks, which leads to data loss or inconsistency when scaled behind a load balancer. To ensure production reliability, developers must externalize these components using persistent vector stores like pgvector, shared state checkpointing via tools like LangGraph, and centralized object storage. Akamai provides reference implementations and workshops to help developers architect these production-ready systems. The article compares static, dynamic, and continuous batching strategies for serving Large Language Models on GPUs, emphasizing that decoding is primarily memory-bound by weight reading. Continuous batching, or iteration-level scheduling, is presented as the most efficient method for handling variable output lengths by allowing requests to enter and exit the batch at every forward pass. To address latency spikes caused by large prompts in continuous batching, the text introduces chunked prefill as a technique to split input processing across multiple iterations. The text addresses the challenge of training classical machine learning models, specifically tree-based ensembles, on large tabular datasets that exceed memory capacity. While frameworks like Spark MLlib offer one solution, the Random Patches method provides an alternative by training individual trees on random subsets of both rows and columns. This approach not only enables training on large-scale data but also tends to improve model performance and reduce variance compared to traditional random forests. By ensuring individual trees are as different as possible, the method effectively achieves the bagging objective of variance reduction.

閱讀原文 ↗

目錄 3 段

01A good technical LLM interview question:
02Static vs. Dynamic vs. Continuous Batching
03Train classical ML models on large datasets

AI ENGINEERING

A good technical LLM interview question:

Local RAG setups fail in production because in-memory variables and local disks are not shared across load-balanced replicas.
Vector indices must be persistent and reachable by all replicas to avoid empty results or redundant embedding processes on boot.
Conversation history requires external checkpointing, such as LangGraph's Postgres integration, to maintain context across different replicas.
Shared object storage is necessary for documents to ensure all replicas access a consistent corpus and avoid per-replica ingestion.
The 'rag-langgraph-k8s-quickstart' repository demonstrates a production architecture using FastAPI, LangChain, and Terraform on LKE.
Akamai's 'akamai-workshop-ai-inference' covers advanced topics like KV cache tradeoffs and continuous batching for self-hosted models.

LLMs

Static vs. Dynamic vs. Continuous Batching

The article compares static, dynamic, and continuous batching strategies for serving Large Language Models on GPUs, emphasizing that decoding is primarily memory-bound by weight reading. Continuous batching, or iteration-level scheduling, is presented as the most efficient method for handling variable output lengths by allowing requests to enter and exit the batch at every forward pass. To address latency spikes caused by large prompts in continuous batching, the text introduces chunked prefill as a technique to split input processing across multiple iterations.

LLM decoding performance is primarily limited by memory bandwidth rather than compute capacity.
Static batching is inefficient for autoregressive models because it forces the entire batch to wait for the slowest sequence to finish.
Dynamic batching uses timers to form batches but still treats the batch as a single execution unit, leading to idle time.
Continuous batching allows requests to enter and exit the batch at each iteration, maximizing GPU utilization for variable output lengths.
The KV cache is the main memory constraint that limits the maximum batch size in continuous batching systems.
Chunked prefill splits large prompts into smaller segments to maintain consistent token generation latency for existing requests.
Static batching remains suitable for fixed-output tasks like classification and embedding generation.

MACHINE LEARNING

Train classical ML models on large datasets

The text addresses the challenge of training classical machine learning models, specifically tree-based ensembles, on large tabular datasets that exceed memory capacity. While frameworks like Spark MLlib offer one solution, the Random Patches method provides an alternative by training individual trees on random subsets of both rows and columns. This approach not only enables training on large-scale data but also tends to improve model performance and reduce variance compared to traditional random forests. By ensuring individual trees are as different as possible, the method effectively achieves the bagging objective of variance reduction.

Scikit-learn has limited support for batch processing in its classical machine learning implementations.
Standard tree-based ensemble methods typically require the full dataset to be loaded into memory.
The Random Patches method involves sampling both random rows and random columns for each tree in an ensemble.
Empirical results indicate that Random Patches often outperforms traditional random forests on various datasets.
Training on distinct data patches reduces the overlap between trees, which effectively lowers the model's variance.
This technique allows for the construction of robust models on datasets that are too large for single-machine memory.