How to Run a Big Model on Cheap Hardware?

An AI model relies on numerical parameters, also known as weights, that are structured into layers to convert inputs into outputs. In language models, input text is broken into tokens, and the model iteratively generates responses by predicting probabilities for subsequent tokens. While training is a computationally intensive process that updates weights on expensive infrastructure, inference utilizes fixed weights and can often be executed on standard consumer hardware.

Fitting large language models on smaller hardware is constrained by total memory capacity, memory location, and processing bottlenecks. An 8-billion-parameter model at 16-bit precision requires about 16 GB just for weights, plus extra space for software overhead, temporary calculations, and context. Because persistent storage like SSDs is too slow for execution, weights must reside in fast working memory such as RAM or GPU VRAM. Beyond capacity, memory bandwidth and compute architecture pose limits, with GPUs providing superior parallelism over CPUs and distinct bottlenecks arising across the prefill and decoding stages of inference.

Local execution allows developers to utilize existing hardware for experimentation and decrease reliance on rented cloud infrastructure. This approach offers enhanced privacy for sensitive documents and proprietary code, alongside offline capability for desktop assistants and remote deployments. Additionally, running models locally provides greater control over model versions and application behavior. However, it is not universally cost-effective, as hardware, electricity, maintenance, and latency must be weighed against usage frequency compared to hosted services.

Quantization reduces the numerical precision used to represent model weights, decreasing storage footprints such as shrinking an 8B model from 16 GB at 16-bit precision to 4 GB at 4-bit precision. While parameter counts stay the same, the lower precision introduces approximation errors that may degrade output quality depending on the task. Lower precision does not guarantee proportional execution speedups, as efficient inference depends on hardware support and runtime conversion software. The technique can be applied after training via post-training quantization or integrated during training to account for approximation errors.

Layer-wise offloading changes where model weights reside and where their computations execute by moving weights onto the GPU only as needed during a forward pass. While this technique enables the execution of models too large to fit in GPU memory at once, it introduces significant data transfer overhead between RAM, disk, and the GPU. Repeated data movement across generation steps can substantially increase latency, potentially making the model unsuitable for interactive applications. Efficiency can be improved by batching requests or permanently partitioning distinct layers between the CPU and GPU.

Mixture of Experts (MoE) modifies transformer architecture by replacing standard computational blocks with multiple alternative networks known as experts. A routing network dynamically assigns each token to a subset of these experts, separating total parameter count from active parameter count. This allows models to maintain high capacity while reducing per-token computation compared to dense architectures. However, because inactive experts must remain accessible for subsequent tokens, MoE models still demand significant memory storage despite their lower compute footprint.

Knowledge distillation is a technique where a larger teacher model helps train a smaller student model using its outputs or predictions. The resulting student model functions independently while demanding significantly less memory and computational power for deployment. Distillation produces an entirely distinct model rather than an exact compression, meaning that strong performance on a targeted task does not guarantee broad equivalence across all tasks. The upfront training expenditure allows cost-effective, repeated deployment, though developers can also choose pre-distilled alternatives.

Pruning reduces computational overhead by eliminating weights or structural components whose removal results in an acceptable loss of quality. While zeroing individual weights introduces sparsity, it does not automatically reduce storage or execution costs without specialized compressed representations, hardware support, or software engines designed to skip zeroed operations. Alternatively, removing entire units or layers directly downsizes the model structure. Realizing actual efficiency gains requires coordination between the pruned model and the underlying execution runtime, and subsequent retraining may be necessary if too much capacity is removed.

Even after reducing model weights, transformers can encounter out-of-memory errors on long sequences due to dynamic memory requirements. During generation, the key-value (KV) cache stores intermediate representations from prior tokens to avoid redundant calculations. Unlike fixed model weights, the KV cache grows dynamically as sequence length and the number of active conversations increase. Mitigating this memory footprint involves reducing input context, quantizing the cache, or offloading parts of it to CPU memory at the expense of speed or quality.

The implementation of an inference runtime can significantly boost performance on existing hardware without altering model weights. FlashAttention improves execution by partitioning attention calculations into blocks, utilizing fast internal GPU memory to lower memory traffic while computing exact attention. PagedAttention, introduced in vLLM, manages the KV cache in blocks to eliminate memory fragmentation and duplication during dynamic request serving. Additionally, serving software improves throughput by batching multiple requests together to share the overhead of reading model weights, requiring more working memory in exchange.

Speculative decoding accelerates autoregressive text generation by employing a smaller, faster draft model to propose several candidate tokens. The larger target model evaluates these candidates concurrently, leveraging greater parallelism than standard sequential generation. Accepted tokens are appended to the generation, while rejected tokens are corrected by the target model. Because maintaining a separate draft model increases memory requirements, speculative decoding does not reduce the baseline memory footprint required for the target model.

Deploying models under hardware constraints, such as a system with 32 GB RAM and 8 GB VRAM, requires balancing memory capacity against response latency and output quality. Applying four-bit quantization reduces an 8B model's theoretical weight footprint from 16 GB to 4 GB, while layer and cache offloading can address remaining memory limits at the expense of transfer speed. A thorough evaluation framework measures answer quality, peak memory consumption, time to first token, and overall generation speed against application-specific demands. Because optimizations like quantization and offloading impact different workload components, their resource savings do not simply compound multiplicatively.