The New American AI Model Designed to be Customized

This section explains four foundational concepts essential to understanding language model architecture: tokens, parameters, training, and layers. Tokens serve as the granular text chunks processed by models, while parameters are the internal numerical values iteratively refined during training. Training optimizes these parameters via next-token prediction, gradients, and backpropagation across vast quantities of text. Finally, inputs pass sequentially through stacked layers—such as the 66 layers in the Inkling model—where each layer executes an attention step followed by a feed-forward step.

Inkling uses a Mixture of Experts (MoE) architecture to decouple the cost of storing a model from the cost of running it. Instead of passing tokens through every feed-forward network, Inkling selects 6 out of 256 experts per token, following an approach published by DeepSeek. Across the full 975-billion-parameter model, only about 41 billion parameters (roughly 4 percent) are active during the processing of a single token. A full-precision checkpoint requires at least 2 TB of GPU memory across specialized hardware like NVIDIA B300 or H200 GPUs, while a quantized checkpoint reduces memory requirements to roughly 600 GB across four B300 cards.

In Mixture of Experts architectures, routers score candidate experts to select a top subset for processing each token, but positive feedback loops during training can cause routing collapse where only a handful of experts are utilized. Traditional auxiliary balance loss penalties degrade output quality by generating gradients that conflict with the primary prediction task. Thinking Machines resolves this by adopting an approach originally introduced by Wang and colleagues and utilized by DeepSeek, which adjusts expert selection scores using non-backpropagated bias values updated by token counts. This keeps expert load balanced without compromising gradient updates, while two additional shared experts run unconditionally alongside the six routed experts.

Standard attention computation scales quadratically with sequence length, making a one-million-token context window computationally prohibitive across 66 layers. To address this, Inkling alternates between sliding-window attention layers and full-attention layers in a 5:1 ratio, using 55 sliding-window and 11 full-attention layers. Long-range information flows through the full-attention layers, while sliding-window layers handle local context at lower computational cost. Inkling also utilizes eight key-value heads to reduce memory overhead during generation.

Attention mechanisms lack inherent sequence order information, necessitating positional representation methods like Rotary Position Embedding (RoPE). However, RoPE can struggle to extrapolate to context lengths far exceeding those seen during training because novel rotation angles fall outside the model's learned range. To address this for long sequences, Thinking Machines chose an older relative positional encoding scheme in the style of Shaw et al. for the Inkling model. This approach adds learned distance-based bias values directly to attention scores and clamps distances past a specific cutoff, avoiding the need to extrapolate unfamiliar values.

Inkling incorporates a localized convolution operation with a sliding window of four tokens to handle local sequence mixing. Because standard attention mechanisms have no built-in bias toward adjacent tokens, convolutions supply this local context structurally using learned weights. Thinking Machines inserts these convolutions onto the keys and values within attention layers as well as onto attention and feed-forward outputs before they rejoin the main residual path. This enables attention to reserve its learning capacity for complex dependencies rather than basic local proximity.

Inkling processes images and audio directly without relying on separately pretrained encoder networks. Audio is represented as mel spectrograms and quantized via the dMel method, which requires no pretraining. Images are divided into 40x40 pixel patches and processed independently through a four-stage hMLP stem with minimal compute overhead. Both modalities pass through a lightweight conversion layer and merge into a single sequence alongside text tokens, with all multimodal components trained from scratch alongside the base model.

Thinking Machines introduced an adjustable effort setting between 0 and 1 to control how long the Inkling model reasons before answering. Rather than enforcing a hard token budget, this behavior was trained into Inkling via reinforcement learning by varying token-cost penalties based on the effort prompt. Mechanically, the effort level is inserted as a system message before the conversation starts, independent of the maximum token limit. On the Terminal Bench 2.1 coding benchmark, Inkling achieves the same performance as NVIDIA's Nemotron 3 Ultra using approximately one-third of the tokens.

The architectural choices behind Inkling involve explicit trade-offs between computational efficiency, hardware requirements, and model behavior. Sparse routing and mostly-local attention lower per-token compute costs and enable large context windows, but they enforce high memory footprints and limited layer context. Furthermore, adopting relative position encoding broke compatibility with the prevailing RoPE-based serving infrastructure. While Thinking Machines acknowledges that other models surpass Inkling in baseline strength, Inkling is optimized for direct adaptation and fine-tuning via day-one quantized checkpoints.

Inkling's architecture separates total parameter storage from per-token compute, storing 975 billion parameters while running approximately 41 billion active parameters. The model balances expert routing by applying a selection bias without including it in output weighting, avoiding conflicting training objectives. Its long-context design utilizes 55 sliding-window layers alongside 11 full-attention layers paired with relative position representations. Furthermore, reasoning effort is formulated as a trained setting, enabling benchmark evaluations to function across an adjustable performance curve.