Built for Reliability: How American Express Processes Payments at Scale

American Express operates as the core payment network connecting acquiring banks to card issuers during transactions. In 2018, the company began modernizing its platform onto cloud-native infrastructure, shifting away from hardware engineered for continuous uptime to an environment where unexpected server failures are common. Traditional patterns like event-driven processing and monolithic designs were rejected due to strict low-latency requirements and scaling constraints. Instead, the engineering team adopted a design rooted in cell-based architecture to isolate failures and maintain real-time performance.

In a cell-based architecture, a cell is a complete and self-contained copy of an application stack designed to function as an independent failure domain. Each cell owns its required components—such as microservices, databases, and infrastructure—and avoids synchronous cross-cell dependencies along the critical execution path. Unlike microservices, which partition systems by functional domain, cells partition architectures by failure boundaries. Engineers at American Express structure these boundaries around specific real-time transaction journeys to isolate failures and maintain platform availability.

American Express manages data locality within its cellular architecture by categorizing data as immutable, semi-static, or dynamic based on change frequency. Immutable and semi-static data are proactively pushed to all cells prior to transactions to prevent cold cache penalties and eliminate external synchronous calls. For dynamic data that updates per transaction, the architecture inverts the paradigm by moving transactions to the data using deterministic routing. Handled by the Global Transaction Router, routing decisions are derived from transaction attributes for consistency, while asynchronous replication runs in the background for failover.

The Global Transaction Router routes traffic and enforces boundaries across cells in a distributed payment system, acting as the sole communication path between cells and external institutions. To prevent this critical component from becoming a failure point, American Express engineers excluded payment business logic, limiting the router to minimal message parsing for routing decisions. Resilience is further maintained by running instances nearly stateless with non-persistent storage, using asynchronous logging and configuration updates, and deploying parallel active instances across regions. This design demonstrates how essential structural chokepoints remain reliable through deliberate simplicity and minimal dependencies.

The credit card authorization flow at American Express demonstrates a decoupled architecture managed by the Global Transaction Router. Upon receiving a transaction initiated from a merchant point-of-sale terminal, the router forwards the request to an internal processing cell using deterministic or priority-based routing. Within the cell, microservices handle data enrichment, transformation, validation, and issuer identification, but they never communicate directly outside their boundary. Instead, the transaction is returned to the router, which interfaces with card issuers and routes responses back to the context-holding cell before returning confirmation to the acquiring bank.

Payments processing at American Express utilizes an orchestrated microservices architecture to manage workflows and monitor service health. When a mid-transaction failure occurs, the orchestrator halts execution and returns the request to the Global Transaction Router to find a healthy cell. Crucially, the system discards any completed partial work and restarts the transaction from scratch in the new cell. This design choice prevents cross-cell shared state and synchronization issues, accepting the minor cost of redoing a sub-second transaction in exchange for loose coupling.

Transaction recovery semantics define a point of no return up to which transactions can be safely restarted or rerouted before reaching external systems like card issuers. American Express optimizes this by placing irreversible steps late in the payment flow to maximize the recoverable window, while relying on idempotency keys to suppress duplicate processing when late rerouting is impossible. When failed cells recover, platforms mitigate stale state writes through fast transaction completion, downstream idempotency duplicate suppression, and paced, percentage-based traffic shifting.

American Express accepts key trade-offs in its cell-based architecture to preserve cell independence and protect data integrity. These trade-offs include duplicating services across cells to eliminate cross-cell network hops, along with dropping non-critical application logs via buffer truncation during high platform pressure. Observability experiences latency because telemetry is written locally first and aggregated asynchronously, and transactions with strong consistency requirements may be rejected if cross-cell data is not validated. Ultimately, partitioning the platform into independent units trades higher management complexity for a significantly reduced blast radius during failures.

The American Express engineering team designed a resilient payment platform capable of surviving cell failures by ensuring transactions never depend on multiple cells simultaneously. This architecture separates data strategies by pushing static reference data to all cells while routing transactions directly to rapidly changing data. System dependability is anchored by a thin Global Transaction Router that avoids complex logic at critical chokepoints. Instead of resuming partial work across shared state, the platform discards partial transactions and restarts them, pushing the point of no return as late as possible to maximize failure survivability.