What AWS announces and which products it covers
AWS has announced the Global Inference Gateway for Amazon SageMaker HyperPod, a managed endpoint that routes inference requests across HyperPod clusters within the same region or across different regions. The announcement was published on November 20, 2026, on the official AWS Machine Learning blog.
This feature is distinct from the Inference Gateway for HyperPod launched in September 2026, which operates at the cluster level (Tier 1) and manages internal pod routing. The global gateway represents the fleet coordination layer (Tier 2), adding cross-cluster failover, global rate limiting, and cost-aware traffic shaping.
The Global Inference Gateway is positioned as a fleet coordination layer above the internal routing of each cluster. Its function is to provide a single logical entry point for inference while the fleet can include clusters in the same region or in different regions, without the application having to know each individual destination. This placement separates fleet-level routing from cluster-internal pod routing.
Differences between SageMaker routing layers
| Feature | Inference Gateway (Tier 1) | Global Inference Gateway (Tier 2) |
|---|---|---|
| Scope | Intra-cluster | Inter-cluster and inter-region |
| Launch | September 2026 | November 2026 |
| Primary function | Optimal pod selection via GPU signals | Fleet coordination and global failover |
| Authentication | Integrated with IAM | SigV4 and IAM over PrivateLink |
How the single endpoint works
The architecture is based on a single endpoint that acts as the entry point for all inference requests. Instead of the application managing multiple cluster addresses, the client sends the request to this centralized gateway, which internally decides which cluster or region to direct the traffic to.
The system selects the destination cluster based on health and capacity criteria. When a cluster experiences a failure or resource exhaustion, the gateway performs automatic failover to another available cluster, maintaining service continuity without manual user intervention.
By concentrating access in a single endpoint, the application keeps a single reference even if the fleet includes several clusters. The global gateway assumes the destination decision, so the client does not need to maintain address lists or manual selection logic between clusters. This keeps the client-facing interface aligned with the single-entry design.
Destination selection and automatic failover
The routing logic evaluates the availability of clusters registered in the fleet. Unlike standard SageMaker AI cross-region routing, which is triggered by platform faults or container 5xx errors, the HyperPod global gateway is designed to manage full fleet coordination, including global rate limiting and cost optimization.
Automatic failover ensures that requests are not lost during a regional unavailability. This mechanism is critical for high-availability applications that rely on large language models deployed on distributed GPU infrastructure.
Global rate limiting and cost-aware traffic shaping extend the scope of failover: the fleet can apply rate limits at the fleet level and direct traffic toward destinations that meet health and capacity requirements. This behavior is relevant when several clusters offer the same model and the goal is to maintain availability while avoiding concentration on a single destination. The distinction matters when the same model is available in more than one cluster.
Security and access integration
The global gateway integrates authentication via SigV4 and IAM over AWS PrivateLink. This means each request is authenticated with standard AWS credentials, allowing granular access policies to be applied without exposing the endpoint to the public network.
The use of PrivateLink ensures that traffic between the application and the gateway remains within the AWS network, improving security and latency. This configuration is consistent with security best practices for inference services in enterprise environments.
Authentication with SigV4 and IAM over PrivateLink allows access control to be based on AWS identities and policies, rather than relying on endpoint-specific credentials. By keeping traffic within the AWS network, the configuration supports granular permissions for services that consume inference without exposing the endpoint to the public network. This keeps access decisions tied to the AWS identity model.
Operational details and pending metrics
Although the announcement confirms the availability of the feature, AWS has not yet published detailed metrics on the latency added by global routing, specific capacity limits, or additional costs associated with inter-regional coordination.
Engineering teams must perform load testing with their own workloads to evaluate the impact on time-to-first-token (TTFT) and availability. Specific technical documentation on fleet configuration and global routing parameters is still under development.
Until official metrics are published, latency, capacity, and cost values should be treated as provisional. Engineering teams should validate behavior with their own workloads, especially the impact on time-to-first-token and service continuity, and confirm which clusters and regions participate in the fleet before assuming uniform behavior. That validation helps teams understand the practical effect of the additional routing layer.
Steps to evaluate implementation
- 01Identify the HyperPod clusters that will participate in the global fleet.
- 02Configure SigV4 and IAM authentication for the gateway endpoint.
- 03Perform latency tests comparing direct cluster access with access through the global gateway.
- 04Validate failover behavior by simulating failures in a cluster.
Keep exploring
Sources consulted
Corrections and transparency
If you spot incorrect or outdated information, send us a correction with the page and source we should review.
Submit a correction