Hyperscale data centers are booming and this market segment is tracking to have a record year within the broader data center industry, with demand extending worldwide. Below, we provide an overview of how hyperscale data centers fit in to the broader data center landscape, why Amazon Web Services, Microsoft Azure, Facebook and ByteDance (owner of TikTok) are propelling the industry’s growth and what the digital infrastructure need is to support this growth.
Hyperscale Data Center Facilities
Hyperscale data center businesses generally have leases with one customer or tenant per data center facility. Customers are typically cloud providers or large enterprises seeking to exit their older enterprise facilities. These customers are looking to outsource their infrastructure needs, and include:
- Cloud Service Providers: Amazon Web Services (AWS), Microsoft Azure, and Google Cloud
- Internet Companies: Facebook, Uber, Twitter, Netflix, Apple, and Salesforce
- Emerging Companies: Dropbox, Spotify, and Snapchat
Investment Merits – Hyperscale Data Centers
Hyperscale data centers experience high market growth during high IT spend cycles. Demand for these facilities is growing because of hyperscale customer (e.g., Amazon Web Services) capital expenditure growth. Hyperscale customers sign long-term leases, have lower churn rates, and offer repeat business to owners of data centers, that they have a strong relationship with (e.g., CyrusOne). Finally, the balance sheet of the hyperscale data center owner is strategic. This is because, long-term leases allow data center owners to secure better costs of debt, and investment grade credit ratings (e.g., Digital Realty).
Investment Risks – Hyperscale Data Centers
Hyperscale customers (e.g., Amazon Web Services) have significant bargaining power with data center owners during renewal processes, lowering renewal leasing spreads. Additionally, this also pushes returns on invested capital (ROIC) lower for new developments, which currently stand at 9% to 11% per annum.
Further, there are only a small number of target customers (from the three categories in the list above) to achieve growth in the hyperscale segment with. Finally, technology obsolescence is a significant risk, particularly when a hyperscale lease comes to the end of its term.
Digital Infrastructure – Supporting Hyperscale Cloud Platforms
The most important trend that is re-shaping the data center industry is the movement of computing resources and workloads to the cloud. Specifically, these workloads are moving to public cloud service providers such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud, Alibaba, and Tencent. Adoption of public cloud services is growing because of the economic rationale for a number of different workloads. Considerations for public cloud adoption include cost, speed, ease of use, ubiquity, and scalability, all of which the public cloud offers.
Cloud service providers are placing the “on-ramps” to their services in existing “network nodes” and architecting their networks and “compute” or “storage” nodes around them. These “on-ramps” are being placed in facilities owned by the largest data center providers including Equinix, CoreSite, and Digital Realty, which have a focus on interconnection. This phenomenon can be explained in greater detail, with a case study using Amazon Web Services (below).
Amazon Web Services – Case Study
Amazon Web Services, is the largest public cloud service provider, and thus is a good case study for how its “on-ramps”, “network nodes” and “compute” or “storage” nodes, are being architected. This can help rationalize why public cloud service providers are using hyperscale data center facilities.
Above is a depiction of Amazon’s different “Regions” across the United States and Western Europe. The dotted lines signify Amazon’s private network of dark fiber or wavelengths, which connect the different “Regions” together.
Amazon Web Services operates in a series of “Regions”. It currently has 24 geographic “Regions” and 77 “Availability Zones” worldwide, including:
- United States: Northern Virginia, Ohio, Northern California, Oregon
- Europe: London, Paris, Frankfurt, Ireland, Stockholm
- Asia: Tokyo, Japan; Osaka, Japan; Seoul, South Korea; Singapore; Hong Kong; Sydney, Australia; Mumbai, India; Beijing, China
- South America: São Paulo, Brazil
Each Amazon Web Services “Region” is connected to Amazon’s private network (as seen above). Amazon’s private network consists of dark fiber or wavelengths leased from the largest fiber providers like Zayo and Lumen Technologies.
In addition to its “Regions” Amazon Web Services has a larger number of “Edge Locations”.
Amazon Web Services – Region Architecture
With the above framework of an Amazon Web Services “Region” outlined, below we visually depict how the different components of this “Region” architecture fit-in with the broader ecosystem. This ecosystem includes “Availability Zones”, “Data Centers” and “Transit Centers” as seen in the diagram below.
Availability Zones
“Availability Zones” are comprised of one or more data centers that house Amazon Web Services equipment AND no single data center is included in more than one “Availability Zone”. Therefore, as seen above, some “Availability Zones” (represented by the light purple dashed boxes) have one data center, others have two data centers, and others have three data centers within the “Availability Zone”.
Additionally, each Amazon Web Services “Region” has at least two “Availability Zones”. In the example shown above, there are four “Availability Zones” displayed.
“Availability Zones” are isolated locations within a “Region” meaning that if one “Availability Zone” fails the other “Availability Zones” will remain unaffected. For example, each “Availability Zone” is in a different flood plain and powered by a different electrical substation.
Customers of Amazon Web Services often place their resources in multiple “Availability Zones” within a “Region”. The purpose of this is to avoid downtime should one “Availability Zone” experience an issue.
Data centers within each “Availability Zone” are all connected to one another by fiber and are typically close enough to ensure latency of less than one millisecond. Data Centers are depicted as the dark blue boxes (in the diagram above) and each data center has a dark blue line connecting it to one another, which are referred to as Intra-Availability Zone Connections.
Each “Availability Zone” is then connected to all the other “Availability Zones” within the “Region” via fiber, with latency of less than two milliseconds (which is depicted as the pink line) and referred to as Inter-Availability Zone Connections.
Transit Centers
Further, each “Availability Zone” in the “Region” is also linked via fiber to what Amazon refers to as “Transit Centers” (which are signified by the triangles with a dark purple border).
“Transit Centers” are network-rich third-party data centers, that are operated by companies like Equinix and CoreSite. They can also be thought of as interconnection-focused data centers. “Transit Centers” are where the physical connections exist so that traffic to and from data centers in the “Availability Zones” can leave Amazon Web Service’s network (which is signified by the lines connecting to #1 through #4).
When this traffic leaves an “Availability Zone” via a “Transit Center”, it can travel to a number of different locations, including:
- Via the Amazon Private Network to other “Regions”
- To Direct Connect customers. Direct Connect is when an enterprise has private connectivity between Amazon Web Services and their data center, office, or colocation environment
- The Internet via a transit or peering relationship
- To a customer sourcing services via a Cloud Exchange. Cloud Exchange is where a third-party data center provider like Equinix provides direct and private connectivity between two participants in the Equinix footprint
Amazon Web Services and other cloud service providers have deployments in “Transit Centers”, which are known as “on-ramps”. The “on-ramps” can be visualized similarly to the ramps used to get on or off a major highway.
Takeaways for Hyperscale Data Centers
Different types of data centers serve different roles within the architecture of a hyperscale customer like Amazon Web Services. Large footprint “hyperscale” data centers within “Availability Zones” (depicted as the dark blue boxes above) are needed to house scaled deployments of servers and other equipment. These “hyperscale” facilities may be situated in less expensive locations – in terms of land, power, and taxes – often referred to as “compute” or “storage” nodes.
Hyperscale companies like Amazon Web Services, Microsoft, and Google Cloud typically lease space on a wholesale basis OR build their own data centers to meet these “compute” or “storage” requirements. Digital Realty, CyrusOne and QTS typically serve as the “compute” or “storage” nodes, which represent the hyperscale facilities.
Smaller deployments also need to be in connectivity-rich locations. This enables different “Regions” to send traffic among one another within a cloud platform, or as a location, to distribute services externally to customers through private connections or via the Internet – often referred to as “network” nodes.
Hyperscale companies like Amazon Web Services also use these interconnection-oriented facilities to serve these “network” requirements. While data center providers like Equinix, CoreSite, and Digital Realty control these “network” nodes.
Hyperscale Customer Investment
Capital Expenditures for Hyperscale Data Centers
Hyperscale customer revenues are continuing to grow strongly, expanding the broader market. This is driving them to ramp-up their investments in cloud infrastructure by growing their capital expenditures significantly. In the below chart (on the left), total capital expenditures and, where available, cloud capital expenditures from 2015 are compared to estimates for the full-year 2020, by hyperscale customer.
Overall, for the 15 hyperscale customers shown, the total capital expenditures from 2015 of $52 billion is expected to grow to $131 billion for the whole of 2020, which represents a staggering compound annual growth rate of 20%. These hyperscale customers include Google, Amazon, Microsoft, Facebook, Apple, Alibaba, Tencent, IBM, Oracle, Baidu, SAP, Cisco, Salesforce, eBay, and Twitter.
Furthermore, as demonstrated by the case study of Amazon Web Services (discussed earlier), the company is clearly a leader in the absolute amount of capital expenditures it is making to grow its business – alongside Google and Microsoft as the top three hyperscale customers.
Outsourcing and Insourcing Drives the Build vs. Lease Decision
Hyperscale capital expenditures are a leading indicator for the multi-tenant data center industry. Largely, this is the case, because 50% of all data center space, power, cooling, and interconnection is outsourced to third-party data centers (i.e., leased) rather than built and maintained by the hyperscale customers themselves, which is known as insourcing. The percentage of outsourcing is expected to begin increasing to the 60% range by 2022. The primary driver of this is trend is the vast scale that the multi-tenant data center operators (like Equinix and Digital Realty) have gained.
Outsourcing will push even more business to multi-tenant data centers (like Equinix and Digital Realty). This is particularly true for wholesale providers that have specialized in large / hyperscale data center developments across top-tier markets. For example, across Tier 1 European Markets (London, Paris, Frankfurt, and Amsterdam) hyperscale customers are outsourcing almost 100% of their capacity needs. Outsourcing in Europe has reached these levels, because these hyperscale companies have headquarters in either the U.S. or China.
As discussed above, Hyperscale companies like Amazon Web Services, Microsoft, and Google Cloud typically lease space on a wholesale basis OR build their own data centers to meet these “compute” or “storage” requirements.
Build vs. Lease Decision for Hyperscale Data Centers
Referring to the right side of the chart above. Certain hyperscale customers have a greater propensity to lease facilities versus building facilities. The dark blue arrow signifies the recent directional trend for hyperscale customers, in terms of lease versus build decision.
Increasingly, companies are looking to outsource, meaning lease, their infrastructure, because of:
- Speed to market requirements
- Scarcity of land and power
- Complexity in construction
- Capital allocation decisions (turning capital expenditures into operating expenditures)
Build vs. Lease Decision – Examples
Overall, Amazon is at 35% build, followed by Google at 30% build and Microsoft at 20% build. Each are pursuing modest self-builds of their data centers, instead of leasing all of their capacity needs from third-party facilities. Whereas, Alibaba, Tencent, Oracle, IBM, SAP, and Salesforce, perform almost no self-build. Thus, these companies have the greatest estimated leasing mix among the hyperscale customers. Directionally, Google is trending towards a greater propensity to lease. Whereas, Amazon and Microsoft are trending towards a greater propensity to build facilities.
Hyperscale Data Centers – Customer Deployments
The map below depicts the locations of the hyperscale customers Amazon Web Services (AWS), Google Cloud, Microsoft Azure and Facebook. Specifically it shows their dedicated facilities throughout the world.
The map demonstrates the key, “Tier 1” markets, have strong coverage by hyperscale data center deployments, including:
- United States markets: Northern Virginia, Silicon Valley, Dallas, Chicago, Phoenix, New York, and Atlanta
- European markets: Frankfurt, London, Amsterdam, Paris (FLAP) and Dublin
Below is further detail on the largest hyperscale data center market:
Northern Virginia is the largest hyperscale data center market in the United States in terms of supply at 1,200 MW. Furthermore, Northern Virginia also leads in terms of capacity under construction at 700+ MW and annual leasing volume. For the first nine months of 2020, Northern Virginia has leased 330 MW. This represents a sharp increase from 125 MW and 295 MW of leasing volume in 2019 and 2018, respectively. Given it is the largest market, Northern Virginia can provide insight into the dynamics involved in hyperscale leasing overall.
The largest hyperscale customers leasing in Northern Virginia, does vary year-by-year. Therefore, it is worth reviewing a few historical years (dating back to 2016). With this information, we can determine who comprised the majority of leasing volume during that particular year:
- 2016: Microsoft and Oracle drove record leasing volumes in Northern Virginia, comprising 70% of total leasing
- 2017: Facebook and Apple were the two largest customers, consisting of 40% of total leasing
- 2018: another year of record leasing volume, with Microsoft and Facebook collectively leasing 65% of capacity
- 2019: leasing was much lighter in Northern Virginia, than in prior years. Additionally, the customer base was much more diversified across a smaller group of transactions. The two largest customers were Alibaba and Uber, comprising only 20% of leasing
- 2020: Microsoft, ByteDance (TikTok), and Facebook account for 75% of leasing during the first nine months of the year
