News

Google's Cloud Makes Performance Gains Against AWS

• By David Ramel
• 01/06/2020

The price/performance gap between Amazon Web Services (AWS) and the other top cloud computing vendors, particularly Google Cloud Platform (GCP), is shrinking.

That's according to recent benchmark tests by Cockroach Labs. The company, known for its open source CockroachDB that's described as a "SQL database for global cloud services," has been testing cloud platforms since internal testing showed better database performance on AWS and a more formalized study found it to be some 40 percent better than Google Cloud Platform (GCP).

"Not only is GCP more expensive than AWS, but it also achieves worse performance," the company concluded in late 2018. "We recommend using AWS for your most important workloads."

This year, for the 2020 Cloud Report (Cockroach Labs skipped a year), Microsoft's Azure cloud platform was added to the benchmarking.

"We completed over 1,000 benchmark tests (including CPU, Network Throughput, Network Latency, Storage Read Performance, Storage Write Performance, and TPC-C), and found that the playing field looks a lot more equitable than it did last year," the company said in a Dec. 11 blog post. "Most notably, we saw that GCP has made noticeable improvements in the TPC-C benchmark such that all three clouds fall within the same relative zone for top-end performance."

Because the company produces an online transaction processing (OLTP) database, it's primarily interested in testing transactional workloads, focusing on the TPC-C OLTP benchmark tool that simulates an e-commerce business to measure performance and efficiency. Most tests -- like one measuring throughput-per-minute type C (tpmC), also known as the number of orders processed per minute -- didn't show too much differentiation.

"Again, all three clouds come close on the cheapest price per tpmC. However, this year we see that the GCP n2-highcpu-16 offers the best performance per dollar in the tested machine types. If price is less of a concern, AWS is the best performer on throughput alone, but when is price not a factor?"

The report's conclusion provided more info. "GCP shows dramatic improvement in the 2020 Cloud Report edging out AWS and Azure on price per performance of TPC-C but slightly underperforming AWS and Azure on max tpmC available on a three node cluster," it said.

The database vendor -- which said it remains committed to providing a cloud-agnostic database -- also noted the importance of continuous testing.

"Setting up a highly performant configuration isn't always intuitive. It's also important to note that over the past couple years of testing, we've seen different cloud providers performance change drastically. Since these results fluctuate as the clouds adopt new hardware, it's important to regularly re-evaluate your configuration (and cloud vendor).

"CockroachDB remains committed to our stance as a cloud-agnostic database. We will continue to use AWS, Azure, GCP, and others for internal stability and performance testing. We also expect that these results will change over time as all three companies continue to invest in a modern infrastructure ecosystem."

Reproducible steps to duplicate the study are provided on a GitHub repository, and source code for the company's Roachprod open source provisioning system is available here for those interested in provisioning nodes exactly the same as the company did.