Issue 3  March 9 2021

Another Azure vs AWS Benchmark Report

This past week Gigaom released the latest SQL Transaction Processing, Price-Performance Testing report.

First, I want to acknowledge the effort put into this report. This type of comparison is difficult, and the authors acknowledge the effort necessary to make this test as fair as possible. The authors then disclose the report was funded by Microsoft.

This is my shocked face that Azure was the winner in a report funded by Microsoft. It's like paying someone to say nice things about you. But it wasn't just funding, Microsoft also participated in the research! "Microsoft chose the competitors, the test, and the Microsoft configuration."

OK, set all of that aside for the moment. Let's talk about why such reports have little practical value for the IT Professional.

These reports are a fairly useless disk-measuring contest between two rich kids arguing over who has the fastest sports car.

The latest Gigaom report shows one particular contrived edge case scenario. This scenario results in Azure having a lower price per performance cost than AWS. Anyone with practical experience working with cloud database technology knows the database engine workload is but one piece of your overall bill. You need to look at the cost for your entire application ecosystem, not just the database engine.

Think of it this way. You need to move across the country and decide to buy a new car for the trip. You read a report that lists the vehicle with the best gas mileage for city driving, and you choose your vehicle for that specific reason. This means you aren't considering the following factors:

• How much of your stuff can fit in the vehicle
• The gas mileage for highway driving
• The gas mileage for the vehicle when it is packed full of your stuff
• The gas mileage for the vehicle loaded with your stuff going over mountains
• The costs associated with gas itself, or food, and shelter, or car maintenance along the way

There are additional costs associated with running a database workload in the cloud. And there are many configuration options, too! In fact, a slightly different but similar AWS server, such as the r5ad.16xlarge, costs about 10% less than the r5d.16xlarge used in the report could have been chosen, but wasn't.

Perhaps using that server would have resulted in not only a lower cost overall, but maybe a few extra transactions would get processed, making the end result a bit more level.

I understand that Azure wants to show they have a competitive offering to AWS, and this is necessary to win or retain business. But these reports are not going to resonate with me until they are independently done and show a variety of configurations and not one specific edge case scenario.

If there is someone out there that has read this type of comparison report, from any vendor, and then made a decision to go all-in on one cloud or the other, I'd love to hear about it.