Compute is often considered the most expensive resource in cloud computing solutions however there is one resource that can add up quick if you are not careful: Storage Transactions.
Yes at only $0.01 per 10,000 transactions these pesky things look very cheap. Very, very, very cheap in fact. However, in just a few days of testing with my servers deployed I logged in to check my Azure billing only to find this picture:
Shocked? No? Now you might be saying, Mark, your bill is $1.96, that’s not huge. In fact that is really, really cheap. However, let me tell you the ad revenue that my phone application generated in that same time period: $0.10. Yes! Ten Cents. A number so small, I can write it out in English.
How many games did I play? 3 or 4, max. So overall, my game isn’t just not making me money, I am losing over $0.40 every game that is played. Not to mention I get 500,000 transactions free thanks to BizSpark and all those other blue bars are free too. This is a HUGE problem.
Architecting for ad-supported software using the cloud is a very different animal than enterprise applications or sites.
I decided I needed to figure out where all these storage transactions were being incurred.
I use a polling mechanism that hits blob storage in order to update game state. Each Windows Phone client hits my blob storage every 1 second.
For a game that takes 1 hour that translates to:
60s/m X 60m/h = 360 s/h
Then I have game state updates and gets every time the AI does something and every time the player does something. In Euchre, that’s 4 pick it up decisions, 4 call trump actions, 5 play card actions per round.
Based on my Blob Storage Analytics what I am actually seeing is around 1250 GetBlob requests every hour. When I played a game with two phones it jumped to about 2300.
Not exactly sure where the phantom 1250 storage transactions are coming from as they seem to hit every hour no matter what the usage is.
Table Storage is a bit different. I pretty much use Table Storage exclusively for all transactional operations within Game Fabric. The beauty of Game Fabric is Euchre doesn’t have anything specific schema setup in Table Storage. I am purely relying on Game Fabric for my entire infrastructure. Euchre’s game state is stored completely in Blob Storage. So where does that leave table storage?
Meta. Almost entirely meta. User Profiles, User Stats, Game Sessions, Player Registration, etc. Table storage gets hit anytime you browse around Game Fabric pages to browse the active sessions, available users, or your own game stats. It is also used for certain in-game logging operations such as game stats as well as game actions (but that is pretty much legacy at this point).
During the hour I played a game I noticed only about 450 storage transactions here. This is definitely not my problem.
I have two workers: 1 for game fabric stuff and 1 for Euchre game actions. I used to have them both poll every 1 second. Which resulted in about 11,500 transactions per hour.
1 t/s * 60 s/m * 60 m/h = 360 t / h * 2 workers = 720 t / h
How I get 11,500 “user;GetMessage” events in my Queue metrics is beyond me. My workers are pretty simple. DoStuff, Thread.Sleep(1000), Repeat. It makes no sense why I am hitting 11,500 t/h.
I found this article which helped a little bit:
The ‘Back Off Strategy’ looked pretty promising but I have to be careful not to slow down to much otherwise it will impact user experience to greatly.
The ‘X’ Factor
Aside from your users costing you money. You can cost you money too. Ultimately, I discovered that the reason for my massive explosion of Storage Transactions has one name:
Cerabrata Azure Diagnostics
Great app, don’t get me wrong. About to become obsolete, yes, but useful now. However, this thing has plenty of features for monitoring performance counters. Compute was a big worry for me for a while so I decided that I needed to watch CPU usage really closely. As a result, I opened up the performance monitor page and configured it to update every minute and pull 60 minutes of history. Then I left it on for a few hours…well overnight…well, over the long weekend. Hence, my explosion of 500k transactions within 7 days.
Yes it was a red haring but it taught me a valuable lesson. When working with eCPM of $1.00 or less, don’t neglect your storage transactions they will eat you alive!
So after all this work, setting up Storage Analytics and tracking my stats for a couple days. I found that my usage for 2 players to play my game create 14,250 storage transactions in the course of an hour and cost me $0.01425 in storage transactions. That’s $0.007125 per user per hour. Let’s hope 1 hour of impressions can earn me more than that!
If I could figure out where the rogue Queue and Blob transactions are coming from by my estimates I could reduce that number to about 2,000. Which could cost me $0.001 for every user.