News

Developer Hit with $6,500 AWS Bill from Visual Studio Bug

• By David Ramel
• 09/08/2015

It's all cleared up now, but a developer last week was surprised to be hit with a $6,500 bill from Amazon Web Services Inc. (AWS) after being victimized by bitcoin miners who used an Amazon access key they found in code mistakenly hosted in a public GitHub repository.

A South African developer named Carlo, experienced with Git, explained that his code was posted to a public repository -- instead of a private repository as he intended -- because of a bug in the GitHub extension shipping with Visual Studio 2015. The bug has been patched, and AWS is refunding the big bill, but Carlo shared his experience to highlight security concerns when working with the cloud.

"As developers, we need to be aware of best practices when it comes to pushing source to the cloud," Carlo said. "Things are not exactly set in stone yet, we are making up the rules as we go along. Ultimately this could make for an interesting case study and I hope that it raises awareness around the potential dangers of version control in the cloud, especially when used in conjunction with limitless cloud accounts like AWS."

Carlo explained how he eschewed his usual practice of using the Git command line to instead use the built-in Visual Studio IDE's functionality to commit a local code repository to a new private repository on GitHub. Unfortunately, unbeknownst to him, it went to a public repository. He received a message from AWS that his account had been compromised with the disclosure of an access key, and he tried to contain the damage, but bitcoin miners had already found it and put it to use -- expensive use.

"Bitcoin miners continuously scan GitHub source code for Amazon access keys," Carlo explained. "They then use these keys to spawn large numbers of EC2 instances to mine for bitcoins. They make big coin while those who were exploited are left with huge bills."

Bitcoin is "an innovative payment network and a new kind of money," Bitcoin.org says. "Bitcoin uses peer-to-peer technology to operate with no central authority or banks; managing transactions and the issuing of bitcoins is carried out collectively by the network." Bitcoin users can leverage exposed access keys to mine the Internet for other bitcoins.

"All they need is a single key," Carlo said. "A single click of a button through a buggy UI, or an unintended click can expose your data to the world in an instant. Most companies store source code and data in the cloud in one form or another, and with multiple developers having access to these repositories, we're going to see some significant data leaks in the future. That's a scary prospect for all of us."

Carlo said he was contacted by Microsoft and AWS execs to resolve the matter, with a resulting but fix and AWS bill refund. In the meantime, he had some advice for developers and AWS:

• Always test new version control GUIs before using them in the wild. There could be a bug that could expose your data.
• Encrypt sensitive information in config files.
• Move access keys to a separate config file, and exclude this from Git deploys with a .gitignore.
• Amazon could implement daily max budgets by default.
• Ideally, Amazon shouldn't allow infinite expenditure.
• Amazon should roll out a feature on AWS that allows you to disable AWS access to your entire account, much in the same way that you can set a Google application or Facebook App to live or development mode. This will allow you to sort the issue out without charges being racked up against your account.