Have you ever checked your credit card statement and then encountered an unexpected charge much higher than you were expecting? This happened to me in February 2023. I was expecting my credit card statement to be around $400. Instead, I was pretty surprised to see charges totaling over $1200. As there was a significant difference between reality and my expectations, I logged into my bank account and checked my statement. At the top of the list, I saw a charge of $900 dollars that had come from Google Cloud.
Baffled, I logged into my GCP account and tried to find out what the charge was related to. I got that sinking feeling of dread when I realized it was my monthly service fee. A small app that I had built 6 months ago that used the Google Location API, went from nothing to over $2000 in charges in under a month! Obviously, I quickly got in contact with Google (which was harder than you would think!), deleted all my public-facing API keys, and then kick-started the process of trying to get a refund.
For many developers who have bills to pay, getting an unexpected bill out of nowhere for over $1000 might mean you fail to pay the rent, bills don't get paid, or you get charged unexpected fees for failed direct debits. Asides from the unexpected financial impact, how would you feel about paying over $2000 for a service you hadn't used?
In this blog, I'm going to discuss a seldom talked about topic, the real risks of using cloud provider services for hobby development projects. In the blog, I'm going to cover what these risks look like, what you can do to prevent your API keys from being compromised, and finally, what you can do if your account does get hacked. Stick around to the end to find out if I got a refund or if I had to pay up🔥🔥🔥
The risk is real...
The first thing that I want to emphasize in this article is the risks around building any hobby project using services offered by a cloud provider. When you configure these types of services incorrectly, there is a very real risk that you could be stung with significant charges that you had not planned for. I have another personal story about how this exact thing happened to me several years ago.
I used to host my main website, www.jondjones.com within Azure. When I set it up, I accidentally, clicked on the wrong scale-set option. I disabled my payment notification email and when I logged into my account 4 months later, I noticed I had been charged over $600 for hosting rather than the $90 I was expecting. As this misconfiguration was all my own fault I stumped up the costs, however, the underlining point is that it is very easy to incorrectly configure cloud services and when you do, the charges can be significant!
As soon as you register a debit or credit card with your cloud provider, without proper monitoring, things can get ugly quickly! This is why when you are using your own credit card to register cloud services, always be mindful that even though you might not think that the app you are building will be used in anger by anyone, it may do later on. In my situation, I had a site that I hadn't looked at in over 9 months, that had no location usage. Out of the blue, within less than four weeks, I was faced with over 1.5k in service fees!
There are two main reasons why you can fall into this trap. First, I personally think that all cloud providers have pretty confusing dashboards. Even though I have used Azure for years, if I have a hiatus from using it for a few months, I completely forget how to use it. I find the same with GCP, as the dashboards are not intuitive is that it is very easy to make mistakes when setting things up!
A problem that I fall into when creating hobby projects, is that I'm more caught up in the moment of trying to get something working, rather than building something in a robust and secure way. This is especially true if it's a learning project and I'm not expecting people to really use the end product. If you are unlucky enough to find yourself in the same situation as me, what should you do?
First, you should try and identify what services were used and what resources have been spun up. If something was created that you were not aware of, it can help prove fraudulent use as opposed to you just leaving something running and accruing charges. Assuming you have spun up the service, below lists some common security mistakes developers can make that can result in your service being compromised.
Exposing your API keys publicly: This sounds like common sense, however, when you are working on something with a development mindset that you host publicly on a GIT account, it is very easy to commit and push your private access keys publicly by accident. If you are unlucky enough to make a mistake, no one will tell you, so this is a very real risk!
In my case, I had hidden my API key server-side within an environment variable, however, my key still got misused. Your service account keys are sensitive data and you should treat them as such.
Lockdown by IP/Domain: When setting up a service, like Google Location, where applicable always lock the usage of the services down to a domain or IP address. When I created my API key, I didn't do this as I was simply trying to get something to work locally. After I got the code to work, I didn't even think to go back to GCP and apply further restrictions.
If you want more information on how to lock a service down on GCP, you can find more information here
Delete Unused Services: If you are not using a service anymore, and you never intend to use it again, to avoid the risk of it being hacked in the future, delete, or at least disable your API keys. You can delete projects from GCP here.
Set up a budget alert: To be honest, none of us probably care if people are using the things we build, however, that thinking will usually change if you need to stump up actual money yourself. If you think your services are being misused the first warning sign can be found in your daily billing fees report. Being able to get notifications or updates to either a mobile or email about your charges will allow you to quickly understand if any of your services are being used in unexpected ways. When using GCP you can find those charges here.
Contact support: Initially I definitely struggled to find out how to raise a support ticket with Google. Like most services they purposefully hide these instructions to discourage people from accessing them. Luckily, after I figured out how to get through, the team at the billing center was pretty good. People got back to me via email, pretty quickly. Everyone was friendly and reassuring.
If you find yourself in a similar situation as mine, reach out to the billing team to ask for help. The big caveat is that on the free tiers, you will not get access to technical support, just billing support. If there is genuine evidence of an account being compromised, they can help. Explain to them the financial impact and prove to them how it has been used unintentionally and I am sure they will do something to help.
The refund process
For people who have found themselves in this situation, do not immediately panic. The good news is that I got a complete refund for the fees related to my unused usage. The caveat made clear by Google is that this was a one-time get-out-of-jail-free card only. If you have been hacked or had a stupid amount of traffic that you dispute, you need to get in contact with Google immediately.
Finding the link to how to talk to someone is hard, I managed to get access to an actual person by using the billing chat. The chat is a bot, however, if you pick the options along the lines of no this does not answer my question, eventually, you should be transferred to an actual human. For reference, I spent about an hour trying to figure out how to through to someone. It is possible, just not easy!
In order to dispute the usage I had to dispute the claim. I needed to provide my credit card bill, I deleted the API immediately, and provide the URL of the website, and the GitHub repo link. The process took about 7/8 email exchanges and throughout the process, I kept reiterating that the site was a hobby project and it was used in a commercial setting. My website URL even had the words optimized-demo in it, which probably helped prove my case. If you are going through a similar process, be professional, emphasize the intention of your site, how you think it was misused, and the steps you took to resolve the issue yourself.
if it is a genuine case of misuse, you will have a good chance Google will help you, however, having to wait 2 weeks in order to figure out if you will need to pay an unexpected 1.5K fee is not fun!
Many developers use cloud providers and spin up demo and hobby projects, however, a majority of these people probably never seriously consider the ramifications of misuse. Being charges thousands of pounds for misuse is possible. If you have demo sites, you need to use production-level security around your keys. Delete anything you no longer need as this just increase the chance of an issue,
In the event of things going wrong, get in contact with google support (which might be hard), however, with preservation there is a chance that you can get refunded the stolen usage amount assuming that it is your first offense. Hope this heps someone who finds themselves in this unfortunate situation. Happy Coding 🤘
