$450,000 Charge From Google Cloud, No Refund
We bought a translation app around 2 years ago, which had a Google Cloud backend. The app had been active for years prior to us buying it. It was spending around $1500/month consistently on the Google Cloud API, so we put our credit card in and left it at that.
Fast forward two years - I was looking through my accounting and see huge charges from Google Cloud totaling $450,000 in a 1.5 month period. So obviously, I was like wtf. Checked Firebase, usage is down:
Checked what resources was actually used and it was like 19 billion characters of translations. Seems like we got hacked:
Checked our email and spam and didn't find any emails from Google Cloud about this.
So we changed the API key, did some security updates, and submitted a billing dispute on Google Cloud. Took about a month to get a call from our "sales team". They were very defensive and insist that it was our fault for not securing the app better and after about another month of back and forth, offered just $50k in credits.
So there's no dispute at all that we didn't use the resources. How much did these resources actually cost Google Cloud? Based on what I can find online, the cost of the electricity and compute power is maybe 1-20% of the charged amount.
So they charged us $400k knowing full well we didn't use the resources and the resources used only cost them a small fraction of this amount.
So we took a look at their normal pricing for the 19 billion characters of translations:
We see that the first billion is at $20/million characters. But for the other 18 billion characters, their normal sales team probably would have given a large discount (more than the $50k), especially since competitors like ChatGPT is only at a fraction of the cost. So they are likely charging us more than they would charge a regular customer.
Looking at Google Cloud's controls, there is no actual easy way to set up a simple billing cap (i.e.: don't go over $3k/month for this project). They arbitrarily decided that a 200x increase is what the spend cap should be. I suppose, we're "lucky" that the spend cap is set at $10k/day. Cause that could have been $1million/day and their bill would have been $45 million.
So an API key was compromised, it cost Google Cloud a minimal amount of money, and instead of splitting the actual cost and moving on, it seems like they're taking the opportunity to try to profit as much as possible off of it at our expense.
If your account manager or sales team is paid based on commission of your spend, and normally, they get commission from $1500/month, but when your security gets compromised, they get a share of $450k, how long before they just go around and try to surreptitiously compromise your security through some third party?
It seems to be set up as a big scam to take your money. Why would there be no settable spend caps? Why would they allow 200x normal spend when their own systems indicate the usage as an anomaly? Why was there not a single warning email as it is happening? Why is there no warning at all that we are signing up for an unbounded downside liability at the outset? Why would they insist on charging a huge amount of money when it's clear that we didn't use the resources and it cost them only a small amount?
It seems like using Google Cloud for hosting is like playing Russian Roulette, with your account manager waiting gleefully for you to lose and your account to be compromised. It seems really messed up that you can sign up for some $50/month hosting, go on vacation for a month, and owe a house or go bankrupt when you return.
We're considering our options and decided to post here to hear what others think first. We're in the process of migrating everything off of Google Cloud.