The other shoe finally dropped on my Google Enterprise cloud storage plan
If you ever wonder why IT people are perpetually cranky, it's this: we have to somehow juggle impossibly conflicting trade-offs, navigate the capricious and inconsistent services of large, unfeeling, and seemingly uncaring corporations, and somehow keep it all within an impossibly low, yet already too high budget.
Also: Ahead of AI, this other technology wave is sweeping in fast
There are days that suck, when one or more of the balls we're trying to keep in the air explode, taking some of the other balls with them. Those are the days when you have to somehow come up with a new plan, fix something now terribly broken, keep your stress level to a point where it's not dangerous, and come up with an answer that's likely to satisfy no one but be the best possible compromise nonetheless.
For me, this time, it was Thursday and Google. And, to make matters worse, I knew this day would come. I didn't want to know, but in my heart-of-hearts/Spidey Sense/tingly feeling on the back of my neck, I knew.
You can catch up on the history of my deep dive into the situation here:
- What happens to your G Suite unlimited storage when Google moves you to Workspace?
- Re-homing G Suite storage: No, you can't find out how much storage your folders use
- Re-homing G Suite storage: Enterprise plan options seem unnecessarily murky
- Why my two-person company bought a Google Workspace Enterprise plan
Backgrounder
One of my most haunting IT problems has long been how to handle offsite backups. I'm a big proponent of the 3-2-1 best practice, which dictates there be at least three copies of each file, on at least two devices, and at least one which is offsite.
When you're wrangling nearly 100TB of local storage, the offsite requirement becomes a challenge.
Also: The best cloud storage services right now
In "Why my two-person company bought a Google Workspace Enterprise plan," I detailed my most recent solution to the problem. After much research, I bought the Google Workspace Enterprise plan, even though my company is comprised of just two employees: my wife, and me. That was in January of 2022, just about 18 months ago.
The deal was, Google promised unlimited storage as long as I bought five seats of their Enterprise plan. So I bought five seats.
It's here that we have to be very careful. Because the reality (as I discussed in that article) is that Google was very noncommittal in exactly what storage was offered. I even reached out to Google PR for an official statement, and what I was told was: "Enterprise customers will be able to request as much storage as they need."
Also: Need more storage? How to quickly set up a Synology DS223j NAS
I knew then that "request" doesn't mean "get." I wrote about it, saying:
"Enterprise customers will be able to request as much storage as they need" did not, in any way, confirm that Enterprise customers would get as much storage as they need.
But I researched it even more. I eventually wound up in a support conversation with one Katherine Steffany of Google Workspace Support. Here, you can read what she said:
She actually used the term "unlimited storage" and instead of saying "request," she said "you can use as much storage as you need."
Even though I then wasn't sure it was an official answer, I said this in that January 2022 article:
Is that an official answer? Well, it's certainly the answer I'm going to quote back to Google if I get any pushback on my 47TB (and growing) storage footprint.
Frankly, I'm not sure a Katherine Steffany even exists. I wanted to reach out to her directly for help, but there's no one with that name on LinkedIn. The only person a Google search turned up was a teen who'd posted a smoochy-faced selfie pic with little heart emojis. She was a kid, and could not possibly be the Google Workspace Support technician I communicated with.
Also: The best network-attached storage devices
So I had to take on faith a reply from a PR person who was unwilling to commit beyond the word "request," and an assurance from a possibly made-up sock puppet name from a Google Workspace Support agent.
And yes, I knew it might all go bad. I said as much, too:
As for the future, I don't think we'll really know the rest of the story until we live it. I sure wish Google would just confirm they're not going to pull any more rug-pulling-out-from-under tricks with the Enterprise plan, but that's not Google.
In the meantime, I hope we've at least bought ourselves a couple of years before having to address the cloud storage problem again.
18 months. I got 18 months, almost to the day.
SitRep (situation report)
The first I knew the bits were about to hit the fan was when this showed up at the top of my email:
It couldn't be my overall storage, because that was unlimited. Or that's what I first thought. Instead, I guessed that one of my accounts had some quota I needed to up. But when I logged into the Enterprise Workspace admin dashboard, I saw this:
Ruh roh.
So I reached out to Google Workspace Support for Enterprise. It did not go well. At first, I was told that I was only allocated 25TB, but I was using in excess of 57TB. Now, way back in January 2022, I was already using 47TB, so how I could have possibly been allocated just 25TB is unclear.
Also: How to set up your own NAS for more reliable data backups
Even so, I requested more storage. The agent couldn't help me at that time, but a few days later he responded that I had been approved for another 25TB. This would bring my allocation up to 50TB. But, of course, I already had 57TB in use.
By this time, I'd been getting two notices per day saying I had been placed in a "60 day grace period."
More interesting, was the line "After the grace period ends on Aug 26, 2023, your account will go into a 'read-only' state." Yep, Google was threatening to put an entire company's enterprise account into a "read-only" state.
So I requested another allocation of 25TB. I was told:
I have contacted the team regarding the increase of the storage, as the team has approved with an additional 25 TB of storage and that is maximum can be given to you. I also see that you have recently requested for more storage on 14-6-2023, and the next request for more storage can be made after 90 days.
Interestingly, the whole problem came to my attention on June 28th, so I certainly didn't request storage on the 14th. My guess is that's the day when Google's automated systems decided to intervene in some way.
But now, we're at the point where Google is telling me I can't ask for more storage until 90 days have passed, but they'll freeze my account in 60 days.
I'm sure nobody there intends to be mean-spirited, but shutting something down in 60 days and requiring 90 days to get a fix feels harsh.
Also: I found the ultimate external SSD - and it can even self-destruct
I asked the support rep why he wasn't honoring the promise made by the mysterious Katherine Steffany, but he completely ignored that direction of inquiry. I asked to have the issue escalated, but he countered by offering to have a phone conversation with me at a time we agreed upon.
I'm always happy to chat, so at the appointed time, I waited for his call. After an hour, I sent an email asking what happened, and said I was pretty disappointed that he didn't show.
Twelve hours later, this appeared in my inbox:
I apologize that we could not make a call on the given time, due to unforeseen circumstance. I would request you to help us with the next available time, so that I will be able to arrange a call back from our superior.
I tried rescheduling. It's now been four days. I've sent two additional email requests to have that phone conversation he wanted, and still no further response. That's where the situation lives right now.
The rest of the story
Roughly 12 hours after I first posted this article, a senior technician from Google Workspace Support reached out to tell me my storage had been increased by another 25TB and the freeze warning was shut off. I'd like to think this was just their process and not the result of my article, but we'll never know.
Nonetheless, needing 10TB+ more storage every year clearly isn't sustainable, so I'm going to audit my backup use and see if there's any way I can reduce my overall storage footprint.
What I'm going to do about it
As you can see, my storage usage is nearly entirely Google Drive. That's no surprise because that's how I'm using this account.
Here's my enterprise's top-level directory and a quick rundown on what's in each folder:
- Cloud Backup: This is an incremental backup of my servers. The incremental backup means that if a file is deleted now and we had it last week, we can go back to previous backups and reconstruct that file.
- Cloud Sync: This is a mirror of my local server. Anything deleted locally is also deleted in Google Drive.
- LocalBackups: These are backups of my individual computers.
- Sharing: Files shared to others.
- Test: Stuff being tested.
Notice the File Size field:
Can you tell how much is stored in each folder? No, neither can I. The three dots will pop open a View Details pane, but that pane also shows no size information. There is no way from Google Drive to tell how much storage is in each folder.
Also: Storage improvements have outperformed Moore's Law by a factor of 800%
And yet, somehow, I'm supposed to trim terabytes of data out of it so my entire corporate infrastructure isn't frozen on August 26.
I did find a number of apps on the Google Workspace Marketplace. I tried a few that failed when they attempted to scan all my data. I did start running an extension called CleanDrive, which looks like it might work. It's been scanning for four days, says it has evaluated 4.7 million files, and is still chugging along. So I'm left with guessing.
My current plan is to delete the entire Cloud Backup directory. Although I have no way of knowing how much is stored there until I delete it, I'm guessing it's probably roughly half of the overall storage footprint. I'm still backing up, in that I have mirrors of the local servers. The only thing I won't have is the ability to roll back a backup and recover older, lost data.
That's certainly not ideal, but faced with the choice of having my entire Google infrastructure frozen on August 26, or giving up on backup rollbacks, I guess I'll give up on backup rollbacks. I still have all my local backups.
My one remaining worry and thoughts about what this all means
My one remaining worry is whether I can even delete that folder. I first have to move it to the trash and then delete the trash. But can Google Drive handle that big a folder, or will I have to somehow delete thousands of subfolders first?
There's a bigger picture issue here, as well. Internet storage works pretty well for most file types, but when you start to move into video, it breaks down. Well more than half of the storage I use is dedicated to the video I've produced, and various supporting media assets.
Also: Optimize Mac Storage can make your files go poof. Ask me how I know
That usage will continue to grow. Those of us who produce a new video each month (and especially those YouTubers who do a video a week) consume vast terabytes of storage. Clearly, the cloud backup and cloud storage providers aren't able to provide storage to meet those needs -- or at least aren't able to do so under their previously offered all-you-can-eat plans.
And that's where we are now. Cloud-based storage solutions work well for non-video data. But they become cost-prohibitive for both backers-up and storage providers once production video is added into the mix.
That said, I value working with Google, but I do wish that my current experience as a Google enterprise customer (which I will still remain, despite this bump in the road) were a bit more smooth. And I do wish vendors who make infrastructure offerings would understand that changing those offerings can cause huge infrastructure earthquakes (infraquakes?) if they don't carefully define their offerings, and then honor those offerings in perpetuity.
What about you? Have you had cloud storage issues? Has a vendor ever subjected you to an infraquake? Let us know in the comments below.
You can follow my day-to-day project updates on social media. Be sure to follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.