Gmail, SMTP, DNS and more: Four things I broke first, fixed later
I've been breaking and fixing things.
Hidden gotchas at your hosting provider
There have been many articles over the years about hosting providers and how their claim of unlimited this and unlimited that are actually limited. I'm not going to pile onto that fight.
Instead, let's talk about hosting providers doing what you'd expect: serve stuff over http. This is where I ran into a gotcha earlier this month that's taken quite some time to resolve.
As you know, when a Web site serves a Web page, what really happens is that your browser sends an http request to a Web server, which -- according to a set of rules -- assembles a stream of bits, and then sends that stream of bits back to your browser.
Http requests aren't limited to just browser-to-host, though. Many hosts talk over http to other hosts. This is the fundamental architecture that allows us to have a mashable Web, with XML and REST and JSON requests shooting back and forth between servers, sharing information, and otherwise doing the business of the Web.
Because these requests are so benign, they're served over the traditional Web architecture, and -- in fact -- if you had the URL, you could see the results in your browser. They consist of a packet of text in some format sent from one server to another, and once the other server gets the packet, another packet of text is sent back.
WordPress, for example, uses this mechanism to see whether or not there are any updates ready for plugins. A message is sent from a given WordPress server to the massive server array at WordPress.org, and the resultant update or plugin data is sent back.
I'm building a plugin, and I wanted my main server to update users similarly. I'm using a commercial software licensing add-on that provides a nice API to do all this, and it should work. It's widely used and quite popular.
But it took about three weeks to determine that the XML message being sent from my plugin clients to my server (via POST) was being blocked by the hosting provider. They provide no details on exactly what they're blocking (when you ask, they just provide an ID to a rule, but not the rule details), and it's nearly impossible to diagnose without that information.
I never expected the hosting provider to selectively block POST requests (and it sure wasn't in the terms of service), but there you go. Unpredictable.
Lesson 1: If something weird is going on between Web servers, it might be your hosting provider blocking something you wouldn't normally think they'd block.
They're cheap. I have a bunch of Web sites on their service that don't do anything important or mission critical. But I moved the important, active site to another provider.
Check before moving your domain
That brings me to the next thing I broke. When I moved that site to another provider, I had to change my domain A (address) record to point from hosting provider A to hosting provider B.
As it turns out, sometime in the dim past, I moved my entire DNS record for that domain off of GoDaddy to that hosting provider. That domain does nothing but serve a Web site, so I redirected the DNS record from GoDaddy management to that of hosting provider A. That way, I could manage everything for that site through one cPanel interface.
However, now that I was moving from provider A to provider B, I had to redirect the A record for the Web site. Since hosting provider B didn't manage domains, I jumped into my Premium Domain Manager account at GoDaddy, enabled the domain name servers at GoDaddy to manage my domain, and set the A record to point to the Web site.
It worked like a charm, as I had no doubt it would. After all, it's just a simple A record. That domain doesn't manage anything other than a Web site.
Yeah, well... wrong!
As it turned out, some time ago, I apparently decided that I was going to use that account to route some email (details coming up). It wasn't hosting email (you couldn't mail to david@thatolddomain.com), but it was relaying mail for Gmail.
When I changed the DNS server from hosting provider A to GoDaddy, I broke that mail relay. All of a sudden my email -- this was my primary email address, mind you -- stopped working.
Lesson 2: Just because you think you know how something is configured, before moving or changing it, double check all the details. Otherwise you might break something.
How are you routing email?
I didn't know about all of this, of course, until I tried to figure out why my email had broken. In fact, it wasn't until I tried to fix my broken email that I discovered that email for one domain was routing itself through a mail server for a completely separate domain.
I did this hack because of Gmail's "Send Mail As" capability. I get a lot of email across a number of accounts, including a personal account, a work account, a university account, and so on.
I have my consumer Gmail account set up as the hub for all of this. All my email comes into one Gmail interface, and I have things set up so when I reply, I reply using the email address that came into Gmail.
That way, if someone sends mail to david@work.com, I reply from david@work.com. If someone sends mail to david@university.com, I reply from david@university.com, and so forth. That means I have to be able to both get mail to come into Gmail, and send mail out of Gmail using the appropriate identity.
Gmail does this quite well, in fact. But there's a technical gotcha. For the longest time, if you wanted to send mail out as david@foo.com, Gmail would just send it. You didn't need to own a corresponding SMTP server that would do the sending.
But a few years ago, that changed. Now, Gmail requires that if you send mail from a domain other than gmail.com, you have to route it through an SMTP server, and you have to be able to login and authenticate on that server.
Back then, I wasn't running a mail server on david@mydomain.com. I was just using it. I owned the domain mydomain.com (well, the real domain, anyway), and I wanted to send email. But because Gmail insisted on an SMTP server with valid authentication, I had to dig one up.
Apparently, I did. Remember that Web site I moved? Sometime in the dark ages, I decided to route all my outgoing Gmail email sent as david@mydomain.com through the SMTP server at that other domain. I have only a dim memory of this now, but when my email first started to break, I had no memory of this at all.
It was probably one of those late night fixes that got something to work, and I moved onto the next thing on my to-do list. So, there was that.
Not only did I have no recollection of routing email through that completely unrelated domain, I had no memory of moving that domain's management over to hosting provider A. That, too, probably was the result of a quick late night fix, and was as forgettable as other fixes made shortly before finally getting to bed.
Breakage happened because when I moved the Web site domain from hosting provider A to hosting provider B, I didn't realize there was this random email routing thing running through that account as well, and the result was I broke my email.
Lesson 3: Before you move domains, check your email client (specifically Gmail) to be sure you're not counting on that domain for email routing.
Google Apps can save the day
So this is where I wound up: I had moved from one Web site hosting provider to another Web site hosting provider, but I left my email routing in the dust. I didn't have an SMTP server to route my mail through, and so my mail was in limbo.
Let me point out first how odd it is to say I didn't have an SMTP server. For years (years and years and years) I ran a ton of mail servers. We used to mail to millions of readers a week and I built list management software and mail server software, and all of that. I lived mail servers.
But I finally reached the point about four or five years ago where I no longer had to spend my days and nights managing the vagaries of Internet email delivery. I could move things into the cloud instead of having a rack of computers and a T-1 going into my linen closet by way of my bathroom mirror. I was able to step away from being an Internet mail management freakazoid.
The unintended consequence of that was finding my mail wasn't being delivered because I no longer operated a mail server to route it through. Don't get me wrong. I sure as heck don't miss the days of trying to get a million or so emails to deliver in a few hours. But it was still odd.
Here's the thing. There is a really easy solution that's surprisingly inexpensive. Use an enterprise hosting provider, but find one that doesn't require enterprise-level volume.
In fact, now I remember why I needed to route my email through a random SMTP server that came with my Web site account. After moving to the cloud, I used to route all my domains through Office 365. If you have an enterprise-class account, they'll let you do a lot of domain-fu, including multiple domains and routing.
A little over a year ago, I moved from Outlook to Gmail. I also moved off of an Office 365 mid-sized business account to consumer Gmail. I've been very happy with that move. I still have a personal Office 365 subscription for the desktop apps. but when I moved off the mid-sized business account, I moved off Microsoft Exchange.
As I recall, that was when I suddenly discovered I needed a place through which to route one of my personal domain email accounts. I probably tinkered around enough with Gmail's SMTP "Send mail as" features and found I could use one of my existing hosting accounts to get it done.
But what to do now?
As it turns out, you can get a Google Apps for Work account for five bucks a month, per user. I actually have a Google Apps for Work account I use for another purpose, for which I spend ten dollars a month (one account for me, one for my wife).
But here's where it gets nice. As it also turns out, Google Apps for Work will let you set up email domain aliases. You can run up to 20 email domain aliases through one $5 account. Plus, of course, Gmail works great with Google Apps for Work because they're essentially the same infrastructure.
So I created a new domain alias and instead of using hosting provider A's SMTP server, I'm now using smtp.gmail.com. All is now right in the world.
Lesson 4: Google Apps for Work is cheap and may be able to save your ass, especially if you're trying to do something in the Google universe.
So there you go. Four if it ain't broke, don't fix it lessons, derived from breaking things and then fixing them. What are some ain't broke, don't fix lessons you can share? Post in the discussion area below.
By the way, I'm doing more updates on Twitter and Facebook than ever before. Be sure to follow me on Twitter at @DavidGewirtz and on Facebook at Facebook.com/DavidGewirtz.