"How's your week been?"
"Nothing too unusual. Went on TV yesterday to help users understand the LastPass breach. Been working on PowerPoints for a briefing. Oh, and I accidentally crashed somewhere between 10,000 and 70,000 nonprofit Web sites."
So yeah. That happened.
This weekend, after working for three months on an open-source plugin I adopted back in March, I released a major update. Despite all my best efforts (and, in fact, because of some best practices), the release had a naming conflict and caused a whole lot of Web site crashes.
Three months ago, I learned that a very well-respected GPL open-source WordPress plugin, Seamless Donations, was about to lose developer support. Its original developer had a full-time job that needed his full-time attention.
As it happens, I was also looking around for a programming-related side project (after spending 18-months writing academic papers, I wanted to freshen my coding skills), and since I have already done a lot of work in WordPress and support nonprofit work, I thought this would be a good fit for my next side project.
On March 16, I took over the coding work for Seamless Donations. Because the plugin had something like 56,000 installs back then (and adding about 2,500 a week), it also had a very active user base. These folks regularly posted support questions on the WordPress.org boards and had a lot of needs for changes in the plugin's functionality.
Without going into too much detail, the original plugin was built by an exceptionally gifted programmer, but he built the plugin monolithically. In other words, there was nothing modular or extensible about the code. To add the features and be able to make changes in the plugin, it needed to be modular and extensible.
So I undertook the process of rewriting the plugin from the ground up to make it both modular and extensible. That would take it from version 3.3 to 4.0. I also wanted to change some of the interface because users had reported some problems, and it was a little dated. So I converted from storing data as packed options and turned things into custom post types (so the donor, donations, and fund data could be managed like any other WordPress data).
Because the donation form was hard-coded, there was no modularity. So I wrote an entirely new forms engine that is completely dynamic and modular.
Finally, my biggest concern was the actual upgrade. I was keenly aware that this plugin was installed in live, active sites providing the lifeblood cash flow to nonprofits. I didn't want to do anything that would interrupt their operation.
Yeah, well, so much for that
To prevent any sort of interruption, I essentially built two systems inside one plugin. I kept the entire running legacy environment with its legacy data structure, so when the plugin would auto-update (WordPress allows plugins to update with a single click), it wouldn't break any of their operations or hurt any of their code.
I recorded a video explaining how the upgrade would operate. I had a very big button telling users to test and stage and backup before clicking over to the new version. I beta tested the heck out of the thing. (As soon as I adopted the plugin, I set up a mailing list so users could sign up and be notified of changes.)
I took every precaution I could possibly think of -- many of which required days of additional work -- just to be sure the update went smoothly.
All told, I put about 360 hours into the plugin -- every weekend day (and night) plus some free nights during the week -- since March 16. Out of pocket expenses turned out to run nearly $1,000. I needed to upgrade my development tools (which were two years out of date). I also needed to buy a distribution license for a library used to drive the plugin. All told, that cost about $500.
An unexpected expense was the use of outside services, which cost about $500. My wife and I live in a fixer-upper that we've been working on for the past few years. By giving all my spare time to the plugin, I wasn't able to work on the house and the yard, which meant my wife had to turn to local contractors and service providers. I got out of mowing the lawn, but I had to pay a service to come do it.
With roughly 24 days when I wasn't available to do home-owner activities, it cost us quite a bit for service providers to pick up the slack. Now, don't get me wrong. I would vastly prefer to be in the air conditioned comfort of my Bear Den writing a forms engine than outside in 90-degree temps riding a lawn tractor or laying out pavers. But, still, it was an expense.
So, yeah, 360 hours. About a thousand bucks. All my free time. All a volunteer effort.
I released Seamless Donations 4.0 to the WordPress.org repository Sunday night. Despite all that work, and despite my constant attention to a smooth and breakage free update, I got a steady stream of "my web site is broken, my plugin won't upgrade" messages starting Monday morning.
Here's what happened...
It turns out this is a behavior that could only happen when the update went live, which is why it was never found during beta testing.
Every plugin has a plugin comment block in one main file that defines the plugin. In the case of Seamless Donations 3.3, the plugin comment block was in a file called dgx-donate.php, located in the seamless-donations directory.
When I first adopted Seamless Donations and looked into best-practices, I read that it's best to have the main plugin file and the plugin folder use the same name. So I created seamless-donations.php and moved the plugin comment block into that file. I then moved dgx-donate.php into a legacy folder with other older code.
Seemed like a good idea at the time.
As it turns out, when WordPress activates a plugin, it records the folder/main-file pair in the active_plugins option in the wp_options table. That means that WordPress recorded seamless-donations/dgx-donate.php as the active plugin information. When the plugin updated, the files in the seamless-donations folder changed, and there was no longer a seamless-donations/dgx-donate.php file where WordPress expected it.
And the upgrade broke. Hard.
There was no way to test this, because the failure occurred during the upgrade process in the live system. While you can test and stage plugins, you can't test and stage repository upgrades. Other than abiding by "if it ain't broke, don't fix it" -- which I blatantly ignored in this project -- there was really nothing I could have done.
I immediately went into triage mode. I contacted some of the nice volunteers at WordPress.org, who advised me to just stick it out, because reverting to the old name would cause more problems. They also suggested I apologize to users. A lot.
I wrote a detailed "how to upgrade" post and posted it on my Lab Notes site. I posted notes in the plugin's support forums. I did a rushed MailChimp mailing to the 710 users who had signed up for the list. And, thankfully, the panicked complaints have slowed down.
There are some bits of good news in this story
New users don't appear to be having any problems with the the upgrade. One of the other big features I added to the upgrade was systematizing the text elements in the source code so they could be easily translated into other languages. Already (since Monday), there are translation efforts to create a Mexican Spanish and Portuguese translation.
So where does this leave us? From my perspective, I feel a huge sense of responsibility to the operators of these nonprofit Web sites. Over the past three months, I've come to know some of them and their great causes.
I may not be spending my time volunteering at a local soup kitchen, but if I can help soup kitchens (and churches, and animal rescue shelters, and women's shelters, and reproductive services volunteers, and health services, and jobs services and on and on), then that's worth giving up some weekend time.
For the next few weeks, I'll work on any bug fixes this release uncovers and then, depending on how stable everything winds up being after the naming faux pas -- and, wife-permitting -- I'll look into adding some of the features and capabilities users have requested.