This has been an interesting week. What was supposed to be just a weekend project turned into a week-long research project. Along the way, I got to make a bunch of new friends, learned a lot, and actually made progress.
Before I go any further, let me say that I did get the iTunes store to accept my ePub file. This article will explain in detail what I tried and what I learned.
I started the process last week, when I attempted to sign in to iTunes Connect using my iPhone developer account and found it wouldn't work.
The next step was preparing the ePub and uploading it to the iTunes store. Everything validated, I got a big green check-mark indicating success, and yet the book didn't appear in iTunes Connect. I tried a bunch of different ideas, ran into mostly brick walls, and then wrote about my experiences.
My favorite TalkBack came from Edesw88 who got the Paranoid-Much? Award for thinking that perhaps Apple's servers blocked the book because it has the word "Jobs" in the title. We have posted our concerns in the past about Apple's unclear approval policy, but I'm pretty sure the book wasn't censored for political reasons.
Aside: As many of you know, I write a lot of politics-related material. Because I'm privy to some sensitive national security and counterterrorism details outside of the media realm, I self-censor much of what I write. In terms of media organizations and companies I've worked with, I've only been censored once for all I've written -- and that one time it was actually a polite request to not publish, rather than any sort of demand. That said, I will be self-censoring some information for this article.
In addition to being contacted by people who experienced the problem, I was also contacted by certain very knowledgeable people who gave me some very helpful hints. It felt like Christmas in June. What follows are those tips, some experiments I tried, and -- later -- my results.
Fixing the iBooks import problem
An ePub file is basically a zip file filled with a bunch of XML documents. The container.xml file contains a reference to the content.opf file, which, in turn, references the toc.ncx file. Here's the thing: the content.opf and toc.ncx files are key to making all this work.
This is the folder organization for my ePub. This isn't the only way to do it, but since InDesign exported using this structure, I've just left it as-is:
Content.opf contains a number of tag sections, including -- most importantly -- the manifest and guide sections.
Tip #1: The manifest must have a tag for every file (apparently except container.xml and mimetype) in your package. Don't leave any un-manifested files left over in your folder or package. This will cause a failure.
After the repeated import failures, I decided that since I was attempting to build this for an iPad, and since I was required to use the Mac for iTunes Producer anyway, I might as well suck it up and start building the package on the Mac. So I moved all my files over to the Mac, converted them so they all ended with Mac line-endings, and made sure all files were UTF-8 encoded.
Tip #2: Make sure all your files are either UTF-8 or UTF-16 encoded. Do not submit any plain ASCII files and do not mix encoding methods. All encodings must match.
I'd originally used UltraEdit on the PC, and when I moved the files to the Mac, I used TextWrangler. TextWrangler allowed UTF-8 encoding, but listed files as "(no BOM)" meaning the encoding isn't prefaced by a multi-byte bill-of-materials. This didn't seem to be a problem.
I also decided to assemble the .ePub file using the Mac. This was probably harder than it should have been, because most of the Mac zip tools I have are for extracting. Instead, I decided to use the command line. One plus for the Mac: it has a real *NIX command line.
I was fortunate enough to find a great tutorial from WebVivant with some interesting hints about properly formatting the command line.
Tip #3: Here are the three commands I used:
One interesting discovery during some of my conversations was that the preliminary validation doesn't really guarantee your book will properly import on the Apple store server-side.
Tip #4: Even though the files properly validate with an EpubCheck site and validate with iTunes Producer, they're not being tested for everything that Apple's system might require. Validation doesn't necessarily mean valid.
Tip #5: My .ePub file clocked in at about 557K, but the zip format can handle up to about 4GB. Your package size (heh, heh, he said "package size" -- settle down, Beavis!) must not exceed 4GB.
Finally, it was suggested that all the XML file references need to be properly URL encoded, so "02 History of Jobs.xhtml" would become "02%20History%20of%20Jobs.xhtml". I'd actually already done this, and the problem still was there.
I thought about this suggestion, though, and I realized that as a programmer, I often normalize everything I can, so that I can be sure of what I'm working with. That means that certain parts of my programs expect data formatted in certain ways, and expect file names to be encoded in certain ways. What if the Apple code wasn't cleaning this stuff up right?
I converted all file names to lower-case and removed spaces, replacing them with dashes. I got rid of all underscores and any other encoding. "02%20History%20of%20Jobs.xhtml" became "02-history-of-jobs.xhtml". I did this for all 57 files in my package. I then went into every file and changed all the existing file references to these new file names.
By this time, I also started wondering why I was doing all this for a book I'm giving away. I have a huge to-do list of better things to do. But, well, you know. Stubborn. Plus, it's for a nonprofit.
Tip #6: Remove all entity encoding and convert your files to the simplest, lower-case Linux-style formatting possible. Make sure all your file references in all your files reflect this.
Did it work?
So, did it work? Yes. Yes, it did. But not without another mystery.
And that brings us to the next phase of our discussion. I went through the now-familiar process in iTunes Producer of uploading the package to Apple. This time, the Package History didn't stop on "Import Error". This time it said "Uploaded" and, then, a few minutes later, "Imported".
I gave it about five minutes and then logged into iTunes connect. This time, Manage Books showed my book! Yay! Oh. Wait...
Yeah, it said "Partial Delivery". Now the thing is, I know how iTunes Connect normally works, at least for the iPhone developer side. If the little circle is yellow, it means your app is in review. And, for the developer side, your app never goes into review unless all the pieces have been uploaded.
Either all the parts hadn't been uploaded (but the screen shows both an import date and a cover graphic) -- or something wasn't right with the server software. Given how difficult it was to get the server to accept my upload, I decided to assume it had been uploaded and the problem was server-side.
I also wanted some Chinese food. I took a break, went for dinner, watched a little Top Gear (James May nearly crashed a flying camper trailer hanging under a blimp into a passenger jet -- can that show possibly be any more fun?), waited a few hours, and came back. This time, iTunes Connect said "Pending".
This was good. This was very good, indeed:
That matches with the iPhone app store process. Pending usually means Apple's reviewing it. I can't imagine Apple would just allow books onto its store without a review and approval process, so I'm guessing the book has just gone into some sort of approval queue.
OK, I'm not guessing. I know.
Hopefully, the book will be available for people to download from the iBooks store in about two weeks.
Some people suggested I should just send the book to an aggregator service company to do the formatting and upload. That is a valid approach. But I'm a techie and it just bothered me that what should have been a moderately simple process turned into a major hassle.
Then, when I got a lot of feedback about others in the same boat, it made more sense to try to solve the problem (or get some clues) -- and then share that information with others. It's called paying it forward, and it's always a good way to go.
A lot more (a lot, lot, lot) went on behind the scenes that I'm not telling you. I'm not holding back on any of the functional tips, so you should be able to use what I've discussed above to fix your ePub submissions.
I am convinced that Apple made some errors here, and it relates to the company's don't-ask-don't-tell policy. If this were a Google service, the iTunes Producer app and iTunes Connect would have had a big "beta" plastered all over it and publishers would have known you rolls your dice and yous takes your chances.
We also wouldn't have complained so much.
But this is Apple, a company known for both its excellent user interfaces and its troubling take-it-or-leave-it attitude. As a result, no one knew if all the problems being encountered were the result of merely a work in progress or how it would be to deal with the company into the future.
I can't answer that here, but after this very interesting week, I am moderately encouraged.
Download and dissect
I've uploaded the final ePub file accepted by the Apple system to the HowToSaveJobs.org download page. Feel free to download it and dissect it. Be aware that this is not perfect. Much of the metadata was created by InDesign and I'm just learning along with you. But the one thing I wanted was a working sample and couldn't find one. Here's one for you.
Note that the file is contained in a zip file. That's just because it's easier to feed zip files off the Web site that way. Inside the zip file is the .epub file. You can open that up with any zip program, either by changing the extension or by setting your program to open .epub.
Good luck, publish, and let me know how things turn out. Oh, and if you bake, send cookies.