Six tips for getting Apple's iBooks to accept your ePub file

Six tips for getting Apple's iBooks to accept your ePub file

Summary: I got the iTunes store to accept my ePub file. This article will explain in detail what I tried and what I learned.

SHARE:
TOPICS: Mobility, Apple, Hardware
9

This has been an interesting week. What was supposed to be just a weekend project turned into a week-long research project. Along the way, I got to make a bunch of new friends, learned a lot, and actually made progress.

Before I go any further, let me say that I did get the iTunes store to accept my ePub file. This article will explain in detail what I tried and what I learned.

I started the process last week, when I attempted to sign in to iTunes Connect using my iPhone developer account and found it wouldn't work.

Read iTunes won't let you publish books if you write software

This wasn't a big deal-breaker. All I needed was another email address and I was in. Amazingly (for Apple), I had a signed and approved publisher contract almost immediately. By contrast it took something like three weeks for my personal iPhone contract to be approved and almost three months for my company's application to be approved.

The next step was preparing the ePub and uploading it to the iTunes store. Everything validated, I got a big green check-mark indicating success, and yet the book didn't appear in iTunes Connect. I tried a bunch of different ideas, ran into mostly brick walls, and then wrote about my experiences.

Read iTunes Publisher inexplicably fails to publish an iBook. Now what?

So, now you're caught up. What followed were two days with a lot of communication. I got a surprisingly large number of email messages from other publishers and individuals who were experiencing the same problem, along with many TalkBack postings from people who also experienced similar problems.

My favorite TalkBack came from Edesw88 who got the Paranoid-Much? Award for thinking that perhaps Apple's servers blocked the book because it has the word "Jobs" in the title. We have posted our concerns in the past about Apple's unclear approval policy, but I'm pretty sure the book wasn't censored for political reasons.

Aside: As many of you know, I write a lot of politics-related material. Because I'm privy to some sensitive national security and counterterrorism details outside of the media realm, I self-censor much of what I write. In terms of media organizations and companies I've worked with, I've only been censored once for all I've written -- and that one time it was actually a polite request to not publish, rather than any sort of demand. That said, I will be self-censoring some information for this article.

In addition to being contacted by people who experienced the problem, I was also contacted by certain very knowledgeable people who gave me some very helpful hints. It felt like Christmas in June. What follows are those tips, some experiments I tried, and -- later -- my results.

Next: Fixing the iBooks import problem »

Fixing the iBooks import problem

An ePub file is basically a zip file filled with a bunch of XML documents. The container.xml file contains a reference to the content.opf file, which, in turn, references the toc.ncx file. Here's the thing: the content.opf and toc.ncx files are key to making all this work.

This is the folder organization for my ePub. This isn't the only way to do it, but since InDesign exported using this structure, I've just left it as-is:

Content.opf contains a number of tag sections, including -- most importantly -- the manifest and guide sections.

Tip #1: The manifest must have a tag for every file (apparently except container.xml and mimetype) in your package. Don't leave any un-manifested files left over in your folder or package. This will cause a failure.

After the repeated import failures, I decided that since I was attempting to build this for an iPad, and since I was required to use the Mac for iTunes Producer anyway, I might as well suck it up and start building the package on the Mac. So I moved all my files over to the Mac, converted them so they all ended with Mac line-endings, and made sure all files were UTF-8 encoded.

Tip #2: Make sure all your files are either UTF-8 or UTF-16 encoded. Do not submit any plain ASCII files and do not mix encoding methods. All encodings must match.

I'd originally used UltraEdit on the PC, and when I moved the files to the Mac, I used TextWrangler. TextWrangler allowed UTF-8 encoding, but listed files as "(no BOM)" meaning the encoding isn't prefaced by a multi-byte bill-of-materials. This didn't seem to be a problem.

I also decided to assemble the .ePub file using the Mac. This was probably harder than it should have been, because most of the Mac zip tools I have are for extracting. Instead, I decided to use the command line. One plus for the Mac: it has a real *NIX command line.

I was fortunate enough to find a great tutorial from WebVivant with some interesting hints about properly formatting the command line.

Tip #3: Here are the three commands I used:

One interesting discovery during some of my conversations was that the preliminary validation doesn't really guarantee your book will properly import on the Apple store server-side.

Tip #4: Even though the files properly validate with an EpubCheck site and validate with iTunes Producer, they're not being tested for everything that Apple's system might require. Validation doesn't necessarily mean valid.

Tip #5: My .ePub file clocked in at about 557K, but the zip format can handle up to about 4GB. Your package size (heh, heh, he said "package size" -- settle down, Beavis!) must not exceed 4GB.

Finally, it was suggested that all the XML file references need to be properly URL encoded, so "02 History of Jobs.xhtml" would become "02%20History%20of%20Jobs.xhtml". I'd actually already done this, and the problem still was there.

I thought about this suggestion, though, and I realized that as a programmer, I often normalize everything I can, so that I can be sure of what I'm working with. That means that certain parts of my programs expect data formatted in certain ways, and expect file names to be encoded in certain ways. What if the Apple code wasn't cleaning this stuff up right?

I converted all file names to lower-case and removed spaces, replacing them with dashes. I got rid of all underscores and any other encoding. "02%20History%20of%20Jobs.xhtml" became "02-history-of-jobs.xhtml". I did this for all 57 files in my package. I then went into every file and changed all the existing file references to these new file names.

By this time, I also started wondering why I was doing all this for a book I'm giving away. I have a huge to-do list of better things to do. But, well, you know. Stubborn. Plus, it's for a nonprofit.

Tip #6: Remove all entity encoding and convert your files to the simplest, lower-case Linux-style formatting possible. Make sure all your file references in all your files reflect this.

Next: Did it work? »

Did it work?

So, did it work? Yes. Yes, it did. But not without another mystery.

And that brings us to the next phase of our discussion. I went through the now-familiar process in iTunes Producer of uploading the package to Apple. This time, the Package History didn't stop on "Import Error". This time it said "Uploaded" and, then, a few minutes later, "Imported".

I gave it about five minutes and then logged into iTunes connect. This time, Manage Books showed my book! Yay! Oh. Wait...

Yeah, it said "Partial Delivery". Now the thing is, I know how iTunes Connect normally works, at least for the iPhone developer side. If the little circle is yellow, it means your app is in review. And, for the developer side, your app never goes into review unless all the pieces have been uploaded.

Either all the parts hadn't been uploaded (but the screen shows both an import date and a cover graphic) -- or something wasn't right with the server software. Given how difficult it was to get the server to accept my upload, I decided to assume it had been uploaded and the problem was server-side.

I also wanted some Chinese food. I took a break, went for dinner, watched a little Top Gear (James May nearly crashed a flying camper trailer hanging under a blimp into a passenger jet -- can that show possibly be any more fun?), waited a few hours, and came back. This time, iTunes Connect said "Pending".

This was good. This was very good, indeed:

That matches with the iPhone app store process. Pending usually means Apple's reviewing it. I can't imagine Apple would just allow books onto its store without a review and approval process, so I'm guessing the book has just gone into some sort of approval queue.

OK, I'm not guessing. I know.

Hopefully, the book will be available for people to download from the iBooks store in about two weeks.

Final thoughts

Some people suggested I should just send the book to an aggregator service company to do the formatting and upload. That is a valid approach. But I'm a techie and it just bothered me that what should have been a moderately simple process turned into a major hassle.

Then, when I got a lot of feedback about others in the same boat, it made more sense to try to solve the problem (or get some clues) -- and then share that information with others. It's called paying it forward, and it's always a good way to go.

A lot more (a lot, lot, lot) went on behind the scenes that I'm not telling you. I'm not holding back on any of the functional tips, so you should be able to use what I've discussed above to fix your ePub submissions.

I am convinced that Apple made some errors here, and it relates to the company's don't-ask-don't-tell policy. If this were a Google service, the iTunes Producer app and iTunes Connect would have had a big "beta" plastered all over it and publishers would have known you rolls your dice and yous takes your chances.

We also wouldn't have complained so much.

But this is Apple, a company known for both its excellent user interfaces and its troubling take-it-or-leave-it attitude. As a result, no one knew if all the problems being encountered were the result of merely a work in progress or how it would be to deal with the company into the future.

I can't answer that here, but after this very interesting week, I am moderately encouraged.

Download and dissect

I've uploaded the final ePub file accepted by the Apple system to the HowToSaveJobs.org download page. Feel free to download it and dissect it. Be aware that this is not perfect. Much of the metadata was created by InDesign and I'm just learning along with you. But the one thing I wanted was a working sample and couldn't find one. Here's one for you.

Note that the file is contained in a zip file. That's just because it's easier to feed zip files off the Web site that way. Inside the zip file is the .epub file. You can open that up with any zip program, either by changing the extension or by setting your program to open .epub.

Good luck, publish, and let me know how things turn out. Oh, and if you bake, send cookies.

Topics: Mobility, Apple, Hardware

About

David Gewirtz, Distinguished Lecturer at CBS Interactive, is an author, U.S. policy advisor, and computer scientist. He is featured in the History Channel special The President's Book of Secrets and is a member of the National Press Club.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

9 comments
Log in or register to join the discussion
  • How many apple post in zdnet for a small percent customer base.

    I don't remeber seeing any sharepoint article here. And many other things. Did blogger lose contact with the real world?
    FADS_z
  • It's the title that triggered the "Pending"...

    ... the system probably thinks it's anti-Steve propaganda.
    nix_hed
    • RE: Six tips for getting Apple's iBooks to accept your ePub file

      @nix_hed - quite possibly
      :(
      HypnoToad72
  • Top Gear!

    Just made my day my friend. Love that show! Go Hamster =)
    privatejarhead
  • RE: Six tips for getting Apple's iBooks to accept your ePub file

    David, I had some interesting experiences with the ePub files that you posted. The one that is currently posted, which supposedly passed Apple's upload criteria, could not be processed by iTunes or Calibre. Couldn't read any of the metadata. The one you posted a few days ago was perfectly readable by Calibre and I was able to use Stanza on my iPhone which easily detected the Calibre server running on my Mac. I also tried directly adding it to my iTunes library. It was added and showed up in the Book list, with metadata visible thru "Get Info". However the cover art was not rendered and a generic book icon appeared in iTunes.

    Regardless, your book is on my iPhone and nicely readable in Stanza, which was my main goal.

    Thanks for this gripping saga. I suspect a sequel may be in the cards.
    NJSteveK
    • RE: Six tips for getting Apple's iBooks to accept your ePub file

      @NJSteveK

      I did some follow-up testing and found that it seems to read pretty well at my end. We'll see if we get more feedback about it not working, but at this point I'm thinking (and, honestly, hoping) it's something on your side. Maybe a bad download?
      David Gewirtz
  • How much is Apple's cut of the profits compared to real books?

    I don't care for e-devices in general when reading material - eye strain. ;)
    HypnoToad72
    • RE: Six tips for getting Apple's iBooks to accept your ePub file

      @HypnoToad72
      Why I do most of my electronic reading is via my Nook...
      No eye strain.
      rhonin
  • RE: Six tips for getting Apple's iBooks to accept your ePub file

    Congratulations on getting published. Is there anywhere a less savy person could learn how to submit an ibook? Thanks
    sysil