Flickr: Too big to fail (We hope?)

Flickr: Too big to fail (We hope?)

Summary: Yahoo's financial troubles as well as recent examples of Flickr account mismanagement highlight the need for personalized disaster recovery and Cloud data permanence.

SHARE:
TOPICS: Outage, Google, Storage
24

Yahoo's financial troubles as well as recent examples of Flickr account mismanagement highlight the need for personalized disaster recovery and Cloud data permanence.

Imagine that you are a blogger, say someone who likes to take a lot of pictures and videos of things. Like myself.

Many bloggers, be it those with sponsored or advertising-free sites use services like Flickr, Google's Picasa Web Albums, PhotobucketYouTube and Vimeo to host this content, because it's much cheaper to pay a nominal $25-$50 per year fee for "Pro" access to each of these services than to try to host them on your own webspace, such as on WordPress.com, Blogger.com, Movable Type or elsewhere.

In terms of raw infrastructure cost and bandwidth, it makes a lot of sense, if you're a one-man/one-woman blogging shop with limited resources.

The problem is that when you outsource your blogging infrastructure to others, you're dependent on the reliability/resiliency of the Cloud rather than your own infrastructure, which by contrast you have full control over (as well as complete responsibility in terms of required maintenance). And if you use a mix of services, that reliability/resiliency may or may not be defined as under the Terms of Service for each of the service providers.

Here's the kicker -- if you read most of these, they're under no obligation to back up your data.

As I discussed in a previous article in November of 2008, "Preparing for a Flickr Apocalypse", losing your data hosted in the Cloud could be utterly catastrophic. Even having good backups of that data doesn't necessarily ensure you can put your blog back together again without a huge amount of aggravation and time sink.

This week, for a Swiss photo blogger, Mirco Wilhelmthat Flickr Apocalypse came.

Apparently, due to an error by one of Flickr's system administrators, Mirco's Flickr account was deleted, along with 4,000 of his photos. All because he reported to Flickr that another user had used his content without his consent. Instead of deleting the other user's photos, Flickr deleted his.

Whoops.

Now, this would be just fine, provided that Yahoo did snapshot-based SAN backups and second-tier tape archives of their storage for disaster recovery and for occasional mishaps like this. Sysadmins do make mistakes. It happens. All Yahoo would need to do is restore those backups and everything would be fine.

Also Read: Preparing for a Flickr Apocalypse (November 2008)

The problem is that Flickr doesn't appear to preserve historical data. So even though they were able to restore Mirco's account, they couldn't bring back his photos. And while Mirco has offline backups of his photos, there's no way to link them to the metadata -- the unique URLs -- that exist on his blogspace. So now he has thousands of broken links on his blog.

To me and many other bloggers who rely on Flickr and other content-hosting services, this is a massive wake-up call.

To Yahoo and Flickr, I say this: you had better get your storage and disaster recovery act together. Soon. And if you don't get Mirco's data fixed, you will be in a world of pain.

[UPDATED, February 3 2011: Flickr has restored Mirco Wilheim's photos.]

Yahoo, If you can't prove that your users can recover from this type of catastrophe, then expect them to start transitioning their data to other services such as Google's. Think of it as a potential "Bank Run" on your Cloud. I'm certainly considering hosting all of my new photos with Picasa already and thinking of ways I can transition all of my data over to a more financially stable company, eventually.

Two years ago, I was worried that Yahoo and the Flickr service could go under due to overall issues with the economy and Yahoo's thrashing about with Microsoft.

Today, with Yahoo's continuing financial woes, facing competitive challenges across the board due to ineffective leadership, with clear signs of asset consolidation and severe workforce reductions underway along with recent indications that if they screw up, you can't come back, I'm even more concerned.

[Next: The implications of a Flickr meltdown]»

If Yahoo were to shut down Flickr tomorrow, it would be utterly catastrophic. Literally hundreds of thousands of blogs and other websites (if not millions) would lose all of their photo content overnight. In my personal case, 90 percent of my food blog's content -- over 17,000 photos -- would simply vanish.

Many of my friends and peers sites in the food blogging community would be irreparably damaged, as much of them rely on Flickr as well due to its ease of use and excellent blog integration.

Of course, I have backups of all of the photos. I use services such as QOOP (which will mail you DVDs of your pictures for a nominal fee) as well as programs like Bulkr and Cloud-based replication services like Backupify. If you're a Flickr user, I encourage you to start using, them, stat. But even using these just isn't good enough.

As of today, there is no known way to take those photo backups, restore them to private webspace or another hosting provider of your choosing, and "automagically" fix all of your broken links on your blog by doing a data transform/parse on an XML backup of your blog posts. The Flickr metadata itself of how the directory structures are represented would need to be copied over to the new site, along with the canonical name change in the URL.

That's not including, of course, the descriptions, groupings, taggings, et cetera that go along with all of those photos on Flickr, if I wanted to painlessly move them to something like Picasa or Photobucket. And in Flickr's case, pictures can be stored in multiple resolutions. I only have the ORIGINAL uploaded, high-res files backed up, and that's 30GB of data, even though I only (tend to) link to the 500 pixel-width sized ones on my blog.

What we need is some sort of universal standard for preserving content. I should, depending on how my customer loyalty wind blows on any particular day, be able to easily switch from Flickr to Google Picasa. Or to PhotoBucket, or whatever else is out there, including my own private webspace.

There needs to be toolsets for doing this. Heck, Flickr needs to provide a means for users to back up their data repositories and restore them, just in case the users make accidental catastrophic mistakes.

My biggest concern is not just for my own personal blog, but for the retention and permanence of data as the years go on. Does anyone even remember SONY's Imagestation?  That content is long gone. How many countless memories from that site have been lost?

Flickr unfortunately is just too big to fail, regardless of Yahoo's financial situation. What we need is a data preservation and a disaster recovery strategy that works, for everyone's photos and content.

Are you concerned about the permanence of your online digital photo collections and your blog content links? Does Yahoo! and Flickr need to do good by its users in assuring their data and providing them tools to migrate and restore their photos? Talk Back and Let Me Know.

Topics: Outage, Google, Storage

About

Jason Perlow, Sr. Technology Editor at ZDNet, is a technologist with over two decades of experience integrating large heterogeneous multi-vendor computing environments in Fortune 500 companies. Jason is currently a Partner Technology Strategist with Microsoft Corp. His expressed views do not necessarily represent those of his employer.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

24 comments
Log in or register to join the discussion
  • Yahoo mail won't even let you use POP/IMAP without paying a fee

    If Yahoo won't let you download your mail into Outlook or a Gmail account without shaking you down, what are the odds of their providing a simple, affordable solution for migrating Flikr data, a much more difficult task?
    urbandk
  • Only

    a fool would trust cloud based storage for ANYTHING! Oh, wait . . . . . . . . . .
    sackbut
    • RE: Flickr: Too big to fail (We hope?)

      @sackbut

      and here's why we should not,ever:

      http://www.pcworld.com/article/218464/update_flickr_user_says_site_deleted_4000picture_account.html?tk=hp_ess
      knash1
      • The dude was asking for it.

        @knash1 Your example of why not is good ... but as an example of DAU (dumb azz user).

        The guy was a cheap bastard who decided to exploit a free server, did ZERO backup of is own, then complained when something happen to his account.

        If you don't pay for a service, you have no right to complain when something goes wrong with your account. Specially when you try to abuse the system for your personal benefit.
        wackoae
      • RE: Flickr: Too big to fail (We hope?)

        @wackoae
        <br>And if you do pay and it goes down, does that make your 'stuff' any less lost?
        Star*Hopper
  • Jason, why calling the blog yours when it is your wife who prepares Mexican

    food?

    She is a nice lady, give her proper exposure.
    DDERSSS
    • RE: Flickr: Too big to fail (We hope?)

      @denisrs she gets a byline. :)
      jperlow
  • I guess this article would matter

    I guess this article would matter if flickr was a backup service. Its not a backup service its a photo sharing service which allows you to organize your photos a number of ways. It doesn't advertise they are a backup service.
    All that being said they owe that photographer alot of money for the screw up but as far as backup is concerned its the photographers responsibility to find a backup service. Which NO online service can be trusted. Digital photos are lousy for the long term.
    Stan57
    • A free service owes nothing to users

      @Stan57 Unless the guy was PAYING for the service, he has no right to complain. He was technically abusing the system for his own personal benefit. The fact that he was (allegedly) saving all his precious photos in a free sharing service shows that he was trying to use the service completely outside the TOS.
      wackoae
      • RE: Flickr: Too big to fail (We hope?)

        @wackoae

        I certainly disagree with that. Things have changed, ad based models are often more successful than subscription based models these days and our patronage is the new payment and damn right we have a right to complain if it fails. What about the businesses that use Open Office or Google Docs for the business needs. Can all their docs just disappear because of some admin mistake and that is okay?
        Mythos7
      • RE: Flickr: Too big to fail (We hope?)

        @Mythos7

        Yes, it will be okay if all of their docs disappear because of an admin mistake. If you truly believe ignoring ads is enough payment for high uptime and multiple backups, in other worlds a business class service, then I hope you are not running a business with people depending on you.
        aep528
      • RE: Flickr: Too big to fail (We hope?)

        @wackoae he was a Flickr PRO user. A subscriber. Therefore, your argument is invalid.
        jperlow
      • Its better if you read the whole article.

        He did have backups of the originals but the links from his blogs were to that account and he will have to recreate the entire site structure, which yahoo should fork over some help on. And as a subscriber he has a right to not have to redo what he already spent time and energy on. But I agree that NO ONE should depend on the cloud alone for DR. I like to have an optical and magnetic copy of the most important stuff. If its not worth me taking the time to do that I have to be ok with the fact that it may not last for ever. Honestly a lot of ecrap that I have would be better as free space anyway!
        forbes9000
  • RE: Flickr: Too big to fail (We hope?)

    This is old news to many people who have had their account(s) deleted. They have THE worst business model possible. flickr has the attitude "I AM THE KING AND CAN DO NO WRONG". This happens with paid accounts as much as with free ones. I have been watching this since 2006. Complaints may get you another deletion as well. They do NOT care - they have too many sheeple happy to keep paying.
    [edit] "25 years of free Flickr Pro membership" <--- I'm taking bets that the account will be deleted long before that expires.
    daboogiemansass
  • RE: Flickr: Too big to fail (We hope?)

    "when you outsource your infrastructure to others, you?re dependent on the reliability/resiliency of the Cloud rather than your own infrastructure, which by contrast you have full control over (as well as complete responsibility in terms of required maintenance)."

    Yep, that's about it. Trust your life to The Google. :rolleyes:
    bb_apptix
  • RE: Flickr: Too big to fail (We hope?)

    So, eventually,when there are no other OS,but in the Cloud,
    we must pay-per-month for the OS,but,
    since the Cloud can't be trusted for storage,
    we make and keep our own backups..
    So,what is the Cloud useful for?
    Just so we can pay-per-month for an OS?
    The Cloud don't have to store anything for us,we don't trust it...
    So, what a deal for the Cloud makers...
    We pay,they don't have to deliver plastic disks.,
    so we can't even pretend to own the OS anymore..
    Constantly updated,without worry about virus,
    I suppose,is what they want for the future.
    Is this what the Cloud is all about?
    knash1
  • RE: Flickr: Too big to fail (We hope?)

    A cloud by definition is a continuously changing, non-permanent thing. Whether it's a cloud of water droplets or a cloud of data, neither will last forever.
    aep528
  • RE: Flickr: Too big to fail (We hope?)

    Same problem with Webshots. I lost many photos a few years ago, but they could not even admit what the issue was, nor offer any recourse. So my photos are no longer there...
    gdanen@...
  • RE: Flickr: Too big to fail (We hope?)

    After initially starting off hosting my photos on a great site fototime, I removed them when the price when from free to way more then free. Now I use JAlbum to make nice albums and host them on my own server. With dyndns and iis it's not that difficult, infact I taught my mother how and she now hosts her own site from her computer. I do not want to go down the road of having my 20,000+ photos disappear or held hostage to a sudden increase in fees.
    RichHalvor
  • RE: Flickr: Too big to fail (We hope?)

    Back in the ancient days of early computing, when the personal computer had just been born, the gray haired IT wizards and gurus of the day (they called it DP back then) taught us that backing up our data, and not just in digital form, was our responsibility. They made it VERY clear, as in "don't go flying without a parachute" terms, "if you don't back it up, don't come crying to me when it's all gone". <br><br>All during my early years of learning about computers and related technologies, this teaching was hammered into us, over and over. Why? Well, it seems kinda obvious ... without your data, the rest is kind of pointless, yes? <br><br>When did this fact fade so far into the background that people simply began assuming that someone else was taking care of our data? Backing up has never been cheaper or easier, and yet, it seems so few people do it. Why? Again, simple answer: it's easier to assume "that's really someone else's job". Guess what. It's not. <br><br>A few good rules of thumb:<br><br>a) Email - always have at least 3 accounts with 3 different email services (gmail, yahoo and hotmail, just as quick examples) and use the forwarding feature to send copies of what arrives in your primary inbox to the other two. And if that's not enough security for you, important emails can be saved to local drives.<br><br>b) Documents and Photos - As a writer and photographer, I care a great deal about what happens to the stuff I take lots of time and energy to create. When it comes to these things, I try to keep at least 4 copies, if not more, and at least two of those copies I want in my personal possession and not somewhere online. Big portable hard drives, DVD's, CD, USB drives ... there are plenty of options, and all pretty cheap. <br><br>c) Back Up Services - It may seem ironic to back up online data with an online service, but these are different humans than those responsible for your primary data. They have all signed written agreements to lose your data on different days and times than your email, photo and document services (ok, not really) . <img border="0" src="http://www.cnet.com/i/mb/emoticons/happy.gif" alt="happy"> But seriously, there are many online storage and back up services available to day. Use them. Several offer 2g or more of free storage and the paid storage is pretty darned cheap insurance if you care about your stuff. <br><br>Imagine driving a car without a seat belt. Imagine crossing the street without looking both ways. Imagine eating at a fast food restaurant. What? You do all those things? O ... well ... never mind then. <img border="0" src="http://www.cnet.com/i/mb/emoticons/wink.gif" alt="wink">
    Trep Ford