ie8 fix

Sidekick outage says more about the future of 'Pink' than Microsoft's cloud

By | October 13, 2009, 5:48am PDT

Summary: I’m not downplaying in the least how serious the ongoing Sidekick outage is or letting Microsoft’s Danger subsidiary — and/or any other companies involved in the loss of users’ data — off the hook. But, to me, the outage says more about the future of “Pink” than it does about “the Microsoft cloud.

I was (trying to) unplug most of the long holiday weekend, but couldn’t help but read all the headlines about how the Sidekick outage spells trouble for Microsoft’s cloud strategy.

I’m not downplaying in the least how serious this outage is or letting Microsoft’s Danger subsidiary  — and/or any other companies involved in the loss of users’ data — off the hook. (The latest: The Danger team is now saying that “some” user data might be recoverable, after all.)

But Sidekicks aren’t running from/on “the Microsoft cloud.” In fact, there is no such thing as a single Microsoft cloud. Microsoft has lots of different remote servers in different data centers running lots of different services.

The Microsoft Azure cloud is what many Microsoft watchers think of these days when someone says “the Microsoft cloud.” But the Azure environment provides the underpinning for very few Microsoft services so far. The Sidekick services don’t run on Azure. Microsoft’s My Phone doesn’t run on Azure. Hotmail, Xbox Live,  Microsoft-hosted Exchange — nope, nope and nope. None of these are running on Azure yet.

The Sidekick outage, to me, says more about Microsoft’s Pink than it does about Azure.

The Danger team, which Microsoft acquired in 2008, is largely responsible for the Pink “premium mobile experience” (PMX) software/services on which Microsoft has been working on secretly. There have been a few recent reports that Microsoft has decided against launching the Pink phone(s) that were going to run these services, but I haven’t heard from any of my sources whether this is true. Sharp supposedly is the manufacturer of the Microsoft-branded/co-branded Pink phones, which Microsoft is said to be planning to market primarily to teens and 20-somethings.

(Microsoft officials have denied repeatedly assertions that Microsoft is making its own smartphone. They have not denied that Microsoft is working with a hardware partner to build Microsoft-branded or co-branded phones.)

What was Microsoft doing to the back-end Danger services that resulted in such a catastrophic outage? Microsoft isn’t talking. There are rumors the problem stemmed from a storage-area-networking debacle but Microsoft isn’t confirming that, either.

The services at the core of Danger’s current offering — contact management, calendaring, instant messaging, e-mail — are all running on a back-end platform that Danger doesn’t describe publicly. Here’s the platform diagram it does provide (click on it to enlarge):

Because Microsoft hasn’t yet launched Pink, company officials have refused to talk at all about which premium services it will encompass and what kind of back-end platform they’ll run on. Is Microsoft designing the Pink services to run on its own servers? Is/was Microsoft intending to allow the Pink services to remain hosted on the existing Danger back-end? Did this past week’s Sidekick outage result from Microsoft (or Hitachi Data Services, or whoever) attempting to move the Danger back-end off the existing servers and onto Microsoft’s own servers? Microsoft officials won’t say.

Now that more everyday users know that Sidekick is connected to Danger and Danger to Microsoft, this week’s outage will cast a shadow over any kind of Pink phone and/or Pink premium services launch Microsoft may be planning.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Topics

Mary Jo has covered the tech industry for more than 25 years for a variety of publications and Web sites, and is a frequent guest on radio, TV and podcasts, speaking about all things Microsoft-related. She is the author of Microsoft 2.0: How Microsoft plans to stay relevant in the post-Gates era (John Wiley & Sons, 2008).

Disclosure

Mary-Jo Foley

Freelance journalist/blogger Mary Jo Foley has nothing to disclose. WYSIWYG (what you see is what you get). I do not own Microsoft stock or stock in any of its partners or competitors. I have no business ventures that are sponsored by/funded by Microsoft or any of its partners or competitors.

Biography

Mary-Jo Foley

Mary Jo Foley has covered the tech industry for 25 years for a variety of publications, including ZDNet, eWeek and Baseline. She has kept close tabs on Microsoft strategy, products and technologies for the past 10 years. In the late 1990s, she penned the award-winning "At The Evil Empire" column for ZDNet, and more recently the Microsoft Watch blog for Ziff Davis.

Got a tip? Send her an email with your rants, rumors, tips and tattles. Confidentiality guaranteed.

Related Discussions on TechRepublic

Did you know you can take part in these discussions with your ZDNet membership?
45
Comments

Join the conversation!

Just In

RE: Sidekick outage says more about the future of 'Pink' than Microsoft's cloud
makrekdw45-24353611923087901074498125825970 Updated - 11th Nov
Heya, your web page on the web is for the most part spectacular, I had any another person 49ers jerseys challenge san francisco 49ers jerseys requirements happy Feel 49er jerseys you utilize Joomla or simply a a lot more script, do you really guidelines to me ?
0 Votes
+ -
This isn't about any service.
bjbrock 13th Oct 2009
This is about Microsoft in general. It brings up the question of whether they can manage a server environment. It doesn't matter what that environment is. Preservation of data is always job one. And this is the job where the failure occurred. So Pink, Azure, cloud, none of this matters. Keeping data safe is a very basic job that applies to every environment. This was a failure at the most basic of IT administration levels. So again, it has nothing to do with the cloud or any service running in the cloud. It only has to do with the most basic of IT administrations. The cloud didn't fail, Pink didn't fail, Azure didn't fail, no service or "as a service" failed, Microsoft failed.
0 Votes
+ -
Amen
rynning 13th Oct 2009
There is no way to sugarcoat what happened, though I see a lot of people trying (e.g., this blog post).

On the other hand, I believe that the failure was on the part of a handful of people within MS who didn't follow the company's policies and procedures. So how far up the chain do you punish? Manager, director, VP, CEO?

Is this indicative of a general lack of control or renegade mentality within the company, all the way up to the top? I'm not sure yet, but it will put this question in the minds of their lifeblood: corporate buyers.
0 Votes
+ -
As you indicate...
bjbrock 13th Oct 2009
there are a lot of unanswered questions. But I'm of the opinion that the buck stops at the top. We don't really know where the system broke down if it did break down or if it was just a poorly designed system.

It boggles my mind that the data wasn't backed up somewhere away from the system they were working on.
  • Flagged
0 Votes
+ -
Before crucifying Microsoft ...
Soyale 13th Oct 2009
Check the backup policies of other cloud services. It's not uncommon to lose photos on Flickr, Picasa etc.

Off site backup makes sense for an enterprise with a few terrabytes of data. The sheer volume of cloud data makes the sort of "off site backup" you're suggesting incredibly impractical. The likelihood and cost of losing a few photos and associated bad will is insignificant compared to the cost of performing regular complete backups of that volume of data.
0 Votes
+ -
Wow.
rynning 14th Oct 2009
"The likelihood and cost of losing a few photos and associated bad will is insignificant compared to the cost of performing regular complete backups"

It was more than a "few photos" and, obviously, you don't own a SideKick. The MS apologists are out in force.
0 Votes
+ -
Did the Titanic sink again?
dariquew@... Updated - 14th Oct 2009
this isnt some small upstart company by some
nerds with no money for real storage and safety
precautions. they can afford to take care of
safety first. If youre running an environment
like this you have to expect his exact issues.
SH*T HAPPENS. DATA GETS LOST. BACK UP ALL
SERVER INFO DAILY. im smh on this one. MS,
Danger is your problem now and T-mobile can
totally throw you under the bus on this one.
good luck, im still loyal but, get your
stuff together
0 Votes
+ -
Exactly
jeremychappell Updated - 13th Oct 2009
Mary-Jo is missing the whole point. The Danger platform was fine,
Microsoft botched up the upgrade at the data centre and didn't have a
backup. It's pretty clear, Microsoft showed by their utter incompetence
they aren't up to the job (or at the very least they weren't up to the
job). So yes, it reflects on everything Microsoft does. Let's face it,
having a backup is not a hard concept when you're about to make
changes to a data centre.

@Mary-Jo: Have a think about it, forget the technology questions, just
think about the management issues - on a scale of 1 to 10 how stupid
do you need to be to make changes like this WITHOUT an offline
backup? Don't let the buzzwords fool you, they were pretty stupid.
Now could they be that stupid with Azure? One would hope not, but
I'd never have believed they could be that stupid with
Danger/Sidekick! This is a shocking insight into the inner workings of
Microsoft - yes one that shows them at their worst, but who would
have believed things could be this bad?
0 Votes
+ -
Actually...
dtn_cybermail@... Updated - 13th Oct 2009
While this incident is about grossly-poor
management (backing up is COMPUTING 101, not
just IT management), it still does not change
the fact that EXCLUSIVELY cloud computing is a
massive failure waiting to happen. And the
reasons are so simple. When your data and apps
(i.e. all your productivity) are in the hands
of a third party at a remote location:

- their efficiency will make or break your
business. Yes, already businesses have to
depend on partners (shipping, assembly, etc.),
but perhaps not to this level.

- their integrity (or integrity of an employee,
which they cannot fully guarantee) can protect
or ruin your business. How long before someone
sells company secrets to a rival for a few tens
of millions of dollars? Think it won't happen?
I GUARANTEE it will; they simply need to cover
their tracks such that the blame will be on the
company and cannot be resolved to one
individual.

- service downtime or outage turns your
computers into paperweight; annoying for the
home user, calamitous for a company.

- YOUR Internet service outage turns your
computers into paperweight. Right now, an ISP
outage will not stop your from completing that
report and slideshow presentation for the
meeting at 2. With cloud computing, the
incomplete documents as well as the apps used
to create them are inaccessible!

Cloud computing exclusively WILL NOT WORK; it's
not a matter of logistics but common sense.

Productivity needs to exist, at least to a
capable extent, offline and without the Web.

Some data are simply too sensitive to be placed
on a computer on the Web, even if the computer
itself were local (some data are best kept off
a computer altogether - like documents in a
safe or vault with no digital, easily copyable
access).

The answer will be a hybrid system: some
functionality on the cloud for easy and mobile
access, manipulation and synchronization;
others local access (apps and files) for
guaranteed access and security, and where
necessary, utter privacy. And synchronization
between the data both host.
0 Votes
+ -
This is about Microshaft's Incompetence
Scalor Updated - 13th Oct 2009
This situation just illustrates that Dibert's Peter Principle is alive and well at Microsoft. If I hadn't already known that Scott Adam's comic was based on his experience at PG&E, I would have guessed it was Microsoft.

A while ago on Leo LaPorte's This Week In Tech, a guest commented that MS had many great programmers who got promoted into middle management and are terrible at managing.

After this and the Vista fiasco, how could anyone with any rational sense trust anything that Microsoft puts out or is behind? I certainly don't.
0 Votes
+ -
Managers...
Ceridan 13th Oct 2009
Actually someone that is either good or great at one thing may be a bad manager... or even someone that is bad at alot of things but is a personnal friend of the CEO or other high level personnal may be put at a managing position... regardless of the fact that the guy is horrible at managing.

So it's not a problem with MS only.
0 Votes
+ -
Food for thought.

Windows Live, hotmail, hosted exchange, hosted sharepoint, never lost anything

Danger is a microscopic part of Microsoft. A recent acquisition, and apparently everything is offshored.

If you tell say, "ho Hotmail is special" because it's an acquisition. Well, Danger is as well, it's not "more Microsoft" than the services that have no problem.

0 Votes
+ -
Loverock?
No_Ax?
Mike Cox?

Bueller?
0 Votes
+ -
Um
Cylon Centurion 13th Oct 2009
If they're an ABMer they're usually the ones bashing Microsoft...

Anything But Microsoft...

0 Votes
+ -
ABM vs ABM
utoddl 13th Oct 2009
Always Buy Microsoft
0 Votes
+ -
Doh
Cylon Centurion 13th Oct 2009
>.
0 Votes
+ -
ah...
Ceridan 13th Oct 2009
Oh great it's the war of the same anagram but different meaning.
both pro and con, so they may as well use the same acronym. Keeps it simple.
0 Votes
+ -
Perhaps they're waiting for the facts
Johnny Vegas 13th Oct 2009
Do you know Danger didn't disclose their infrastructure and it's shortcomings to TMobile long ago? Do you know that TMobile didn't decline to invest in it because they don't make any data reliability claims to their customers? Perhaps they clearly state that customers are responsible for their own backups in order to reduce service cost/price. Perhaps MS revisited these issues with TMobile and Danger when they bought them and TMobile declined again. I don't know do you? It's stupid to waste time speculating without facts.
0 Votes
+ -
Maybe they have Sidekicks...
phatkat 13th Oct 2009
and the lost their service, thus the silence.
It is scary that Microsoft that prides itself being in the Enterprise could lose that information for those users and T-Mobile.
0 Votes
+ -
No Surprise: Something Crashed. Oops.
i-want-gizmos 13th Oct 2009
I am not surprised by this Sidekick debacle. I would not be more surprised if Microsoft or its Pink project people were doing something on the actual Danger/Sidekick production servers that somehow crashed it all in a horrible corruption event. What you didn't do a backup in how long? What no backups were budgeted anymore for Sidekick because Microsoft is drop-kicking Sidekick in favor of Pink? Oops. More than a big Oops. But a you-are-dead-fired and pink-is-now-a-dead-project Oops. You know that someone somewhere is getting totally fired over this. Meanwhile, the finger-pointing has all the public blame focused on Hitachi and T-Mobile as the bad guys.
0 Votes
+ -
What platform DOES Danger run on?
apexwm 13th Oct 2009
And the platform that Danger backend DOES run on is????????? Windows perhaps?
0 Votes
+ -
Danger platform
Wittsend 13th Oct 2009
To the engineers familiar with Microsoft?s internal operations who
spoke with us, that suggests two possible scenarios. First, that
Microsoft decided to suddenly replace Danger?s existing infrastructure
with its own, and simply failed to carry this out. Danger?s existing
system to support Sidekick users was built using an Oracle Real
Application Cluster, storing its data in a SAN (storage area network) so
that the information would be available to a cluster of high availability
servers. This approach is expressly designed to be resilient to
hardware failure.

Microsoft is well known for wanting to replace competitor?s
technologies with its own. The company famously failed to do this
after buying up HoTMaiL in 1996 and attempting to replace its Sun
Solaris servers with PCs running NT; it similarly failed to smoothly
transition WebTV from its original Sun-infrastructure to one based on
Windows Server and WinCE clients in the late 90s. Microsoft also
struggled to help Dell replace its WebObjects-based web store after
Apple bought NeXT in 1997.
0 Votes
+ -
source needed
smkudelko 13th Oct 2009
You really should have cited Daniel Eran Dilger on either Roughly Drafted
or Apple Insider for copying a part of his article verbatim as your reply.
0 Votes
+ -
SUN / Solaris / J2EE / Oracle
Johnny Vegas 13th Oct 2009
According to the so far published accounts it's all still run on Danger's original non cloud architecture in Danger's datacenter just as it was pre MS acquisition. What does that matter if Hitachi messed up the SAN and the backups? What does it matter if Hitachi didn't mess up the SAN?

Please add facts if you actually have any and leave the idiocy to the ABM fanbois living in their parents houses...
0 Votes
+ -
Follow Google Advice...
jznoy-dallas 13th Oct 2009
As google often states, with every mishap they learn and refine to create a better product for the consumer. Let's just hope MSFT learned to create a stronger Pink product upon release.
0 Votes
+ -
What?
PIXguy 13th Oct 2009
So basically you are suggesting managers follow the new IT philosophy where management is completely ignorant of the technology they are in charge of. So they vend everything out and rely totally on the vendor to do the right things. And you accept the ridiculous statement that "every screw-up makes our product better". There's no one left in IT management with the skill or experience that would allow them to think ahead and PREVENT stupid mistakes from happening. Vendors need to be managed, you know.
0 Votes
+ -
So are YOU
Pete "athynz" Athens 13th Oct 2009
saying you do not learn from you own screw ups? That when you make a mistake you will repeat it because you did not learn from it? Or when you screw up you need to be managed? Personally I learn from my mistakes and very seldom repeat them - And so do quite a few people. And in doing so I improve. THAT is the theory behind that statement you find so ridiculous.

I do not disagree that vendors need to be managed - but I fail to see why learning from one's mistakes and managing vendors should be so mutually exclusive. Perhaps in your all knowing wisdom you could explain this to me.
C'mon

Even the lowliest IT guys, like me, know that.
0 Votes
+ -
"But Sidekicks aren?t running from/on 'the Microsoft
cloud.'"

Does it really matter?

How is the end user going to know if their data is stored
in a "true cloud" or not?

This makes foresight difficult, even if hindsight is
20/20.
0 Votes
+ -
You've all missed the bigger issue...
Heatlesssun 13th Oct 2009
Yes this was bad and too much will be said about that anyway.

The question is do you have you OWN backups? Stuff happens. Depending on a third party to for ever and always keep your important data is asking for trouble.

I'll NEVER trust any cloud service to be failsafe. I'll ALWAYS have backups and redundancy of of own.
0 Votes
+ -
True, however
Michael Kelly 13th Oct 2009
Better tools to make your own backups need to be available. It wasn't impossible to back up contacts and e-mail on a Sidekick, but they sure as heck didn't make it easy or obvious.
0 Votes
+ -
I agree!
JAG39 13th Oct 2009
From the first moment I heard about "The Cloud," I new that someone, sometime was going to lose their shirt. The reason? People were going to trust someone else to guard their data.

I don't trust ANYONE to do that. I don't care whether it is Microsoft, Apple, or Google. I will take tare of my own data, thank you.

I will also have my apps on my computer. I don't want to be working on some important project, and suddenly realize my internet connection is down.
0 Votes
+ -
good point, except
unclefixer@... 13th Oct 2009
as it's been noted in some other posts, Sidekicks are not easy to back up. They're not like BlackBerries or iPhones where you can connect them to your desktop and sync with Outlook or your music folder or whatever. The only way to backup Sidekicks, that I know of, is either by the server which crashed, or with a paid piece of software you have to first convince TMobile that you need before they'll even let you purchase it. This would be ok, but this just means that you really have no practical choice but to depend on what's available.
I made the mistake of buying my daughter one of these POS- and had such a terrible experience with the constant charges and crappy performance, lack of features, completely closed architecture (not even a third party backup app available) that even though she asked for another one, I had to draw the line.
I honestly wouldn't trust anyone to be the sole provider of your data reliability or stability. You should always get the key data synced. Ideally apps ready via a Disaster Recovery or Business Continutity Plan but at least get a copy of the backup/data...

Regardless of the service provider, contacts and other critical information should always be at least synced or backed up locally.

I personally store my contacts on my phone (encrypted with password) which are synced with my laptop which I auto-backup to my PC when I am home (about twice a week). That PC is then auto-backed up online (carbonite, etc).

No excuse for a failure of this magnitude.
0 Votes
+ -
again, the words of a non-sidekick user
unclefixer@... 13th Oct 2009
In most cases, you would have a point. In this case, Danger used absolutely no common sense in designing this pos-
They cannot be just "backed up."
PLEASE, someone correct me if I'm mistaken, but what does it say when you reset the phone by taking the battery out?
I keep all my contacts in mighty Google's Gmail.

I also keep a copy backed up to my own hard drive. How stupid are you not to?
0 Votes
+ -
What SK backup?
jonniva 13th Oct 2009
The network-based desktop interface IS the backup. My guess is there is more to the story, and it won't be good news. My exp w/ SK, your backup is to logon to the desktop service and print your contacts and calendar from the screen. One of the reasons I left the platform. Oddly, the very closed nature of that system was supposed to be more secure.
0 Votes
+ -
@ jonniva
unclefixer@... 13th Oct 2009
It seems as though we're the only people on this board who have heard (or experienced) what you're explaining here...
www.dfwsupergeek.com
It has been said before, and it bears repeating again. The cloud is no place for your data. If your data is in someone's hands other than yours, you don't have data security.
0 Votes
+ -
I am severely shaken
bizcad 13th Oct 2009
This data loss negates one of my primary reasons for keeping Sidekick. One of the reasons I stick to my old Sidekick 3 is that my data is backed up on a server.

Over the years I have lost or destroyed my Sidekick two or three times, and I was able to recover by buying a new phone and sim card.

I do not actually know if they lost all my stuff because I have kept my sidekick off and powered up since the loss. One of these days I will turn it back on to find out the bad news.
0 Votes
+ -
Its easy to point fingers ...
LifeNoBorders 13th Oct 2009
I suspect they were taking backups but had not been testing complete D/R restores. In complex interdependent infrastructures, small, yet critical things can get missed (e.g. Oracle control files). I've seen _many_ environments where normal operational restores worked fine but D/R recoveries from tape just failed dismally. Other protection methods like synchronous mirroring (which they probably had as well), doesnt protect you against the "oops I just dropped the wrong tablespace" admin style errors, and sometimes subtle corruption which prevents systems from coming back from a crashed state gets preserved in backups.

In my experience errors of this magnitude requires a "perfect storm" of failures across multiple systems and diciplines. Simply blaming Microsoft, or Danger, or HDS, or the storage/backup admin with "THEY SUCK" doesnt help anyone, and prevents the kind of transparency that is required to prevent this kind of thing happening elsewhere.

IT could learn a lot from the way this kind of thing is dealt with in the healthcare system.

IBM Launched PINK in 1992. it was the great take on Apple that vanished very quickly. It supposedly got rid of the blue screen of death but did't deliver (early vista).
This is 100% about Microsoft's ability to host, which it would like to be
- but is obviously not - their core competency.

When they bought Danger Microsoft also bought the contracts of 1
million SideKick owners (which now comprise 3-4% of M$'s phone
customers), in contrast to owners of WinMo phones part of every
SideKick owner's monthly service fee is passed from T-Mobile to
Danger specifically to pay for the hosting service, which has been
down since October 1, that's a 2 week outage, imagine if Google had
an outage that lasted so long and affected nearly a million paying
customers under contract. The press would be eating Google alive,
and its stock price would take a hit. For Microsoft to blame "Danger
software" is a dodge, the issue is the data got munged and they didn't
have a good backup, any IT guy will tell you that you always keep
multiple up-to-date backups, especially when people are paying you
for uptime, yet somehow the pooch got screwed, and T-Mobile is left
holding the bag. The Danger server farm is so redundant that some
commentators are saying it must have been sabotage, but if it had
been don't you think Microsoft would have said so to protect their
reputation? Because the alternative is that Microsoft didn't dedicate
the resources required to support its paying customers, which in
contrast to the second article you refer to, I think does say a lot about
Microsoft's cloud offerings, and how Microsoft responded to the crisis
says volumes about their attitude toward their customers. Hosting is
not Microsoft's core business, if it were this wouldn't have happened,
and it wouldn't have lasted so long. Hosting is Google's core business,
and they take it so seriously they design their own server
motherboards, write their own hard disk drivers, analyze every hard
disk & RAM failure to better determine when to pull devices out of
service before they fail, and they store every single bit of information
in at least 3 different nonvolatile locations. Even sabotage wouldn't
have killed the data if it had been hosted by Google.

No, this fiasco very seriously threatens Microsoft's cloud reputation, in
addition to its reputation in the phone market. Anybody who says
otherwise is either in Microsoft's pocket, or they should be prevented
from influencing even basic IT decision-making.
0 Votes
+ -
RE: Sidekick outage says more about the future of 'Pink' than Microsoft's cloud
jackson1984-24316069205748857739440257893812 10th Oct
I used to be astonishingly nfljersey blissful to track down this web site.I wished to thanks for this outstanding endure!! I positively using benefit of every and just about every smallish little bit of it and I've you bookmarked to take a look at out new things you publish.
0 Votes
+ -
RE: Sidekick outage says more about the future of 'Pink' than Microsoft's cloud
makrekdw45-24353611923087901074498125825970 Updated - 11th Nov
Heya, your web page on the web is for the most part spectacular, I had any another person 49ers jerseys challenge san francisco 49ers jerseys requirements happy Feel 49er jerseys you utilize Joomla or simply a a lot more script, do you really guidelines to me ?

Join the conversation!

Formatting +
BB Codes - Note: HTML is not supported in forums
  • [b] Bold [/b]
  • [i] Italic [/i]
  • [u] Underline [/u]
  • [s] Strikethrough [/s]
  • [q] "Quote" [/q]
  • [ol][*] 1. Ordered List [/ol]
  • [ul][*] · Unordered List [/ul]
  • [pre] Preformat [/pre]
  • [quote] "Blockquote" [/quote]
ie8 fix

The best of ZDNet, delivered

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Facebook Activity

White Papers, Webcasts, & Resources
ie8 fix