What it means to be Scobleized

This past Friday, before signing off for the weekend, I took Microsoft's Robert Scoble to task for what, in my opinion, was a grossly unjust review of the services provided by Technorati. Robert Scoble is the publisher of the very popular blog Scobleizer and I felt his coverage was unjust for two reasons.  First, he cited the wrong numbers in his conclusions.  Second, he tested Technorati's service for something it doesn't claim to do.   I also made a third point about whether one of the commonly applied rules of tech journalism -- fact checking test results with vendors prior to publication to make sure you're not making a mistake -- should have been applied in this case.  I think it should have. 

Using his blog, Scoble responded with 11 points.  The last two were throw-aways, but here's my response to the first nine.  In my responses, I address Scoble directly.

1. You claim to have missed the right numbers because you didn't scroll to the bottom of the page. The only Technorati page I went to was the one you linked to from your blog.  I got to it by clicking on the link.  The numbers (links, sites) are displayed right near the top of the page as indicated in the graphic on my original blog.  The same data is shown again at the bottom. 

2. You said "I provide the links so you can compare for yourself, something that David didn't do on his blog."  This is not true.  I took the exact same links you provided and hyperlinked to them from my blog.  Regarding your other comments, I think the duplicate issue is a very important issue when it comes to putting any faith in the numbers you're seeing.  Judging by the way there were duplicates on the first results page of Bloglines, but not on Technorati (and that it says at the bottom of the Technorati page how many duplicates were removed),  you can't help but come to the conclusion that the math in both cases is impacted by the duplicate issue.  Lest you think I'm comparing Technorati to Bloglines or passing judgement on Bloglines, I'm not.   It's just clear to me that a clear understanding of what each service's numbers stand for wasn't established before you compared them.  If they stand for different things -- and you never said whether they do or they don't  -- then how can you compare them.  If my apple grove has 50 trees and my orange grove has 60, can you compare the two in any meaningful way?

3. Regarding me holding you to journalistic standards, in my blog, I say it doesn't matter what you call yourself. You've got a running comparison in your blog of Technorati against Bloglines, and you're holding Technorati up to a benchmark that Technorati, by it's own admission, isn't seeking to attain.  Perhaps you should test a Jetta to see if it can drive through a river the way a Hummer can.  Sifry deserved a call before the criticism was levied, not after -- when your original comments are already out in the "remix" and doing their damage (something that after-the-fact corrections can never undo).  As far as saying I'm holding you to journalist standards, you said yourself that sometimes you're a journalist.  So, what were you when you wrote-up the review of Technorati?  Were you wearing your journalist hat, or not?  How would we know?  Do you put a sign on each entry that says "Hey this time I'm a journalist" and "this time I'm not?"  

Regarding your question as to whether I'm Sifry's marketing guy?  No, quite the opposite.   I'm a potential customer of Technorati's who took the time to study what the company offers and who thought the numbers were off.  But before I jumped to conclusions and ruled Technorati out as a potential service provider to ZDNet, I called to ask questions.  You say I called one guy.  Actually, I called two.  You and Sifry.  You didn't answer your phone (and I reported as such).   If you're implying I should have called Bloglines as well, I didn't call Bloglines because my point had nothing to do with whether Technorati is better or worse than Bloglines.   It had to do with results that you inaccurately reported (by your own admission, you reported the wrong numbers) and it also had to do with the way you tested Technorati for something that it admits it doesn't do or even wants to do. Yet you didn't mention that detail. 

The two in combination produced a profound injustice and maybe even some damage that you seem to be unapologetic for.  It sounds as though you're suggesting that the burden should be on the subjects of your write-ups to correct you after you've published your write-up and after any damage is done (as opposed to taking a few extra minutes to fact check your findings).  Shoot first, ask questions later.   I just noticed that Dave Winer advocates this approach as well, saying "Scoble is doing the right thing. They should try to understand how he works, because it produces much better results than the system Gartenberg and Berlind are advocating. Demonstrably much better." 

For the sake of helping readers make their own judgements about credibility, I think it's great that you and Winer are letting everyone know where you stand on trying your best to get the story right the first time (which is not very hard).  This by the way, is the same Dave Winer that complained:

• "I guess it's not surprising that the mainstream press only talks with [Adam Curry], even when they acknowledge that there's another side to his story (as News.Com did)."
• "I guess Adam will keep accelerating the lies, and the pro journos will keep reporting them." 
  
• "Of course I could end up forgetting the story, but as long as he continues to say really nasty shit about me personally I don't think I will."

By the way, that News.com, the one that Winer is showing his appreciation for, is a part of CNET Networks.  The same CNET Networks that ZDNet is a part of. 

4.  You say you think David Sifry is making the wrong business decision.  OK, now we're getting somewhere. If you think his business decision is wrong and if that's what you had written originally, you would have never heard from me.  That's your opinion and it's also a completely fair point.  In my interview of him, Sifry even acknowleged that others may not agree with his approach.  You and others may very well disagree with his business decisions.  How can I or anyone else not respect that opinion?  You may even disagree that authority is something worth measuring, as Technorati tries to do.   But you never provided that context in your coverage or your criticism.  You never said, "Oh, as it turns out, Sifry has all that data I'm looking for and, as a business decision, he's chosen not to display it." That would have set you up to do what you just did in your fourth point -- criticize the business decision. 

As I said in my blog, if you want to study the business decisions that Technorati, Bloglines, or anybody else in that business is making, and then be critical of those, I think that's a great debate to have.  Should an attempt to measure authority be made and if so, what's the best way to do it? Or, would some sort of end-all be-all raw search results be be better?  Whereas you disagree with the business decision,  I for one, stated that I agree with it and explained why.   I'll be the first one to admit that my opinion doesn't mean I'm right.  As for the "other bloggers" that you talked to, I think the blogosphere is a great place to hash out the answers to these questions.  Perhaps you should point to where they said so.  But, before taking a survey, I think it's fair that they get a full explanation of why Sifry has chosen to do what he does.  They might change their minds as I did.   By the way, one option Sifry and I discussed, since he said he has all that data in his database, is giving Technorati users the option of seeing it.  Then, you'd have the best of both worlds.  Some like their coffee black.  Some want cream and sugar.

5.  Did I also fail as a journalist for not calling Bloglines too?  The reason I didn't call Bloglines was because I wasn't looking to compare Bloglines to Technorati.  As I said in my original blog, "Scoble's coverage could have been much more interesting had he uncovered some differences in methdology and started a debate over how best to measure authority."  I was pointing out that if you really wanted to get to the heart of the matter -- the differences between the two -- not just in the numbers be displayed on their Web pages, but in the businesses they want to be in, then that would have made for a very interesting story and/or debate.  My blog was not about trying to get to the heart of that matter.  My blog was about explaining why you saw what you saw on Technorati.com and pointing out that there is a perfectly logical explanation for it: David Sifry's business decision.   Not a fix, as you're blog's headline (Technorati is working to respond to complaints, Sifry says) implies.  Based on my interview of Sifry, this is not a complaint or problem that Sifry is looking to fix any more than Volkswagen is looking to somehow get its Jetta's to drive through rivers.

6. Again, you're saying Bloglines does a better job overall.  In the context of what? What Technorati says it doesn't do?  Hummers can drive through rivers.  Jettas, probably not.  They both hold passengers and have four tires.  I guess it's fair to compare them on a equal playing field (never mind that their manufacturers seem them as satisfying different needs).   As far as me doing an objective analysis of the two (Bloglines and Technorati), I'm not sure what you're implying.  That if I tested it with your methdology, that I'd come to the same conclusion as you?  Of course I would, as would anybody.   Or, are you suggesting that it would be difficult for me to come up with something more objective?  Anybody who knows me knows this to be untrue since a significant part of my career as a tech journalist was spent in a testing lab, designing methodologies that were not only objective, but that carefully took the promise of the products we were testing into careful consideration before writing up the results.

Should an objective anlaysis be done? Probably (Doc Searls thinks one will come out of it).  Should I be the one to do it?  Maybe, in the course of trying to figure out who can best service ZDNet's information needs.   But it's really not my beat.  I spoke up in this case because I felt that your criticisms were a gross injustice given that you were quoting the wrong numbers in your report, and, n comparing those incorrect numbers to someone else's numbers, you were testing Technorati for something it never claimed to do.

7.  No, you're right.  I didn't call the vaulted "three sources."  The goal of my coverage was not to compare Technorati to Bloglines, Feedster, or any other service.  It was to point out that you were asking something of Technorati that Technorati itself says it doesn't do.  So, of course it's going to fail your test.  How could it not?  Just the same way a Jetta will never drive through a stream.  So, you tell me.  How many sources beyond the CEO of Technorati do I need to tell me what it is Technorati is designed to do? 

8. "Can they leave comments on David's post?" I fail to see the point.  It appears to be a rhetorical question as though people can't leave comments on our blogs.  The answer is, you can.  You can leave posts on just about anything that's written on ZDNet and thousands of people do.  In addition to commenting on our blogs, you can also TrackBack to our blogs. 

Perhaps you're referring to the fact that we require registration on our systems before you can leave a comment (trackback requires no registration, by the way).   This serves several purposes.  First, it keeps comment spam down to an incredibly manageable level because the spam bots aren't programmed to register and login to our systems.  [Update: A-List blogger Steve Rubel (Micro Persuasion) says he just started requiring registration for the same reason].  Second, once you're  registered on our systems, no one can impersonate you.  To comment, you have to be logged in and only you can login under your ID (as long as you haven't given out your ID and password). 

ZDNet also offers other services.  For example, if you're interested, you can get one of the newsletters edited by me and Dan Farber. In the context of newsletters, our registration system equates to a double-opt-in with the ability to always come back and manage your preferences.  For example, no one else can sign your inbox up to receive one of our e-mail newsletters without that same inbox getting a confirmation notice that you must click on in order for the subscription to be processed.  Then, at any point, you can always log back into ZDNet's systems, and deselect that newsletter (to stop getting it), select others, or change other preferences (for example, whether you want to receive special offers from hardware and software companies).   What you get in your inbox is completely under your control and someone else cannot register your inbox on our systems in such a way that your inbox starts receiving mail from ZDNet that you never asked for (spam).  We take a very high road on this front.  More importantly, your identity and presence for all that you do on ZDNet, whether it's managing your newsletter preferences or using the Talkback system  that spans everything from our news stories to our columns to our blogs (that's right... our comment system, known as Talkback, is the same system no matter where you are on ZDNet), is one single sign-on identity. 

9.  You're right Robert.  I never complained all those times that you wrote something nice about Technorati.  That's because I didn't see them.  I scan, looking for things that are of interest to me and Technorati wasn't of much interest to me until very lately.  But, had I seen you write something that was glowlingly postive that I also knew to be untrue, I probably would have written to you as well.  So, in this situation, I came across one post, and it was unjust based on what I know of the company and what it offers as well as on the numbers you so innaccurately cited.   And there's no number of good reviews you could have done of Technorati to justify that sort of injustice. 

Finally, I hope nobody takes what I've said to mean that Technorati is perfect in my eyes.  It's not and the company's CEO David Sifry will be the first to tell you so and why.  [Updated 30 minutes after original post].  Finally finally,  Bloglines may very well be better than Technorati.  Or Feedster.  Or whatever.  The most important point is that none of what Robert Scoble has published so far can or should be treated as serious evidence of that.