Gartner fails miserably in product test, immediate action required

UPDATE 1: About six hours after this post, Andrew Frank commented (see below) and also clarified on his blog in a post This is Not a Product Test. Thank you Andrew.

UPDATE 2: The day after this post, Andrew Frank posted The Watchdogs List which aggregated the responses from the two Gartner blogs in order that they were received. This is a welcome and appropriate followup to my blog post below. Once again. thank you Andrew.

++++++++++++

ORIGINAL POST:
Andrew Frank, a Research VP at Gartner "the world’s leading information technology research and advisory company" conducted an interesting experiment in a blog post on October 2 called Which Social Media Monitors Eat Their Own Dog Food?

Unfortunately the experiment went terribly awry and Gartner did nothing about it. I find this behavior by a research firm appalling and misleading and completely counter to the sorts of things that they advise clients. In fact, I think given the post in question, Gartner should be required to issue a formal apology.

Frank issued a challenge to firms that monitor social media by including the names of some companies in the space in a blog post. The test was to see who noticed and you told Frank by leaving a comment.

The companies included 1st2c, Biz360, BrandIntel, BuzzLogic, Nielsen Buzzmetrics, CIC, Clarabridge, Collective Intellect, Converseon, CoreX Technologies, Crawdad Technologies, CSC NameProtect, CustomScoop, TNS Cymfony, Echo Research, Envisional, Factiva, Kaava, Market Sentinel, MotiveQuest, Networked Insights, New Media Strategies, Onalytica, Opinmind, Popularmedia, Radian6 Technologies, RelevantNoise, ScoutLabs, SentiMetrix, Techrigy, Trackur, Umbria, Unbound Technologies, Visible Technologies, Waggener Edstrom Narrative Network

So far so good. It’s sort of clever actually.

However the blog comments that were a part of the Gartner blog system made this a terribly flawed experiment. Several people’s comments did not appear and they had to resend. One person commented within 24 hours and that comment never appeared. It is quite likely that some people commented and their comment never appeared at all.

Here are some of the comments that led me to write this post:

Marcel LeBrun, CEO of Radian6 said: "Hi Andrew, We are listening. Didn't see my earlier comment appear on your post so trying again."

Andy Beal of Trackur.com said: "Unfortunately, I see all and hear all. Nice test! **Either you need a better confirmation system, or my last comment vanished. Feel free to delete this one if needed. **"

Alecia O'Brien of dna13 said: "We are in fact listening (and have been..). We posted a comment several days ago but I guess comments are being moderated?"

At the New Marketing Summit #NMS08 yesterday, I had an opportunity to speak with several of the companies in this experiment. Many were, excuse my language, ROYALLY PISSED OFF.

One person said that they reached out to Gartner through the blog about the issue and got an email from Tom Kobak, Communications Director of Gartner's Product Platforms Group who said: "Thanks for your email to Site Feedback about your comments on the Social Media Monitoring on one of the Gartner blogs. I have passed on to the appropriate person here to take care of, so should be fixed shortly. Sorry for any problems, and please let me know if you have any questions." Interestingly, Kobak's email signature line included this: "Gartner delivers the technology-related insight necessary for our clients to make the right decisions, every day."

Okay, so here is why this is an appalling situation that Garner must make right:

1. Gartner is an analyst firm. Clients trust them to make decisions on technology. In this case, clients may make actual purchase decisions based on this flawed post.
2. In this post, Gartner was testing media monitoring company speed of response to a post. They were TESTING PRODUCTS on the blog but the test was ridiculously flawed. (The test implies that when you are quick to respond, you are better, yet some people were unable to respond).
3. The flaw was pointed out yet Gartner did nothing about it.
4. The vendors in question would probably be too timid to call Gartner out on this for fear of retribution in the form of bad product reviews.

Here's another test. How long will it take Gartner to respond to this?

Comments

[Ron Miller]

Nice post, David. I've had this happen on other sites as well. On one blog recently, they trashed my blog post. I responded twice, with humor and a defense of my post. Neither comment ever posted.

You're right that this a lame social experiment and even more so without a decent comment system. Gartner should know better. They may get a response to their experiment they hadn't considered when they conceived it.

You picked up on it and now it's going to be fully exposed. I'm sure that's not what they had in mind.

Ron Miller
By Ron Miller
http://byronmiller.typepad.com

[Lisa McGrath]

And even funnier ... the Gartner Social Media blog is now down.

[Andy Beal]

David, it gets worse!

Take a look at this little experiment I ran: http://blog.trackur.com/2008/10/05/trackur-passes-the-test/

So, Gartner plays its little game with us monitoring firms, yet doesn't monitor its own name?

Very funny!

[Michael D. Wentworth]

David you've revealed one of the new risks of transparency. We're all now on such a public stage with everything we do...any failure, any mistake, and it can become catastrophic in seconds.

Of course there is likely another common corporate problem at work here. Someone had a great idea and there was little support for execution.

This used to be the invisible dysfunction of corporations, but now it can become a PR nightmare in seconds.

How does a large company address this kind of a problem when it is so much a part of corporate culture?

[Adi]

It doesn't help that when they were emailed about the problem they appear to reply with a canned email, not a personal response.

[Jen Zingsheim]

When it appeared as though my comment was consumed/lost/disappeared, I posted another--when the same thing appeared to happen to the second, I posted a "Yes, we're listening" comment on the Media Bullseye blog.

No comments on that post...

Jen

[Dan Kidd]

David,

This was a very common problem with replies to the original post even for those who did not complain. Gartner should have, in the interest of transparency, highlighted this deficiency in their "test".
Dan Kidd
Biz360

[Edw3rd]

Sorry, the Tech Analyst community is mostly Pay-to-Play and Play-to-Pay. Far as I'm concerned, this was a lame attempt to goad vendors into buying a Gartner "strategic advisory session".

[David Rabjohns, Founder, MotiveQuest]

Thanks for highlighting this. My comment also disappeared. I agree I think an appology is in order.

[Rob Key]

We at Converseon are listening (again). While it might have been an imperfect test, i'm glad that Gartner is paying attention (and resources) to this category.

[David Rabjohns, Founder, MotiveQuest]

Teachers pet :)

[John Wall]

I've been saying for over a year that Forrester is beating the hell out of Gartner on this front.

[Peter I.]

Gartner is still a world class analyst firm but they are certainly weak on social media and Web 2.0. If you look at their Hype Cycles on related topics, they often miss basic definitions of key technologies. I'd also challenge you to name one of the analysts that is vocal and visible in this industry. Forrester, with analysts like Jeremiah Owyang, are head and shoulders above them on these kinds of topics. I hear that they're intent on changing this though.

[Marcel LeBrun]

Part of the problem is that they published the same post on two different blogs. Andrew published it on his own blog and then it also was published on the Gartner Media Industry blog about 30 minutes later (if I recall correctly). Some companies posted comments on one blog and not the other, some on both. The Media Industry blog comments appeared to have been moderated, but no indication was given to the commenter that they were moderated - that is why people re-posted their comments (myself included) as it looked like the comment didn't work. Andrew's blog, however, worked fine. I suspect that Andrew's post might have even be automatically cross posted as Gartner does seem to do this, and so perhaps no one was paying close attention to the Gartner blog for incoming comments - given the nature of the post.

I don't think any harm came from it and it seems to me that Andrew was just making it a general test (and not part of his research) as per his question, "do they eat their own dog food". He later posted on the subject saying that it appears they do.

Marcel
Radian6

[David Meerman Scott]

It is six hours since my post with 13 people commenting but nothing from Gartner (so far).

Thanks all for your interest in this post.

David

[Juliann Grant]

Gartner is not always connected with reality, just their reality and how they paint it. Now if they were in the position to make a comment on a vendor who failed in execution of a test like this vs. themselves, they'd have alot more to say.

[Alecia O'Brien]

Thanks David for the mention. I was glad to see my second comment about dna13 monitoring the situation did get posted.
The 'Watchdog is (in fact) alert'!
Cheers, Alecia

[Andrew Frank]

Here I am, and here's my response: http://blogs.gartner.com/andrew_frank/2008/10/16/this-is-not-a-product-test/

As I note in the post, Marcel got it right by identifying that the test was poorly timed to coincide with a blog transition during which we were operating two blogs at the same time that caused the comments to be split. I apologize to all who found this irritating.

BTW, David: nice use of the call-out technique.

Andrew

[Valerie Combs]

Agree that the issue of two separate blogs/platforms should have been called out - as well as proactive, public acknowledgement of the folks making comments who didn't appear. That said, we all make mistakes. This is a good reminder to all that initiating a conversation is a responsibility that requires active follow through and transparency from all participants.

Valerie Combs
BuzzLogic

[Jeremiah Owyang]

John Wall, Peter I

Thanks, I'm really glad to have the chance to help our clients and follow my passion for social media.

Gartner has smart folks, and they're moving very quickly in this space, I've been watching --and linking-- to their work.

http://www.web-strategist.com/blog/index.php?s=gartner&sbutt=Go

[Tom O'Brien]

Nice call out - and it isn't a technique - just a human reaction.

We (MotiveQuest) are listening - and responded to the Gartner posts - and re-posts. My comment to the re-post never went up. Did Gartner not like it or are they not paying attention? Who knows.

Note to self: when you issue a challenge and start a conversation, be ready to stay in it and finish it.

Tom O'Brien
MotiveQuest LLC

[Blake Cahill]

Others have jumped in but glad to see that the two blog site issue has been highlighted as this was the main cause of confusion for many of the firms that were posting comments. Pleased to see Andrew speaking out about it(look even he double posted here!) I have a lot of respect for his work and others at Gartner as well as many at Forrester and Aberdeen.

Blake Cahill
Visible Technologies

[Josh Carr Superstar]

It seems that mentioning all of the buzz monitoring firms in a blog post is a fantastic way to get a whole bunch of comments - seeing your own brand name pop up is like pooring blood in the water.

as far as Gartner goes I didn't feel like this tests was serious research but your point about how some people will still make buying decisions based on it is a good one.

I would love to see some kind of benchmark testing to compare the different services.

I said this on the Gartner blog and I will say it here too. Buzz.io is so new we were just excited that we made it on their radar.

[Scott Hepburn]

I tweeted about your post today David, and sure enough, @Gartner followed me on Twitter.

Anyone else notice this?

[Scott Hepburn]

Disregard my last comment. She updated her Twitter profile. Turns out its a graphic design student.

So...Gartner still not monitoring then?