« EMC Corporation and social media marketing | Main | Comfortable air travel or the lowest possible price? »

October 16, 2008


TrackBack URL for this entry:

Listed below are links to weblogs that reference Gartner fails miserably in product test, immediate action required:


Feed You can follow this conversation by subscribing to the comment feed for this post.

Ron Miller

Nice post, David. I've had this happen on other sites as well. On one blog recently, they trashed my blog post. I responded twice, with humor and a defense of my post. Neither comment ever posted.

You're right that this a lame social experiment and even more so without a decent comment system. Gartner should know better. They may get a response to their experiment they hadn't considered when they conceived it.

You picked up on it and now it's going to be fully exposed. I'm sure that's not what they had in mind.

Ron Miller
By Ron Miller

Lisa McGrath

And even funnier ... the Gartner Social Media blog is now down.

Andy Beal

David, it gets worse!

Take a look at this little experiment I ran: http://blog.trackur.com/2008/10/05/trackur-passes-the-test/

So, Gartner plays its little game with us monitoring firms, yet doesn't monitor its own name?

Very funny!

Michael D. Wentworth

David you've revealed one of the new risks of transparency. We're all now on such a public stage with everything we do...any failure, any mistake, and it can become catastrophic in seconds.

Of course there is likely another common corporate problem at work here. Someone had a great idea and there was little support for execution.

This used to be the invisible dysfunction of corporations, but now it can become a PR nightmare in seconds.

How does a large company address this kind of a problem when it is so much a part of corporate culture?


It doesn't help that when they were emailed about the problem they appear to reply with a canned email, not a personal response.

Jen Zingsheim

When it appeared as though my comment was consumed/lost/disappeared, I posted another--when the same thing appeared to happen to the second, I posted a "Yes, we're listening" comment on the Media Bullseye blog.

No comments on that post...


Dan Kidd


This was a very common problem with replies to the original post even for those who did not complain. Gartner should have, in the interest of transparency, highlighted this deficiency in their "test".
Dan Kidd


Sorry, the Tech Analyst community is mostly Pay-to-Play and Play-to-Pay. Far as I'm concerned, this was a lame attempt to goad vendors into buying a Gartner "strategic advisory session".

David Rabjohns, Founder, MotiveQuest

Thanks for highlighting this. My comment also disappeared. I agree I think an appology is in order.

Rob Key

We at Converseon are listening (again). While it might have been an imperfect test, i'm glad that Gartner is paying attention (and resources) to this category.

John Wall

I've been saying for over a year that Forrester is beating the hell out of Gartner on this front.

Peter I.

Gartner is still a world class analyst firm but they are certainly weak on social media and Web 2.0. If you look at their Hype Cycles on related topics, they often miss basic definitions of key technologies. I'd also challenge you to name one of the analysts that is vocal and visible in this industry. Forrester, with analysts like Jeremiah Owyang, are head and shoulders above them on these kinds of topics. I hear that they're intent on changing this though.

Marcel LeBrun

Part of the problem is that they published the same post on two different blogs. Andrew published it on his own blog and then it also was published on the Gartner Media Industry blog about 30 minutes later (if I recall correctly). Some companies posted comments on one blog and not the other, some on both. The Media Industry blog comments appeared to have been moderated, but no indication was given to the commenter that they were moderated - that is why people re-posted their comments (myself included) as it looked like the comment didn't work. Andrew's blog, however, worked fine. I suspect that Andrew's post might have even be automatically cross posted as Gartner does seem to do this, and so perhaps no one was paying close attention to the Gartner blog for incoming comments - given the nature of the post.

I don't think any harm came from it and it seems to me that Andrew was just making it a general test (and not part of his research) as per his question, "do they eat their own dog food". He later posted on the subject saying that it appears they do.


David Meerman Scott

It is six hours since my post with 13 people commenting but nothing from Gartner (so far).

Thanks all for your interest in this post.


Juliann Grant

Gartner is not always connected with reality, just their reality and how they paint it. Now if they were in the position to make a comment on a vendor who failed in execution of a test like this vs. themselves, they'd have alot more to say.

Alecia O'Brien

Thanks David for the mention. I was glad to see my second comment about dna13 monitoring the situation did get posted.
The 'Watchdog is (in fact) alert'!
Cheers, Alecia

Andrew Frank

Here I am, and here's my response: http://blogs.gartner.com/andrew_frank/2008/10/16/this-is-not-a-product-test/

As I note in the post, Marcel got it right by identifying that the test was poorly timed to coincide with a blog transition during which we were operating two blogs at the same time that caused the comments to be split. I apologize to all who found this irritating.

BTW, David: nice use of the call-out technique.


Valerie Combs

Agree that the issue of two separate blogs/platforms should have been called out - as well as proactive, public acknowledgement of the folks making comments who didn't appear. That said, we all make mistakes. This is a good reminder to all that initiating a conversation is a responsibility that requires active follow through and transparency from all participants.

Valerie Combs

Jeremiah Owyang

John Wall, Peter I

Thanks, I'm really glad to have the chance to help our clients and follow my passion for social media.

Gartner has smart folks, and they're moving very quickly in this space, I've been watching --and linking-- to their work.


Tom O'Brien

Nice call out - and it isn't a technique - just a human reaction.

We (MotiveQuest) are listening - and responded to the Gartner posts - and re-posts. My comment to the re-post never went up. Did Gartner not like it or are they not paying attention? Who knows.

Note to self: when you issue a challenge and start a conversation, be ready to stay in it and finish it.

Tom O'Brien
MotiveQuest LLC

Blake Cahill

Others have jumped in but glad to see that the two blog site issue has been highlighted as this was the main cause of confusion for many of the firms that were posting comments. Pleased to see Andrew speaking out about it(look even he double posted here!) I have a lot of respect for his work and others at Gartner as well as many at Forrester and Aberdeen.

Blake Cahill
Visible Technologies

Josh Carr Superstar

It seems that mentioning all of the buzz monitoring firms in a blog post is a fantastic way to get a whole bunch of comments - seeing your own brand name pop up is like pooring blood in the water.

as far as Gartner goes I didn't feel like this tests was serious research but your point about how some people will still make buying decisions based on it is a good one.

I would love to see some kind of benchmark testing to compare the different services.

I said this on the Gartner blog and I will say it here too. Buzz.io is so new we were just excited that we made it on their radar.

Scott Hepburn

I tweeted about your post today David, and sure enough, @Gartner followed me on Twitter.

Anyone else notice this?

Scott Hepburn

Disregard my last comment. She updated her Twitter profile. Turns out its a graphic design student.

So...Gartner still not monitoring then?

David Meerman Scott

Hi Andrew,

Many thanks for your timely comment and the clarification post on your site. I appreciate it and I'm sure that the companies involved do too.

The reason for the issues with comments makes total sense. These things happen. I hope that the explanation will be accepted by those affected.

As we've all learned from this, it is a good idea to be prepared when running a test like this.

Gartner is very influential and even a casual test on a blog is considered an important form of "research" and should be taken seriously. Some of the people who mentioned it to me yesterday were very concerned.

I think your next post with all of the comments will go a long way to making it all right.

Best, David

Steve Dodd - Sysomos

Well, it's interesting to see everyone's comments and frustrations. We also posted and had to repost. But, I think the important issue here (other than the reliability of the Gartner site) is that it pays to pay attention to activity and make sure that what you think is done, is really done. After being in the IT industry for 25 years, I've learned that automated systems can never be totally trusted and must be audited. This especially holds true in the Social Media space with the vastly different technologies at play......and most importantly when trusting analysis and engaging with influencers.

Bryan Person, LiveWorld

I think Andrew's new blog post and follow-up comment here explain the confusion over some of the comments to the original blog posts, but not all (Jen Zingsheim's, for example).

But what I find most interesting is that neither Andrew nor any other Gartner staffer addressed those initial concerns raised by Jen, Andy Beal, and others in the comments area. Why not jump into the comment stream and say, "Hey, we're having a technical glitch and are working on it," or "you might have commented over there and are now commenting over here."

For a post that wonders aloud whether other companies are doing a good job of social media monitoring, shouldn't Andrew/Gartner have demonstrated it was doing the same in its own comment stream?

I look forward to the follow-up post from Andrew.

Bryan Person | @BryanPerson

David Meerman Scott

Scott, I assume that this Twitter profile really is Gartner -- @Gartner_inc -- Looks real to me. I am being followed and a tweet pointed me and others to Andrew's post. So they are watching on Twitter.


You'll see that Andrew Frank replied to this blog, wrote about this issue on his blog and Gartner alerted folks on Twitter about it (and started to follow some of those involved in the discussion, including David). Gartner on Twitter is @Gartner_inc, not @Gartner (the student!). Thanks.

Martin Edic (Techrigy)

They should try using a third party system like Disqus or Intense Debate for comments across all their blogs (both of which we track...)
All of our comments appeared.


From the perspective of someone not involved, this is a fascinating demonstration of the topics that are discussed here and in the rest of your blogs. Powerful stuff indeed!

John Flynn

Great job David. Keep those big guys honest!


This is really not surprising. For those who have followed the analyst community for a while, some of the best and brightest out there who would NEVER screw up like this are no longer at Gartner or Forrester or AMR or IDC, etc.

I don't agree that this is part of the "pay-for-play" mindset that exists in many quarters - I can't paint the canvas with that brush. I just think you have analysts at these firms who are little naive about their firm's ability to support a challenge via the social networks under the Gartner banner. Gartner has had a long reputation of being smash-mouth in their reports and in analysts briefings since the early 90's, but that culture and style does not work in these forums and some of their analysts just don't understand that. You would would never have seen Erik Keller or Chris Jones or Beth Enslow or several other of the former Gartner stallwarts do something like this.

Whoever posted that Forrester has left Gartner in the dust in the new world order of marketing and social networks was spot on.

Where oh where have the really good analysts gone?????

Andrew Frank

The list of companies which responded to the initial call is here: http://blogs.gartner.com/andrew_frank/2008/10/17/the-watchdogs-list/

Steve Dodd - Sysomos

You know, with all the intense and varied commentary about this topic, it seems to me that the experiment worked exceptionally well, Congrats Andrew! Isn't this what it is all about?

Debbie Weil

Hmmm... normally, David, I think everything you write is spot on. But here I'm not so sure.

You're jumping all over Gartner for a technical glitch and for poor execution. It does make them look a bit inept (with cross posting on two blogs and somehow "losing" some of the comments).

And the whole thing is way too casual if they were serious about drawing any meaningful conclusions about the vendors.

It's impossible to imagine Gartner or Forrester or any firm running an analysis like this *offline* in such a casual manner.

Still the only damage done IMHO is to Gartner. All the monitoring brands mentioned come out just fine in the end because everyone agrees it's a flawed experiment.

What it illustrates is that Gartner is still a newbie in the corporate blogosphere. They're still learning... and they screwed up.

I agree that an apology from them would be appropriate.

Now I can't remember... am I still disagreeing with you?? This is the challenge with blogging. How to be thoughtful and real and publish fast. Without making an a** of yourself. :)

David Meerman Scott

Debbie, I think you are missing the most important point. Until I posted on this, nobody (except a handful of vendors) knew it was flawed.

We're talking Gartner here - an analyst firm - and they were testing with this post.

Imagine a potential client of one of these services finding the Gartner post as a result of a Google search and then seeing that their vendor of choice did not comment. The potential client might draw the conclusion that that vendor's people and/or product wasn't up to snuff.

If the test had been flawless, that's just fine and perfectly valid. But quite a few vendor comments were lost or late. Since the test was flawed, someone could have lost business as a result of the poor execution by Gartner.

If an analyst firm such as Gartner does a test of a blog monitoring service speed at responding to a blog, it is not casual. It is not "just a blog post." It is a real test by a noted analyst covering a space and it must be flawless.



Hii dear Frenz!,...

Thanks, I'm really glad to have the chance to help our clients and follow my passion for social media.

Gartner has smart folks, and they're moving very quickly in this space, I've been watching --and linking-- to their work.

Ioffersearch.com Blogs - Just another Ioffersearch.com weblog

The comments to this entry are closed.


Your email address:

Powered by FeedBlitz

follow me

David Meerman Scott books

I want to speak at your next event!


David Meerman Scott e-books

David's iPhone and iPad apps

Blog powered by Typepad
Member since 12/2004