All Polls and Surveys Are Not Equal

Linkedin Google+ Facebook Twitter StumbleUpon Email

linkedin polls
Image by renaissancechambara

In Washington, polls and surveys drive policy decisions, particularly around campaign season. For a presidential election, Gallup polls are considered accurate within four points, and this has yet to be proven wrong. However, several online polls and surveys last week did produce highly questionable results, and in once case, was outed as a hoax tarnishing the Microsoft Explorer brand.

This degradation in quality is indicative of a larger trend on the social web, the erosion of expertise (and professionalism) caused by social media content. Launching a poll or a test on a web site is so easy now that anyone can claim to execute research. Indeed, they are. The quality and value of their data is another story. Mind you, this erosion has not only impacted the new media content producer, but also the traditional journalism field as both our Microsoft and Google+ examples will show.

Interactive firm AptiQuant ran a test on its site purportedly measuring the IQ of visitors and correlating that data with IQ. Explorer users were deemed least intelligent.

Unfortunately for Microsoft, the browser IQ test was widely reported by the media, which did not verify the data. Finally, the BBC determined the research was a hoax, but not before the media had popularized Explorer as a low IQ tool. AptiQuant is defending its study, and says it will battle any lawsuits.

But does it matter? The damage has been done to an already lagging brand. Publications that may print retractions won’t push them to the top of their sites with the same zeal they did in their original reporting. A successful lawsuit would only provide a consolation prize for being a called stupid Internet Explorer user.

Google+ Polls

Several polls came out surveying Google+ users about abandoning Facebook for the new circle based social network. Of all the polls only the Christian Post labeled their effort as an unofficial poll, and their numbers were the lowest with 7% moving solely to Google+.

The Brian Solis, Mashable and PC Magazine posts ranged from 23% to 50%. However, all of their readers are extremely tech or social media centric, in essence polling the early adopters. They do not represent the general population, and as such their polls can be pretty much dismissed as industry and demographic specific.

The average reader of these stories would not be able to discern that three tech/social media polls are in essence, “inside baseball.” Mashable did add a little conjecture: “Users may be reacting to the novelty of a new social network. Facebook.”

What is most notable about these four polls is the 40+ point spread between them in response. In the case of the three social media and tech polls, there was still a 27 point spread. Such wild variations should be a clear indicator that the data is inaccurate or compromised in some way.

Keep in mind on line polls — particularly those on social media — often suffer from fan based flash mobbing towards a favored outcome. Also, given the subject matter a survey of the non-indoctrinated general public’s opinion about Google+ would have offered an interesting context to the data.

Conclusion

Without stronger open methodology and wider population samplings, polls cannot be considered representative of likely trends. Polls that deserve respect like Gallup and Pew Internet research are painstaking about their methodology.

In the information age are readers and the media — as the Internet Explorer hoax revealed — savvy enough to discern quality information? Yet another series of examples why we need to teach children and adults alike how to mindfully accept information, and question sources.

What do you think of the polling trend?

Linkedin Google+ Facebook Twitter StumbleUpon Email
  • Kevin Dugan

    Got Sample? It’s my first thought every time I see a story citing percentages in the headline. Constantin Basturea helped me step up my skepticism in this department. You’re right that we need to educate others to question the source.

    • http://twitter.com/geoffliving Geoff Livingston

      Constantin. A name I have not heard in a while. I hope he is doing well. Have you heard from?

  • http://twitter.com/swhitley Shannon Whitley

    How many
    other “facts” are passed along without any research? It’s very
    disturbing when you see how little it matters. The retractions for the IE hoax
    were printed, but there weren’t any repercussions.

    I read the report
    for the IE story, and it was clearly not a professional paper (let alone a
    hoax). I can only hope that it was the subject matter that allowed them to pay
    so little attention to the source.

     

  • http://twitter.com/swhitley Shannon Whitley

    How many
    other “facts” are passed along without any research? It’s very
    disturbing when you see how little it matters. The retractions for the IE hoax
    were printed, but there weren’t any repercussions.

    I read the report
    for the IE story, and it was clearly not a professional paper (let alone a
    hoax). I can only hope that it was the subject matter that allowed them to pay
    so little attention to the source.

     

    • http://twitter.com/geoffliving Geoff Livingston

      I felt bad for Microsoft. They are already down in this category, and this was just like a childish taunt that turned into a pile-on. 

  • http://soloprpro.com KellyeCrane

    This is a huge peeve of mine! Virtually all of the “data” we see on social media has an enormous self-selection bias. I’m hoping to get my professorial spouse on video to explain some of these things to watch for — your call for better education gives me new motivation to do this. Of course, that won’t help with those who don’t *want* to be critical in their analysis (a situation that happens far too often).

    • http://twitter.com/geoffliving Geoff Livingston

      I am having a huge issue with this and Google+ right now. Making it into a panacea with really nebulous data…  It is certainly a powerful new medium for content marketers, but I feel like 50% of this is because people haven’t had new things to talk about for so long.

  • http://www.realityburst.com Eugene Farber

    There’s lies. There’s damn lies. And then there’s statistics.

    • http://twitter.com/geoffliving Geoff Livingston

      LOL, damn it!  Where are the taxes?

    • http://www.lookuppage.com/users/stephenscott/ Janet Ann

      I find it hard to believe polls and surveys … there just numbers and who benefits from all this data anyway?

      • Anonymous

        Usually, the issuer!

  • http://twitter.com/cvharquail cv harquail

    G,
    This is a pet peeve of mine too.

    It’s important to be clear, though: Bad science and careless, meaningless polling cannot be fixed by *sample size*. In no way does a larger sample lead to a more valid result, unless that sample is randomized.  Or, until that sample size approaches the size of the populations you’re interested in…

    We can put aside concerns about kurtosis and heteroskedasticity, even about statistical power, as long as we have a random sample. And yet, online, we never do.

    By definition, no sample that people ‘opt in’ to online is ever random. Hence, it’s pretty much a fact that _no_ online poll is _ever_ ‘scientific’ or statistically valid.  That’s the plain unvarnished truth.

    Ergo, well…. QED.

    I wrote more about online surveys and “crimes against science’ here:
    Are Online Surveys Making Us Stupid?
     
    cvh, PhD (in science)

    • Anonymous

      LOL, there you have it folks, from a real scientist!

  • Pingback: #TheDailySpin: Riots in London halt new Levi’s ad | PRbuilder.com

  • http://equinejointsupplements.blogspot.com Ricardo

    WOW! You make me understand. You helped me so much.
    Thanks for sharing tips.
    It made me understand something, and it is that I never knew before.

  • Anonymous

    i dont trust information result from polls and surveys. It may be the pulse of majority but its not worth in the sense that who signed in? maybe there are some groups who posted it on social media and suddenly gain votes because of that advertisements.

  • http://www.a1qualityarticles.com/computers/lenovo-g560-series-067999u-best-laptop-under-500/ Kimberly

    I do agree in some comments here that polls and surveys are not that reliable. But the purpose of it is just to give some insights of the people being surveyed. Its not really the basis of everything.

  • http://www.i95dev.com/ecommerce Ecommerce Solutions

    I foresee the problem with the participants of the poll. They would not have had used the product/service but they would have had participated in the poll. How can we trust these kind of polls??

A New Novel from Geoff Livingston

Want news and an advanced copy of the book?
Your info is never shared

Work with My Company

Tenacity5 Media Logo

Buy Exodus for Just $0.99

You can buy the book from these vendors, including $0.99 on the Kindle store.

101 Things, a Bucket List

Pacific Sunset

Posts on Other Blogs

Vocus Marketing Blog
(2012-present)

Inspiring Generosity
(2011-2012)

Mashable
(2009-2011)

The Buzz Bin
(2006-2010)

Archives

Categories

My Photos

Devonshire Cow by Geoff Livingston
Devonshire Cow Painted by Geoff Livingston
Tonight's Moon by Geoff Livingston
Tonight's Moon by Geoff Livingston
Massive Cherry Blossom Tree by Geoff Livingston
Jason Jue at the Tidal Basin by Geoff Livingston
Diana Rodriguez at the Tidal Basin by Geoff Livingston
Put Your Hands in the Air by Geoff Livingston
Jump Through the Air by Geoff Livingston
Sea of Green - Capitol Building Centered by Geoff Livingston
Just Dance by Geoff Livingston