December 2012 Archives

Survey Critique


In a recent email, Professor Ball invited us to critique a survey put together by a fellow J-school student. I figured I would take this opportunity as a way to demonstrate how developed my research skills have become over the course of this semester.

The survey was very short and consisted of nine open-ended questions. The responses will be hard to code and analyze on a mass scale, but that's OK as she wants our opinions to help guide in developing a questionnaire for a future study. All of the questions referenced bottled water, and it is clear the student is trying to gauge bottled water use and perceptions of its use among college students.

It leaves me wondering what her research questions and hypotheses are. Obviously that is privileged information as participants only get a snippet of that kind of information when they get briefed before taking surveys. Yet I am very interested in what they would be as I am helping to formulate an ad campaign about bottled water in another class.

That ad campaign is actually going to expose some disturbing realities behind bottled water. For example, one fact we uncovered is that 40% of bottled water is actually taken from municipal water sources also known as "tap water," and Bottled Water can be distributed even if it doesn't meet the quality standards of tap water.

I wonder if this student should expose respondents to some of these facts and have them re-take the survey? I guess that would depend on his/her research objectives. I would do exactly that, however, if I wanted to measure the effects of my group's ad campaign.



The Colbert ReportMon - Thurs 11:30pm / 10:30c
The Word - Truthiness
Colbert Report Full EpisodesPolitical Humor & Satire BlogVideo Archive

Most of us here in the Journalism School are very familiar with Stephen Colbert. He and counterpart, Jon Stewart, have essentially created a brand new way of bringing the news to the world. From a strategic communicator's standpoint, they've hit the jackpot as they've gotten people to listen. So it's only appropriate that I include a blog partially devoted to at least one of them.

Years ago, Colbert invented a new world called "truthiness." He described it as truth that comes from the gut, rather than a book. The word turned some heads in academia, and ultimately found its way into the dictionary. Webster now defines it as, "the quality of preferring concepts or facts one wishes to be true, rather than concepts or facts known to be true." Unfortunately, for many that means irrationally denying cold, hard researched facts in favor of their own intuition or gut feelings.

As future professionals, who will be relying on research to influence and make multi-million dollar decisions, we must be very weary of our own feelings of truthiness. Just because we may believe that the best Unique Selling Position for a car might be the leather seats doesn't mean we should focus an ad on the seats. If research shows that the target audience truly cares about how fast a car can go, then that is what the ad should focus on.

The same can go for a PR message. Just because we believe that a company's mission statement is right doesn't mean that is the mission statement we should present to the whole world. We must present a message that research shows is appealing to a mass audience, one that will present the company as friendly to its customers.

Only in the most obvious of circumstances should we be relying on our intuition. So long as we have the time and money needed to researching the data required to make decisions, then we should be relying on research more than anything.

Primary vs. Secondary Research


This Gallup poll measured how much the average consumer spent per week in the weeks leading up to Thanksgiving, and per day in the three days before it. The data is gathered using Gallup's familiar random-digit dialing, and they reached 2,000 respondents for the first poll and 1,500 for the second.

All the data is, in Gallup's words, "self-reported," and my only question is why did they do primary research for this with a phone survey? Shouldn't they have done some secondary research and measured sales from retailers and other businesses?

Gallup does admit the "data aligns with chain store sales," which does offer some reliability to the poll. However, I feel as though a phone survey just simply is not a very appropriate way of measuring consumer spending.

For starters, I don't think the sample is right. The average American doesn't pay enough attention to their budgets to accurately answer Gallup's questions. Businesses, on the other hand, pay professionals to track every penny in sales. They could offer exact quotes as to how much consumers have spent in their stores, which could then be averaged and fairly generalized to the overall consumer population. Businesses make a much more valid sample.

Furthermore, the polls actually measured a dip in consumer spending compared to previous years, yet other Gallup polls (included some mentioned in my blogs) have actually measured increasing consumer confidence and outlook on things like jobs and the economy. If consumers feel that spending is OK again, then why are they spending less this year? The data gathered may have been consistent with chain store data, but it is not consistent with other related data.

Lastly, Gallup's summary theorizes that the dip in spending is caused by the rise of Cyber Monday and the upcoming "fiscal cliff" as reasons that consumers didn't spend as much this time around. Yet these are two contradicting explanations. If consumers were truly worried about a "fiscal cliff" would they not spend less on Cyber Monday as well? And if they were saving for Cyber Monday, wouldn't they not be worried about the "fiscal cliff?"

I feel as though this was a poor poll, and the researchers should have been gathering their data from secondary sources to measure consumer spending. It just goes to show that sometimes the answer doesn't lie in your own data, it lies in someone else's.

Tag Cloud