Yves here. One of my summer jobs in college was conducting survey research, so I have an old-school bias as to the value of this sort of information-gathering and the importance of doing it well. The article below describes how this process can become corrupted, either by accident of design.
By Teresa Carr, a Colorado-based investigative journalist and the author of Undark’s Matters of Fact column. Originally published at Undark
Last December, a joint survey by The Economist and the polling organization YouGov claimed to reveal a striking antisemitic streak among America’s youth. One in five young Americans thinks the Holocaust is a myth, according to the poll. And 28 percent think Jews in America have too much power.
“Our new poll makes alarming reading,” declared The Economist. The results inflamed discourse over the Israel-Hamas war on social media and made international news.
There was one problem: The survey was almost certainly wrong. The Economist/YouGov poll was a so-called opt-in poll, in which pollsters often pay people they’ve recruited online to take surveys. According to a recent analysis from the nonprofit Pew Research Center, such polls are plagued by “bogus respondents” who answer questions disingenuously for fun, or to get through the survey as quickly as possible to earn their reward.
In the case of the antisemitism poll, Pew’s analysis suggested that the Economist/YouGov team’s methods had yielded wildly inflated numbers. In a more rigorous poll posing some of the same questions, Pew found that only 3 percent of young Americans agreed with the statement “the Holocaust is myth.”
These are strange times for survey science. Traditional polling, which relies on responses from a randomly selected group that represents the entire population, remains the gold standard for gauging public opinion, said Stanford political scientist Jon Krosnick. But as it’s become harder to reach people on the phone, response rates have plummeted, and those surveys have grown exponentially more expensive to run. Meanwhile, cheaper, less-accurate online polls have proliferated.
“Unfortunately, the world is seeing much more of the nonscientific methods that are put forth as if they’re scientific,” said Krosnick.
Meanwhile, some pollsters defend those opt-in methods — and say traditional polling has its own serious issues. Random sampling is a great scientific method, agreed Krosnick’s Stanford colleague Douglas Rivers, chief scientist at YouGov. But these days, he said, it suffers from the reality that almost everyone contacted refuses to participate. Pollsters systematically underestimated support for Donald Trump in 2016 and 2020, he pointed out, because they failed to hear from enough of those voters. While lax quality controls for younger respondents, since tightened, led to misleading results on the antisemitism poll, YouGov’s overall track record is good, said Rivers: “We’re competitive with anybody who’s doing election polls.”
Nonetheless, headlines as outrageous as they are implausible continue to proliferate: 7 percent of American adults thinkchocolate milk comes from brown cows; 10 percent of college graduates think Judge Judy is on the Supreme Court; and 4 percent of American adults (about 10 million people) drank or gargled bleach to prevent Covid-19. And although YouGov is one of the more respected opt-in pollsters, some of its findings — one third of young millennials aren’t surethe Earth is round, for example — strain credulity.
Amidst a sea of surveys, it’s hard to distinguish solid findings from those that dissolve under scrutiny. And that confusion, some experts say, reflects deep-seated problems with new methods in the field — developed in response to a modern era in which a representative sample of the public no longer picks up the phone.
The fractious evolution in polling science is likely to receive fresh attention as the 2024 elections heat up, not least because the consequences of failed or misleading surveys can go well beyond social science. Such “survey clickbait” erodes society’s self-esteem, said Duke University political scientist Sunshine Hillygus: It “undermines people’s trust that the American public is capable of self-governance.”
Veteran pollster Gary Langer compares traditional randomized polling methods, known as probability polling, to dipping a ladle into a well-stirred pot of minestrone soup. “We can look in and see some cannellini beans, little escarole, chunks of tomato,” he said. “We get a good representation of what’s in the soup.”
It doesn’t matter if the pot is the size of Yankee Stadium, he said. If the contents are thoroughly mixed, one ladle is enough to determine what’s in it. That’s why probability surveys of 1,000 people can, in theory, represent what the entire country thinks.
The problem is that getting a truly representative sample is virtually impossible, said YouGov’s Douglas Rivers, who pointed out that these days a good response rate to a randomized poll is 2 percent.
Pew expends a great deal of effort to maintain a randomized panel of about 10,000 people willing to take surveys. For the most recent annual recruitment, the organization mailed letters to a random selection of 13,500 residential addresses obtained from the U.S. Postal Service, receiving around 4,000 responses according to Pew researcher Courtney Kennedy. They only invite one-quarter of responders to the panel. Otherwise, Kennedy explained, the panel would be overrun with the types of people most amenable to taking surveys. Eventually, they wound up with 933 new recruits.
Some groups — in particular young people, people of color, and those who didn’t go to college — are generally more reluctant to take surveys, said Kennedy: That’s where they lose the perfect representative. Like every other pollster, she said, Pew adjusts their data, giving more weight to the responses of those underrepresented in the sample, so that the results represent the country in terms of demographics such as age, gender, race, education level, and political affiliation.
But those weighting methods are imperfect. And the people in a poll are still unrepresentative in at least one way: They are the Americans who are willing to answer a pollster’s message. Those difficulties have prompted a quiet revolution in survey research over the past two decades.
In 2000, nearly all pollsters simply called people on the phone, according to a 2023 Pew study of polling methods. But use of calls alone plummeted starting in 2012, while online opt-in surveys like the Economist/YouGov survey, one of the main forms of what are known as nonprobability polls, soared.
Nonprobability surveys don’t stir the pot so that each ingredient has an equal chance of being selected. Instead, they scoop up what’s referred to as a convenience sample of respondents, typically recruited online. Opt-in pollsters differ in how they recruit and select participants, and they are not always transparent about their methods. Once they have assembled a group of participants, pollsters can weight the sample so that it matches the broader U.S. population. But it’s much harder to accurately weight nonprobability polls, since there is less information on how people who opt into polls compare to the public at large.
“Probability sampling tends to yield more representative samples that nonprobability approaches,” Kennedy wrote in an email.
However, nonprobability surveys are typically much cheaper than probability polls. As Americans have ditched their landlines and stopped answering their cell to unknown callers, contacting people takes far more time and effort than it used to. As a result, according to Duke University political scientist Sunshine Hillygus, while it can cost as little as $1 per response to run a short online opt-in poll, it can cost 50 to 500 times that for a high-quality random-sample survey.
To create a pool of people to take opt-in surveys, polling companies recruit through ads that pop up on social media, internet search engines, and even during video games, offering cash or rewards to complete surveys, said Kennedy. YouGov, for example, pays people in points — 500 to 1,000, for example, to take a short survey. At 25,000 points, you can cash in for a $15 gift card; 55,000 points earns $50 in cash.
Pew and other pollsters who do randomized polling also pay people a small amount to take the occasional survey. But with opt-in polling, survey taking can become a full- or part-time job for many people. The job search website Indeed, for example, lists companies that pay for surveys in its career guide. And in the Reddit community Beer Money, which has 1.3 million members, people frequently discuss the pros (time flexibility) and cons (skimpy pay; frequently getting screened out) of taking surveys for money.
Some of those surveys are for academic research. (Many psychology papers, for example, rely on paid respondents recruited through platforms like Amazon Mechanical Turk.) Others help companies with market research — or feed the insatiable media market for polls.
My main objection to surveys is anonymity. You are being asked to reveal your political leanings by a complete stranger who will then tell everyone else about your responses. There is, additionally, no way to verify these polls are being conducted by these organizations, and no way to confirm that they will actually protect your information.
Or report your responses without some, uh, tweaking for the correct magnitude and direction.
P-hacking, curve fitting, cheating.
:(
> while it can cost as little as $1 per response to run a short online opt-in poll, it can cost 50 to 500 times that for a high-quality random-sample survey.
A platter of sugar sandwiches is more nutritious than a five-course meal if your measure is weight gain. The high-n approach is adjacent to mass production, industrialization by credentialed professionals. Easier than knocking on doors.
Dehumanize the process and you dehumanize the results. Knocking on doors required perseverance, attention to nuance, eye contact. Phone surveys lost eye contact but still had a person asking questions. Forty years ago, at the Center for Survey Research, we’d call back a third time after being refused twice. The industrial approach is taking non-responses as defective parts and tossing them.
I’ll suggest you can still get good data within-groups, where there are common interests providing a motivational frame for accurate answers and a high response rate. Modern electoral polling seems more about divisiveness, like gerrymandering and finding ways to exclude rather than include. Same incentive structure as voter registration drives, why sign someone up when they might go the other way? Better to have a tightly controlled cadre, whether voters or customer base. Or paid respondents.
I’m assuming honest inquiry here, not push-polling. And that AI bots haven’t figured out how to opt-in. But taking climate change as an example, when the outcome of the poll is more about messaging than substance, the incentive structure changes to making sure the rice bowls are filled. As long as the polling results lead to further polling, the results don’t matter; deluded losers wash out while winners have their bias confirmed. IBGYBG, except I’ll still be here. Pity the honest pollster.
Todays links includes Topol interviewing Christopher Labos on subgroup analysis and p-hacking. Worth the read.
> overall the data is negative, but if we slice it up, we can find something that’s positive.
There are multiple websites where anyone can opt-in, like described above.
I tried a couple of polls for curiousity….but not sharing my real demos of course because I don’t want my name added to “some list”.
the marketing co. got 20 min. of my time for answers re. the hotel chains/credit card brands that I am familiar with. I got x,000 points in my account. Alas I need xx,000 points to get a $10 Amazon gift card. womp womp.
On an hourly basis, I imagine one would get paid less than $3/hour for these surveys.
Even with good-faith statistical adjustments, IMO, no way ypu can get a good topline results…..the data is so skewed by a respondent pool willing to trudge through all those questions for $3/hr.
there was a Bloomberg??? article that I read once re. focus groups…poorly run product focus groups produce awful outputs given the self-selection bias of the respondent pool.
The new Jalapeno, mustard cinnamon Oreos did not catch on? blame the focus group process. garbage in, same out.
“The new Jalapeno, mustard cinnamon Oreos did not catch on? blame the focus group process. garbage in, same out.”
That’s a big red flag there. I never suspected the depth of the public antipathy to “Cookies of Colour.” We obviously need more mandatory “Alternate Foodstuffs Sensitivity Training” courses in the schools. (Now, where is that grant proposal I wrote for the “Alternate Foodstuffs Sensitivity Training Course Methodology” got too?)
See you at the Job Fair!
Lol! The new Jalapeno, mustard cinnamon Oreos did not catch on? blame the focus group process. garbage in, same out.
My black heart hates those damn cookies. Unfortunately, I’ve yet to meet a gaggle of Girl Scouts selling them that I can refuse.
Surveys are also full of bots. Not necessarily “I’ll intentioned” ones, but those that are designed to parse the questions (If they are sophisticated enough) or are designed to give genetic answers that might fit a wide variety of questions–that is, if the questions require text input and not multiple choice.
One of former colleagues (I have no idea how he did it) looked into the MO of professional survey takers by getting in touch with some of them. They were pretty frank about how they go about survey taking, including how they use bots. Basically, they aren’t being paid enough to give answers that demand any more than the minimal attention and considering the rates, I can’t blame them. But, by the same token, I don’t have too much faith in what we get out of them.
The more serious problem, though, with political surveys, is how people use weights. We don’t know what the turn out pattern will look like so we have to extrapolate using the responses we have, knowing we have bad samples. But how you go about extrapolating is far more of an art than science, and, as such, a “good” occasion to make stuff up on gut feelings. I can’t disparage this too much since these were exactly those who were willing to jump off the cliff (and this is how Gallup got the 1936 election right and the Literary Digest screwed up)…
Thank you for this, it is a valuable area of inquiry as surveys and polling have become so important. Of interest, perhaps tangential, but I suspect still germane in this season, is the development of political polling, consulting and, effectively, propaganda.
This is a piece in the New Yorker from Harvard historian Jill Lepore.
The Lie Factory
Its first crusade was against Upton Sinclair in his campaign for governor of California (the DC Democrats supported the Republican).
I was disappointed that the article only mentioned address-based sampling in passing, and that in reference to mail surveys. In his comment above, Steve H. mentioned knocking on doors. That is still how address-based sampling is carried out for many government surveys, employing the kind of perseverance if no one is home that Steve H. mentioned. Even there response rates aren’t as high as they used to be, but they are vastly higher than with most online surveys.
I do love surveys and I used them for my work, some being my bread and butter, and I try to squeeze the most out of them, which in my line of work it is seldom done.
People indeed lie in surveys, or troll masively… I have seen some provincial surveys where gender option had quite a few penguins in it… When that question gets hundreds of distinct answers, you know that something is very wrong with the notion…
But my greatest peev with surveys is how many avoid the real questions.
I complained to the PM for his lying about changing the electoral system from FPTP to something else. After he was elected, he greatly backpadelled and gave us a web based survey that had no question on it that would elicit from the population whether they want FPTP replaced or not…
Another argument I had with Statistics Canada and their long form Census, where there are lots of questions and splicings of income but not about wealth. There was only one national survey, more than a decade ago that was related to wealth.
Lots of other examples that I could think of when it comes to unearthing behavioural aspects and reasons, but I will leave it just at the above.
Yup, I am looking for the dark matter out there, and it is very hard to find it when those in charge with the surveys avoid like the plague certain aspects.
Are those the Banded Hudson’s Bay Penguins or the Labrador “Retriever” Penguins? (We won’t get into the curious state of the Siberian Penguin. The old Soviet Era “Penguin Preservation Directorate” is still chugging along under the aegis of the new “Russian Federation Species Diversity Program.” [See their strenuous efforts to protect the endangered Crimean Penguin.])
And: “said Duke University political scientist Sunshine Hillygus: It “undermines people’s trust that the American public is capable of self-governance.”
Is this some kind of joke?
D. Sunshine Hillygus is a real professor of Political Science at Duke University. So, that much seems to be legitimate.
I’m guessing that the penguin reference is Canadian slang for something not natural. Since penguins are not native to the Arctic, finding a penguin in Canada would rate as an impossibility. Thus, my facetious reply concerning several “species of Northern Penguin.” It’s similar to the Norwegian Blue Parrot of Monty Python fame.
The NC commentariat is known for its dry wit.
Nope, it is just a trolling I found in some provincial surveys that I have access to, in all their gory details.
I became highly suspicious of all surveys based on something I saw in Europe in the 80s. There was a famous English language paper sold there (I forget their name) which I use to read from time to time. And then they printed a survey to found out who their average readers were. But in the section regarding smoking they had options like cigarettes, cigars, pipes – but none for non-smokers. It was a rigged survey. A few years later there was this survey done in America and it got a lot of coverage in the news. The survey found that Americans were definitely getting more conservative but the actual number of people that they surveyed was in the low hundreds which meant that it was garbage, no matter how “scientific” that they claimed it was. You cannot make such a claim for about 240 million people back then based on about 200 people surveyed. After that I had a healthy suspicion on the truth of any survey.
PBS did a segment on their own polling with Marist. They said only 1:100 answer their poll. Personally I don’t answer any unknown callers. Text messages, emails straight to spam.
I’m curious. Does anybody believe the polls we are allowed to see? How about reports of “record” or “higher than expected” voter turnouts? I imagine we’ll be hearing that one again this year.
Wikipedia says 2016 turnout was ~60% and 2020 was ~66%. I am very doubtful that 2024 will beat 2020 or 2016.
My gut instinct is that 2024 turnout will be less than 2016 due to the “why bother, if you it’s once again turd sandwich versus douchebag” effect.
https://en.wikipedia.org/wiki/Douche_and_Turd
https://www.youtube.com/watch?v=E7pfsneLSSM
Study Shows Which Kids Are Getting Periods Younger Than Others Radiolab (15 minute audio)
This is a survey that has an impressive number of subjects and is both interesting and important. That it is being supported by a profit-seeking entity that requires subjects to own their pricey product to participate does give me pause on various points. For just one example, what if the results threaten the interests of other large for-profit entities? High input agriculture and big pharma come immediately to mind. Will we be told?
To me, the biggest problem with survey rigging is surveyors rigging the questions. By asking a question a certain way, pollsters can be made to advocate sending their own mothers to prison.
I believe we should start, or at least include, in the survey debate, this problem of survey rigging to achieve a pre-desired outcome on the part of the surveyors.
Not as big a problem in actual surveys: most reputable surveys don’t do this because this is such a well known problem (although there are billion research papers on how even slight question variations can lead to distorted answers. Whether instability in survey responses is driven by changing minds or changing questions (even very minor variability) is the eternally unanswerable question in survey research.)
The more challenging problem in the practice of surveys is how much the survey taker (even in an old fashioned phone surveys) might press for an answer (you can see variations on this in the instructions to survey takers–most respectable surveys publish these–ranginy from “must choose one of the options” to “do nothing further if they say ‘not sure.'”). In a close election with conflicted attitudes towards candidates (ie I hate both of these guys) like this one, this can make for a lot of difference.
It’s not just that — it’s by not providing an option for an answer that many people might choose if there was a choice for that answer among the multiple-choice answers provided. Those answers are deliberately excluded from the multiple choice because the survey takers don’t want answerers taking those paths.
A classic, if crude, example would be a survey saying “Which person do you want to be President? A Joe Biden B Donald Trump”
But maybe I’m rephrasing what you’re saying in your second paragraph?
But I suggest this is deliberate, not accidental.
It’s deciding what the public narrative should be and what people should be allowed to think, through survey manipulation.
Yes, wrt my second paragraph.
Polls without “I don’t know” as an option are useless, imho. Ideally, we’d want to have some idea of “what kind of I don’t know” that is, but we won’t get that in a survey.
I used to tell my methodology classes that “If you can’t rig a poll, you don’t belong in the business.” And to demonstrate I would give them a rigged poll where in the space of half an hour, the way questions were worded, the order in which they appeared, etc. got them to come down (by wide margins) on both sides of every issue.
Ever make the mistake of contributing to a political party? My wife did that and of course her phone is jammed with hundreds
of texts (“I’m not angry at you for not responding- just disappointed”) designed to make you feel bad, or otherwise manipulate you.
Worse, if you do contribute, they will set up automatic withdrawals from your checking account forever. You have to close and then
reopen with a new account to get rid of the withdrawals. Such behavior makes people leery of interacting with charities in general
and political ‘charities’ specifically. These organizations are not operating for free. The election industry is a multi-billion dollar
business, a purely corporate enterprise. Back in the 19th century, a voter got free beer and a ham sandwich for their vote.
Yes, it was certainly corrupt, but at least the voters got a tangible reward, instead of buying megayachts for anonymous
C-Suite marketers
A number of factors have driven down the unit cost of an interview (online survey methods, cheap sampling) so that fieldwork is no longer a $10,000+ investment but a $500 one or less. This brings a lot of new “researchers” into the supply side, many with limited skills and training. All kinds of people can run a survey now, with predictable negative results on the credibility of the research business. Gresham’s Law in action.