Yves here. This post gives some generally good guidance about interpreting polls, using impeachment as the topic of interest.
I would add:
1. As indicated, poll results are very sensitive to the formulation of the question. But it would have been nice to provide illustrations. Some studies have found that seemingly minor changes can shift results by ten points.
2. As readers have discussed, polling is in a crisis due to how few people use landlines and answer calls from unknown parties. Pew has revealed it can reach only about 6% of the public, and landline phone users skew older and better off.W While pollsters try to weight their sample to compensate, if you have hardly anyone in certain demographic slices, it’s way too easy to gross up unrepresentative results
By Michael Traugott, Research Professor at the Center for Political Studies, University of Michigan. Originally published at The Conversation
Pollsters are trying their best to track public opinion about the House Democrats’ decision to initiate an impeachment inquiry against President Donald Trump.
Currently, it is doubtful the required two-thirds of the Senate, now controlled by Republicans, would vote to convict Trump. But senators’ votes could change by the time they receive the articles of impeachment from the House.
That is why tracking the changes in the president’s support in the polls is so important. Members of Congress will factor in what the public thinks as well as their own assessments of the president’s behavior.
For a variety of reasons, these are difficult measurements to make. As with all polling, the results depend to some extent on who is polled, what they are asked and when.
Whom Are You Asking?
New polls on the topic are being released every couple of days, and journalists are reporting on the latest shifts in opinions.
Many stories link attitudes on impeachment to the president’s level of support as measured in his approval ratings.
Attitudes on impeachment – and presidential approval – reflect the current state of political polarization in the U.S. Trump’s approval ratings are determined by strong support among Republicans and disapproval among Democrats; party identification has become the primary determinant of most attitudes in the U.S.
Most polls involving political issues focus on one of three target populations: adults 18 years of age and older, registered voters or likely voters.
Each of these is a subset of the previous one. Most importantly, each subset is generally more Republican-leaning as well.
Presidential approval is commonly measured for all adults. But questions about candidate support – relevant as the U.S. works through the primaries and next year’s election approaches – are typically analyzed for registered or likely voters.
As these samples get smaller by definition and therefore more Republican, this usually mean greater support for Donald Trump and less for impeachment.
Political groups will sometimes produce press releases based on selected samples, such as people “in Republican-held congressional seats and National Republican Congressional Committee targets.” These groups are obviously more likely to be partisan and unrepresentative of the national population.
Partisanship is strongly related to attitudes about impeachment. Consumers should look for differences in opinions by party identification in any single poll. Changing attitudes over time could be related to differences in the partisan composition of samples from different polls, even from the same polling organization.
Democrats are generally more willing to participate in polls while Republicans disproportionately refuse. This potential problem has appeared in some polls in the past. This makes it especially important to compare the partisan makeup of poll samples from different firms, as well as whether the partisan composition is changing over time for the same polling organization.
What Are You Asking?
But what does “support” for impeachment mean?
Pollsters are asking a variety of questions to measure that concept.
Some use the phrase “start an impeachment inquiry” while others use the phrase “impeach President Donald Trump and remove him from office,” asking for a harsher judgment. “Support” is generally higher in response to the first question than in the second.
Due to the variations in question wordings used in different polls, the search for shifts in opinions over time can be sensitive to the wording used in the polls.
Some polls are asking about how high a priority impeachment and removal of Trump should be.
In some polls, the impeachment question includes an explicit “don’t know” option, while in others it is a volunteered response. The proportion of “don’t know” responses is quite variable right now from poll to poll, but I expect it to decline over time as people are exposed to more news coverage.
Comparing results from different polls using different wording can be confusing and misleading, so it is best to compare polls conducted by the same organization at different points in time to see whether support for impeachment is stable, increasing or decreasing.
When Are You Asking?
Public opinion on impeachment will change as more information becomes available from additional sources. Those sources could include a second whistleblower and documents disclosed by the congressional inquiry.
Polls about impeachment have been conducted since Robert Mueller began his investigation. However, the number of polls increased substantially after news about President Trump’s phone conversation with the president of Ukraine.
These opinions were pretty stable until that disclosure, with slightly more Americans opposing impeachment than not. Support for impeachment among Democrats was about twice as high as among Republicans, with self-described Independents split evenly.
Since the disclosure of the phone call, support for impeachment among all groups has increased. It increased the most among Democrats, but also among Republicans, and now a majority of Americans support impeachment.
In my view as a polling researcher, the most consequential shifts in opinion on impeachment will be among Republicans and Independents, as Democrats are already pretty solid in their views.
21 comments
Comments are closed.
However detailed, I am putting this all down to static. There is only one poll that matters and that is the one that will be held on Tuesday, the 3rd November 2020.
That’s the point of the impeachment process – to make sure that Trump isn’t around in November 2020. If the Democrats have a 50/50 chance of beating Trump I gotta believe they have even a lesser chance of beating “generic Republican” President Mike Pence. Nancy Pelosi will win the battle and lose the war.
Hooray Democrats.
On the graph, Number of impeachment polls.
It is clear that there is some magnitude here, but where is the y axis?
If you hover your cursor over the columns in the graph, the numbers they represent appear. But agreed that a y-axis would have been much more user friendly.
Another point I have not seen addressed is that the question “Do you support impeaching Trump?” can be taken at least two ways. It usually is taken to mean “Do you believe that Trump has committed high crimes and misdemeanors in office?” Well, of course he has. duh. (Or, if we go with “impeachment investigation” we can change this to belief that there is credible evidence of said crimes and misdemeanors. Also ‘duh.’) But it also can mean “Do you believe that impeaching (or investigating) Trump is politically expedient at this time?” I have a lot of respect for Pelosi’s political instincts: more than I do for mine, and she is better informed. So in this light, my opinion on the political expediency switched when Pelosi announced the investigation is on. Given the numbers, I suspect I am not alone.
Is impeachment an extension of the soft coup ongoing since January 2017?
Well of course, duh.
Pelosi’s political instincts led the Democrat party to not impeach Bush and Cheney after begging voters in 2006 to put them back in power because “elections have consequences”. But of course they didn’t, Bush finished out his term and is now working on his reputational rehabilitation tour with the help of said Democrat party rather than rotting in the Hague where he belongs.
If they wouldn’t impeach Bush for glaringly obvious war crimes that set the Middle East on fire, a conflagration that continues almost 20 years later, then how in the hell do you impeach a reality TV star for talking to a comedian about a buffoonish and obviously corrupt former US official?
You want to impeach Trump, impeach him for the attempted coup in Venezuela – but they won’t because the Dems were on board with that little but of illegality. Pelosi political instincts must be telling her that’s OK.
I’d like two questions asked:.
1. Do you believe that Trump has committed more egregious and harmful high crimes and misdemeanors in office compared with Bush Jr (who instituted $6Trillion cost Iraq war with more than 200K dead Iraqis) and Barack Obama (who droned to death civilians in foreign lands, continued Bush Jr’s wars, persecuted whistle-blowers exposing government malfeasance, and was instrumental in destabilizing Libya)?
2. If Trump merits impeachment compared to the prior two residents of the Oval Office, what are Trump’s comparative high crimes and misdemeanors that cause one to believe that?
If Trump gets away with privatized medicare, he may end up impoverishing and or killing a lot of old folks who don’t have the money to afford market based health care, or at least quality health care from such a system.
Are you implying that Trump merits impeachment because of something he MIGHT do in the future (perhaps with a Democratic House’s acquiescence)?
Seems like pretty weak impeachment tea to me.
I have a lot of respect for Pelosi’s political instincts.
Damn. My coffee ended up on my screen. Have to clean it off now.
hell with the polls, i’d like to know which impeachment stories to believe…
None of them?
or all of them, to
in spire our Russian spy?
Having contracted for polls in years past, I have serious doubts about the current ability of pollsters to measure anything meaningful.
First, the only statistically accurate polls are those where participants are selected at random. Email and internet polls are suspect, because the randomness of the respondents is questionable. Historically that usually meant that the only meaningful results came from telephone polls where participants could truly be selected at random. But those were the days when most people used land lines. Today, land lines are going the way of the dodo bird.
Second, polls depend on people’s willing participation. Telemarketing scams have largely poisoned that source. Now telephone polls are largely reduced to a sample of people willing to pick up the phone. Among those willing to pick up, it is further reduced to a pool of trusting souls who are willing to believe that the stranger at the other end of the line isn’t going to end up trying to sell him something or steal his identity at the end of the call.
Bottom line: the pool of people available for a random sample has been decimated. How big is it? How truly representative is it?
I have never seen a pollster divulge that information.
As somebody who used to do large-scale surveys (samples > 40K) for a living, this article makes solid technical points. It is misleading to lump survey items from different questionnaires when their wording and response options are not the same. In addition, samples are often cherry-picked out of convenience, and often based upon very small samples – often under 1,200 weighted (i.e., handicapped) by demographic self-descriptions to impute generalizability (i.e., smoke and mirrors). Reports seldom report how many refusals, hang-ups, or v-mail transfers they got, or if relying on paid internet respondents (yes they have such things a paid poll pools.
For example, % agreeing often lumps dichotomous responses with 5-point Likert responses, often they add up as agreeing 3)unsure 4) agree 5) strongly agree – if the author’s bias is to disagree, then they add up 1,2, and 3. In either case, not showing the distribution of responses can be very misleading. A U-shaped curve reflects polarization, whereas a bell-shaped curve might reflect high uncertainty.
As shown, wording in these polls is often very different. “Should Donald Trump be impeached?” is not the same question as “Should Congress conduct impeachment hearings?”
Wording is an art and in polls can be used to ensure the desired poll results, which in turn can be used to influence voters.
This post describes the technical points of wording beautifully, and of how they can be used skillfully to manipulate. The same concerns of wording crop up when considering resolutions, constitutional amendments or referenda. Brexit comes to mind. So much depends on precisely how the question is worded, and on how that wording was chosen, and by whom.
As ever, Sir Humphrey got there ages ago. Seeing this episode at an impressionable age did wonders for my powers of skepticism. Well, the whole series did, actually.
https://youtu.be/G0ZZJXw4MTA
Yeah, there is a bit of difference between an inquiry and getting kicked out of office. The headlines smear the distinction consistently
Would be interesting to compare polls to betting markets.
As an observer of politics for decades, I believe the number of true independents is inflated. There are many people that claim to be independents on both sides that aren’t independent.In listening to callers on CSPAN’s Washington Journal for years , a large percentage of people calling in on the independent line simply repeat the republican or democrat talking points. These people really aren’t independents. The same is probably true when asked by pollsters if they are democrat, republican, or independent. Many claiming to be independents aren’t independents. I don’t remember the exact percentage but my gut feeling is that the percentage of real independents is about half of the stated number. Polls are very dependent on the question(s) asked and the people polled. The internal polls , that usually aren’t made public, probably are the most accurate.