One of my pet peeves is lousy research, and even worse is lousy-with-intent pseudo-research. The airwaves are filled with fake factoids crowding out real information.
I was once tempted to write a post on what it takes to do good survey research, simply because surveys are widely cited and most people probably mistakenly think they deliver reliable findings. In fact, it takes quite a lot of work and testing to devise a good survey instrument, and most people sponsoring studies don’t want to spend the time and money.
John Kay of the Financial Times has taken aim at the same topic, for different reasons: too many surveys these days are done for public relations purposes, to stir up some noise, and were never even undertaken as serious research.
The more the public starts to be skeptical of pronouncements based on surveys, the happier I will be. From the Financial Times:
This is the age of the bogus survey. I woke up recently to the news that 95 per cent of children in Britain had been victims of crime. Of course they had. From a legal perspective, pushing a classmate or taking a pencil without the intention of returning it is a crime. School playgrounds are hotbeds of crime and always have been.
The difference between the bogus survey and real research is that real research has the objective of yielding new information, while bogus surveys are designed to generate publicity. The organisation that had undertaken this bogus survey – I forbear from mentioning its name – did not disguise that it had done so in order to draw attention to the problem of abuse of children.
Statistics about the incidence of real criminal activity against and among children are hard to come by and hard to interpret. We do not really know whether things are getting better or worse, or by how much – at least not without careful research and analysis, which would be hard to explain on television. Programme producers will not ask you to appear to spell out these complexities, but will allow you to horrify viewers and listeners with alarming news.
Public relations professionals understand these triggers, to such an extent that commissioning a bogus survey is now a standard element in the pitch they present to potential clients and conducting these surveys is an increasingly large part of the activity of market research organisations.
The agencies appreciate, although they are normally too polite to spell it out to their clients, that Universal Widgets is not a very interesting company, widgets are not a very interesting product, and Nigel Snooks, the chief executive, is not a very interesting man. But a survey that shows that two-thirds of men have contemplated hitting their wives with a widget will produce many media slots in which Mr Snooks of Universal Widgets can recount the findings.
There is even a term for this kind of activity. It is called “thought leadership”. That term illustrates the problem. It probably does not matter much that the bogus survey is used to generate spurious news. The danger is that opinion polls designed to produce eye-catching answers displace serious thought and analysis. The organisation that announced that 95 cent of children had been victims of crime judged, correctly, that its survey better served its needs than serious research into the problems with which it was concerned, that had not been done.
The study of business is afflicted by confusion between the results of a survey of what people think about the world and a survey of what the world is really like. At another recent meeting I heard a platform speaker announce that 40 per cent of books would be electronically published by 2020. A pesky academic asked exactly what this number meant and what evidence it was based on. The speaker assured the audience that the number had been obtained in a survey by eminent consultants of the opinions of the industry’s thought leaders.
I imagine most of the thought leaders had no more idea than anyone else what the question implied, or what the answer was, and did not devote more than the briefest consideration to their response, so I am not surprised that the median answer was close to a half. If you want to know the future of publishing, you will learn more by peering into a crystal ball. It will at least give you time to think.
Newspapers, broadcasters and consultants will start to distinguish bogus surveys from substantive knowledge only when their audience demonstrates that it knows the difference. Academics and think-tanks need to be reminded that generating publicity is not a legitimate research objective. The column in The Week magazine called “what the scientists are saying”, a compendium of silly claims from scientists trying to attract attention, is as embarrassing to the cause of real science as Private Eye’s Pseuds’ Corner is to real literature.
When you are asked for your opinion in your role as thought leader, put the phone down. You will be serving the public interest as well as saving your time.
One comment
Comments are closed.
Such methods, as you might know, have been and are a near essential practice of the public relations industry in its now well proven ability to convince people that what isn’t is or the contrary, to manipulate and mold ‘the public mind’ while, so far as possible, remaining what Scott Cutlip called an ‘unseen power’.