It's amazing how many times I run across this "logic." 'It's not very good, but it's all we have.' I hear this regarding risk analysis, security planning, and yes - survey methodologies.

When I challenge the orthodoxy, I am considered a pessimist. "If we listen to you, there's no management tools left! What we we supposed to do, just give up?"

More interesting to me is how we actually broach these conversations about uselessness with public officials.

The thing that gets me is how many of these studies are actually poorly executed from a social scinces and pure statistics perspective. I have come across questions which are redundant, pointless or impossible to answer and graphs and measurements that mean nothing. It's all very impressive looking though, and often the brief to the consultant is "here review this data and then let's put together a process to go forward." When I suggest that we need to start over and do real engagement, that doesn't go down all that well.

Hi, are you referring to a particular country ? And just local / provincial ?

There is a lot of info on the ONS site for the UK:

http://www.ons.gov.uk/about/our-statistics/how-they-are-produced

You can request their annual report and other breakdown of costs via their freedom of info:

http://www.ons.gov.uk/contact-us/freedom-of-information

I would disagree about their services being poorly executed as a whole. They do have a number of good professionals working throughout their organisation but I am not saying they are perfect.I do know conversation regularly takes place regarding the public perception of statistics and the value of them.

They have like just about every other UK government service (local or national) been part of the e / transformational agendas in the last few years, so have and still are going through huge changes.

As to whether they work or not, it depends whether you mean the government takes any notice of the statistics they are asked to produce and/or understand them or just does their own thing anyway I guess.

Good question, but complicated.

As someone involved in said research for 18 years, I concur with Chris' comment to a degree. There is bad research... but there are also bad doctors, lawyers, etc. That problem is not unique to research, although there is little in terms of professional misconduct rules in research in most countries so that likely makes it worse, but not sure by what degree.

Good research is often more about the motivation (to do it) than any technical aptitude I believe. One of the things that brought me to CE is my desire for continually doing things better and my real and true desire to make government work (outcome focused or what Dave calls 'impact focused'). But for many, the act of research is less lofty (or idealistic). And clearly people end up in positions characterized by the Peter Principle... (again not unique to research).

And I still utilize these applied research methods btw. But for years, many people avoid me like the plague, because of my expectations of rigour and thoughtfulness and committment in the research process. Some engage but eventually walk away exasperated, and look for someone else to help.

This hasn't been a great answer to the question perhaps, but if anyone involved in CE has an expectation that CE methods will replace the entrenched market/policy research industry in the next 5-10 years, I think you may be dillusional.

As to a more straightforward answer to the question, the number spent is likely in the $billions in Canada. The amount of research that has no impact may also be in the $billions. But again I argue, all of our governments do things for reasons we can't possibly comprehend (nor do we want to). So this issue of bone-headedness (not sure how that will make the cultural translation) is not, I would argue, unique to research. When we "don't know what we don't know" we have exposure.

And BTW, CE methods, poorly applied will contribute to that pile of waste. So let's not be overly omnipotent about this.

Dave, even in your short stay here this week, you changed some very important minds. So for me I do see light at the end of the tunnel, that for once may not be an oncoming train.. 8:)

My $.02 (CDN) worth.

Ray

This answer is a little old (2000) but gives an order of magnitude figure for all of government:

"According to parliamentary written answers, the Government spent more than £2m on focus groups and opinion polls in its first year. Most spending appeared to have been undertaken by arts-related quangoes. The Department of Culture, Media and Sport said its non-departmental public bodies and agencies had spent £530,000 on focus groups and £1.4m on other forms of market research in Labour's first year.

"The Lord Chancellor's Department spent £5,920 on a survey of attitudes on law and order. The Scottish Office spent £14,000 on groups to develop a Scottish Parliament information campaign. The Home Office spent £7,500 researchingattitudes on the use of CS gas."

Source: http://www.independent.co.uk/news/uk/politics/officials-warned-on-use-of-focus-groups-718723.html

More up to date figures for one government department (Culture, Media and Sport are as follows:

(a) Focus groups

Yr £
2003-04
221,670

2004-05
102,005

2005-06
32,515

2006-07
24,400

2007-08
80,105

2008-09
30,623

Other research commissioned is likely to have included a focus group element but it is not possible to determine the costs of these elements. Further estimated costs for 2008-09 will be £100,000.

(b) Opinion polls

£
2001-02
43,662

2002-03
74,701

2003-04
83,010

2004-05
20,000

2005-06
42,700

2006-07
13,200

2007-08
98,304

2008-09
58,700

There are no further estimated costs for 2008-09.

Source: http://www.publications.parliament.uk/pa/cm200708/cmhansrd/cm081126/text/81126w0158.htm

Hmm, looking at the numbers it seems that Focus groups have gone out of fashion in the lasy decade whereas opinion polls are pretty contant?

On a bit of a roll here (OK, I realise this is central government spending but still good to have some stats): The Ministry of Justice has spent more than £1.2m on focus groups and opinion polls since May 2007...Almost £500,000 has been spent on focus groups and over £750,000 on opinion polls, figures revealed in a Parliamentary answer show. (dated Nov 2008, source: http://www.libdems.org.uk/home/ministry-of-justice-spent-£12m-on-focus-groups-browne-73892410;show)

During the past five years, spend on opinion polling by the Home Office and the UK Border Agency and predecessors has been as follows (IND and BIA were the UKBA predecessors):

Spend on opinion polling (£)
2003-04

HO (Inc. IND)

128,850

2004-05

HO (Inc. IND)

124,610

2005-06

0

2006-07

Home Office

97,075

BIA

82,500

2007-08

Home Office

69,700

UKBA

45,000

Source: http://conservativehome.blogs.com/parliament/2008/10/the-governmen-1.html

Will keep looking for info on local/regional government...

Total Government spending figures can be found here http://www.ingenium-ids.org/uk_gov_spending/ so if figures known for a few departments might e able to make some pro-rata estimates for local/regional government.

there is nothing more a government hates than to be well informed, because then it has to make proper decisions.

I saw a quote along these lines a few years ago.

Hi, having faced the same problem in Greece, I think two good reasons for their orthodoxy are:
a) their need for something solid, "objective" and "resultfull" in return for the money they pay, and
b) their assumption that numbers and bars and pies are generally considered as such and therefore they are covered (although all admit that questionarries results can be seen from a different perspective).

Dave

You open up a whole can of worms here.

To boldly state that surveys and focus groups 'don't work' requires you, I think, to define 'work'.

I think the issue is not the technique per say that needs to be addressed, but rather the application of that technique, the context, the skills of individual researchers, the intention of commissioning body etc etc.

My experience is that practitioners, and those commissioning the practitioners, often have little understanding of the methods being used or the development of the methods. This often means that they don't understand the caveats in the model or technique.

Conversely, some people do understand and are cynically able to pick and choose methods designed to produce the required outcome.

I experienced this all the time during my days as a climate modeller and we used it all the time during public inquiries as a tactic to win cases eg throw doubt on the measurements/model.

Most recently we have challenged some of the more insance econometric modelling that goes around - especially in construction which has played a partial role in current crisis.

I am constantly amazed to find people who believe that the forecasts produced by these models are some form of accurate 'truth'.

Final point, whilst CE methods offer another way of approaching this, they are not immune from any of the issues listed above.

Therefore, any post or message which implies "CE methods right, old methods wrong" leads you down a dangerous road.

I am going to write a longer blog on this at some stage in the future. However I disagree with Stuart in several respects. Firstly some techniques are simply inappropriate (questionnaires to understand attitudes etc) so matter how well executed. You have to understand the theoretical basis on which the method or tool has been constructed to understand its limitations.

Oh and nothing in my post could remotely be construed as saying CE methods right, old methods wrong. That would be crude

Dave

I actually think we agree more than disagree.

Of course questionnaires are not suitable for understanding attitudes. If that is what you meant by 'they don't work' then I agree.

That is what I meant by 'context'. The use of a technique in the wrong circumstance - happens all the time. Probably could have phrased that better.

Agree that you need to understand the theoretical basis of the method to understand its limitations. Hence my hope that newly structured CE accreditation courses don't lose this.

I didn't intend to suggest that this particular post 'construed' that. Apologies if it came across that way.

Look forward to the long post on this one.

Another way in might be to look at some of the players.
http://www.ipsos.com/docs/pdf/MORI%20Analyst%20EnglishDef.pdf
has some numbers to aim at e.g. 60% of Euro 64M in 2004 for MORI.