Damned Lies and ‘ Problem Gambling ’ Prevalence Rates : The Example of Victoria , Australia

This paper takes up the challenge to consider whether it is possible reasonably to overcome a number of sobering methodological deficiencies in published prevalence-rate estimates of ‘problem gambling’. The prompt to do so was the 2006 claim by the Victorian Government to have halved ‘problem gambling’ in the State. The paper shows the claim to be unscientific. It reinforces this view by demonstrating, using evidence and argument from the Productivity Commission, the Australian Bureau of Statistics and statistical/epidemiological sources, that extant sample survey techniques used to estimate prevalence rates are seriously flawed.


Introduction
The Government of the State of Victoria in Australia made a remarkable claim in its 2006 gambling policy document, Taking action on problem gambling: A strategy for combating problem gambling in Victoria (Department of Justice (DoJ) 2006).The claim was that: Since coming to office in 1999, the Government has reduced the size of Victoria's gambling problem.In 2006 the growth rate of gaming machine expenditure has fallen to 3.3 per cent.The rate of problem gambling is now estimated by independent research to have halved to 1.1 per cent of the Victorian adult population or 43,957 individuals … Research shows that the prevalence of problem gambling in the Victorian population has declined from 2.14 per cent in 1999 to 1.12 per cent in 20031.12 per cent in . (DoJ 2006, pp. 6, 33;, pp. 6, 33;citing the 2003 Victorian longitudinal community attitudes survey, Centre for Gambling Research, Australian National University) 3  The fact is that McMillen and Marshall provided blunt warnings about the prevalence rates that they reported.Their caveats amount almost to denying that we can rely on sample-survey methods to estimate the actual prevalence of 'problem gambling' in any meaningful way.Rather than put words in their mouths, it is better to let the authors speak for themselves.The SOGS 5 scores to which they refer derive from survey questions designed to 'screen' or test for the identification of gambling problems.
As measured by SOGS 5+ the overall figures (assuming problem gambling prevalence rate of 1 per cent, a severe problem gambling rate of 0.3 per cent and an adult Victorian population of 3,475,197) suggest that in 2003 approximately 34,751 adults in the Victorian population could have gambling problems, with 10,425 of them experiencing severe problems (SOGS 10+).
In the US it is suggested that people scoring SOGS 3-4 are also at risk of gambling problems.In this Victorian survey this would account for 33,015 being at-risk of gambling problems.However this lower threshold would likely generate an unacceptable number of false positives amongst the problem gambling population.
We emphasise, however, that the application of such screens in population surveys can only provide a broad tentative indication of the prevalence of gambling problems in the community.As noted by the Chairman of the Productivity Commission: 'all survey gambling screens are likely to underestimate the extent of problem gambling -however they may choose to define it -simply because people have a natural reluctance to reveal the facts about such matters'.
It is likely that the inherent downward bias of survey data is particularly marked with 'problem gamblers'.In the Commission's 1999 survey of self-confessed problem gamblers in counselling, 'only 29 per cent said they would have responded to a survey honestly; one-third said they would have concealed their problems, and some 24 per cent said they would have refused to answer the survey!'If that response pattern has been replicated in this 2003 survey, the problem gambling prevalence data presented here is merely symptomatic of a much larger problem in the Victorian community.(McMillen & Marshall, pp. 87-8;citing Banks 2002) These are strong words, careful words.In addition to indicating grounds for argument about where to draw the cut-off line for the SOGS categories, they stress that 'the application of such screens in population surveys can only provide a broad tentative indication of the prevalence of gambling problems in the community'.They cite the Chair of the Productivity Commission as evidence that fewer than one-third of 'problem gamblers' would be likely to answer survey questions honestly.They conclude that 'the problem gambling prevalence data presented here is merely symptomatic of a much larger problem in the Victorian community'.
Did those who wrote the Government's policy document read these caveats?It would be surprising if they did not.Are they aware of problems with sample prevalence surveys?Again, we would be surprised if they were not.Did they heed McMillen and Marshall's warnings?Obviously not.Were they under an obligation to repeat the warning when they marketed the prevalence-rate product to an unsuspecting public?Were the Premier and the Minister for Gaming obliged to exercise equivalent caution when they signed off on the policy and their introductions to it, which also trumpeted to usthe public -the claim that the Government had halved the number of problem gamblers (DoJ, pp. 2, 3)?If it is reasonable to hope that ministers of the Crown and purportedly independent public servants are honest in discharging their official duties the answer to both questions must be yes.Otherwise we would have to accept that it is reasonable for ministers and 'our' servants to participate wilfully in misleading conduct in their engagement with us as citizens.(1997)(1998)(1999)(2000) Table 1.'Problem-gambling' prevalence surveys in Victoria 1996-2003 Juxtaposing what the authors of the 2003 Victorian longitudinal community attitudes survey actually said about prevalence rates with the use made of these rates by the Government shows the latter to have been misleading.Understanding more about why sample-survey prevalence-rate data are suspect will reinforce this point.I will turn to this question in the next section.

Inherent Problems with 'Problem-Gambling' Sample Surveys: 'False Negatives'
Advocates of sample surveys to measure the prevalence of 'problem gambling' might offer the defence that their methods are indistinguishable from standard epidemiological survey research.Insofar as epidemiologists use sample surveys, this might be true.However, we should accept three very important qualifications.The first is that epidemiologists would urge greater caution in using survey data, as would statisticians (see e.g.Rothman 2002).The second I will discuss in a subsequent section.It is that, when dealing with small proportions or numbers, researchers should use different and more sensitive statistical techniques (Rothman 2002, pp. 130-4).The third is that measuring 'problem gambling' by 'self-reporting' is very different from measuring other health problems for precisely the reason indicated in the previous section: under-and false-reporting.
To make the third point more forcefully -for it is the fundamental problem with prevalence surveys -I will discuss in detail the Productivity Commission's warnings.In 1998 the Productivity Commission engaged Roy Morgan Research7 to survey by telephone an initial random sample (n = 10,609) of Australian adults, of whom about 2,200 were in Victoria.Some did not want to participate, and others turned out to be unsuitable.In any event the effective sample size turned out to be 3,498 adults, of whom about 725 were in Victoria.Some were not gamblers, and others gambled infrequently.Callers eventually administered the SOGS to 1,225 adults nationally who had gambled 52 or more times in the past year or who had been heavy losers ($4,000+ p.a.).The Productivity Commission survey did not administer the SOGS to those who merely participated in lotto draws or bought scratch tickets.In Victoria the number who were asked SOGS questions was 254.The Productivity Commission weighted the sample to make it more representative of the Australian population.Its guide was the approach of the Australian Bureau of Statistics Household expenditure survey of about 8,500 adults every five years (Productivity Commission 1999, Summary p. 24).
The Productivity Commission survey suggested that Australia-wide the problem-gambling prevalence rate was 2.07 per cent.In Victoria the rate was 2.14 per cent.This figure the Victorian Government says that it has halved.Now, despite the effort to which the Productivity Commission went to administer its 'National Gambling Survey', it evidently had deep misgivings about its prevalence-rate findings.Chapter 6 and appendix F of the Commission's report discussed them succinctly.Its concerns were that sample surveys (both telephone and face to face) would encounter: 1. Problems in contacting some gamblers, who may be out at venues or have their telephones disconnected because of financial hardship.2. Problems of representation, in particular that 'People in certain cultural groups may be more uncomfortable about openly divulging personal issues, like problem gambling'.(1999, p. 6.35) 3. Problems of non-response and refusal, namely that 'problem gamblers are more likely to refuse to participate because they are unwilling to answer potentially embarrassing questions'.(1999, p. F2) 4. Problems of denial, which is to say that, 'even where problem gamblers agree to participate in a survey, they are more likely to be reluctant to provide truthful responses and to minimise the problems their gambling has created for themselves or others'.(1999, p. F2; see also 6.34-6.35) Moreover, 'precise tests of problem gambling are impossible, because … the phenomenon itself lies on a continuum of differing degrees of severity' (Productivity Commission 1999, p. 6.18).This raises: 5. Problems of cut-off scores or thresholds in problem-gambling screens such as the SOGS.'Therefore, constructing a threshold depends on judgements about what levels of severity are policy relevant.' (1999, p. 6.18; see also Dickerson et al. 1996 re the 'at-risk' category).6. Problems of appropriate application, such as whether the SOGS, a clinical tool, is appropriate in population-screening, in which it might yield an unacceptable level of false-positive counts of problem gamblers (1999, pp. 6.19-6.23).
Of the six problems with prevalence surveys above, numbers three and four worried the Productivity Commission the most.False negatives will overwhelm any false positives.The following long quotation makes this clear unequivocally, as does table 2 below (the 'Table 6.9' to which the quotation refers):  Note that the quotation from the Productivity Commission above possibly underestimates the likely error.Table 2, the Commission's table 6.9, shows why.In saying that 38 per cent said that they would have answered honestly, the Commission excluded the 23.7 per cent who said that they would have refused to answer the survey.In other words, the Commission's survey violated the assumption of randomness that underpins using survey methods to estimate prevalence.This is so because we cannot say whether those with gambling problems will refuse to participate in gambling surveys in the same proportion as would the general population.For all the reasons I have discussed it is likely that more 'problem gamblers' would bail out.Table 2 has it that 28.9 per cent of 'problem gamblers' in counselling said that they would have answered honestly, while 0.2 per cent said that they would have exaggerated.Perhaps, then, it is fair to say that the Commission's data suggest that between 30.1 per cent and 38.0 would not have concealed that they had gambling problems.At the lower end of the range, if only 0.3 per cent were the SOGS10+ group (severe problem gamblers) then the actual rate would be 1.0 per cent not 0.7 percent (as in the quotation).Were we to apply the same factors to the various prevalence rates from Victorian studies in table 1, the actual prevalence rates would be as shown in  Understanding just how much users of poker machines falsely and under-report their losses will further reinforce this point.Reported Victorian poker-machine losses in the ABS Household expenditure survey (HES) comprise a laughably small fraction of total losses, as table 4 shows.Indeed the ABS's submission to the Productivity Commission inquiry was explicit about causes of the differences between the HES losses and known industry data: These include reporting issues (eg householders may have difficulty in recalling and isolating gambling expenditure separately from other forms of expenditure such as food, drink and entertainment and/or may more readily recall winnings as opposed to losses) and conceptual differences (eg HES excludes expenditure by overseas visitors to Australia whereas they are included in industry estimates).However, the … [differences] indicate a very significant degree of under reporting in the HES which is unlikely to be explained by the reporting errors mentioned above.This clearly indicates that respondents are deliberately failing to report the full extent of their gambling activities.This may be due to a concern that they have that other members of the household and ABS staff will see the diary of expenditures and may judge their gambling activities as excessive and/or anti social.These difficulties are demonstrated … to the extent that the HES estimates show households in NSW, SA, WA and NT all with negative expenditure (or all winnings) on the 'TAB, on course betting etc ' for 1993' for -94. (ABS 1998, pp. 4-5;, pp. 4-5;our emphasis) The ABS submission warned that 'HES data on gambling is significantly under-reported and hence any analysis based on, for example income distribution, may be questionable' (1998a, p. 10).We know that heavy losers contribute more than half of total losses in Victoria, with one estimate being that six per cent of Victorians lose 60 per cent of the total: a staggering $1.5 billion per annum, on current estimates (Doughney 2007;2006).Yet precisely these heavy users will be least likely to be honest about the true extent of their gambling.Often the extent of the problem is not revealed until well into counselling, and it is hidden from loved ones and friends to avoid embarrassment and shame.
Unwillingness to speak about gambling can be especially acute for some cultures (see, e.g.1998, pp. 30-31).
None of the criticisms in the preceding paragraphs should be read as criticisms of the Productivity Commission or of the ABS.The Commission was candid about its findings and their limitations; the ABS likewise.The criticisms are of using sample surveys of self-reported data to estimate population prevalence in the particular area of gambling.We simply cannot express any confidence that the reported prevalence rates accurately reflect the actual prevalence of gambling problems in the population.This is what tables 3 and 4 are saying.Moreover the problems with the method seem to be inherent and insurmountable.

Population Prevalence Rates for 'Problem Gambling' are Imprecise Statistically
Our consideration of the issues above has exposed the main reason that sample surveys of selfreported data to estimate prevalence of 'problem gambling' are scientific on the surface only.This is that the source data are unsound because most will go unreported.In this section I will outline another deficiency, namely that the level of statistical precision possible is weak even with the reported data.
The first motive for us to call claims to statistical precision into question is that the reported data are unstable.Three reasons exist: 1. the data are sensitive to sampling choices; 2. different sampling choices mean that it may be difficult to compare surveys; and 3. small prevalence rates mean that estimates can deviate widely by chance, which also makes it difficult to compare different surveys.
Our second motive for challenging the statistical soundness of the reported prevalence rates also has to do with their small size.I will suggest that the various studies do not measure confidence intervals well.
The first reason seems rather more prosaic than are the other two, but its effect can be significant.Refer again to the table 1 row 'Criteria for selecting those to whom SOGS administered'.The information in this row designates who, from the original sample, will be tested for 'problemgambling' behaviours.That is, it delimits who might possibly be called a 'problem gambler'.Note the differences in criteria.The fourth to the seventh community surveys administered the SOGS to all, and they included Lotto and scratch-ticket buyers in the sample.The Productivity Commission excluded the latter, and it selectively administered SOGS to 52-times per year gamblers and those who lost (on self-report) more than $4,000 per year.This inclusion helped to compensate for excluding binge gamblers, who might gamble less frequently than 52 times per year but still lose heavily.In contrast KPMG administered SOGS to those who had gambled in the last six months, including Lotto and scratch-tickets buyers.Finally the 2003 Victorian longitudinal community attitudes survey excluded Lotto and scratch-tickets buyers and used a once-per-week cut-off criterion without provision for binge gamblers.A once-per-week criterion also may be interpreted more strictly than a 52-times per year criterion.Exclusion of possible binge gamblers is a serious omission, in our opinion, but it was honestly acknowledged in the Foreword to the report by then Chair of the Gambling Research Panel Linda Hancock but not subsequently by the Government: … the application of gambling prevalence screens only to regular gamblers (defined as those who gambled at least weekly or 52 times per year, in gambling activities other than lottery games or instant scratch tickets) may have excluded others who gamble less frequently but page 48 Vol 2, No 1 for whom gambling is nevertheless problematic.(Hancock, in McMillen & Marshall 2004, p. 3) The second reason that the reported prevalence data are unstable follows directly from the first.If surveyors make different sampling choices, it becomes unsafe to compare surveys.Part of the deviation in prevalence rates illustrated in tables 1 and 3 is likely to be due to such choices.Hence, on this ground alone, it was unwise to compare, without apparent qualification, the Productivity Commission's 1999 survey with the 2003 Victorian longitudinal community attitudes survey.They were 'measuring' different things.The former was more sensitive to binge-gambling; the latter not.
For the authors of the Government policy to have ignored this basic rule is slipshod.
The third reason for instability in prevalence rates is that, because the numbers of identified 'problem gamblers' are small, estimates of prevalence can deviate widely by chance.Notwithstanding the preceding paragraph, note how small the numbers are in table 5 below and how much the prevalence data go up and down, even in contemporaneous surveys.Notice also the relatively large standard errors and relative standard errors of the prevalence rate, wide confidence intervals at the 95 per cent level and high percentage confidence intervals.With such small weighted numbers of adults scoring SOGS5+ -the largest being 72 for Australia and 27 for Victoria -something intuitively tells us to be careful with the results.Again Hancock gave the correct warning in her Foreword: 'In reading the problem gambling prevalence rates in this study, the standard errors of estimate suggest caution' (Hancock, in McMillen & Marshall 2004, p. 3).
Our intuition, however, might amount to little.We need to explore the numbers more rigorously.Again the Productivity Commission, echoing Dickerson et al. (1996), had suggested an appropriate route by sounding a signal warning about small prevalence rates.Footnote 34 on pages 6.47-6.48 of Australia's gambling industries explains where we can go wrong in such cases: If something is rare among a population then different samples of that population will tend to provide estimates of prevalence which deviate considerably.For example, suppose that the true prevalence rate was 0.5 per cent and a random sample of 1000 adults was taken … The likelihood of discovering just 5 problem gamblers (the expected number of problem gamblers) in the sample is only 17.6 per cent.There is a 12.4 per cent chance of finding 2 or less problem gamblers, and a 13.3 per cent chance of finding 8 or more problem gamblers.
As Dickerson et al. [1996a] note, small prevalence rates stretch the accuracy of the survey method to its limits.Indeed, apart from the early national study, with sample surveys ranging in size from around 1200 up to 2000 participants, the number of problem gamblers identified across the various state studies ranges from only 2 to 9, a variation which could arise purely from chance.This is evidenced by the fact that the 95 per cent confidence intervals for the SOGS 10+ prevalence rates overlap for all states.(1999, pp. 6.47-6.48; our emphasis) The formula the Productivity Commission used to calculate the probabilities in footnote 34 I will also use. 8The symbol Pr(s) in equation ( 1  The inverse of the probability of getting it right or actually identifying s 'problem gamblers' from t telephone calls -assuming improbably that we have conquered all the problems generating false responses discussed in previous sections -is the odds against getting it right.The third and fourth rows of data in table 6 give the results for each of the Victorian prevalence surveys.None of the probabilities of getting it right are auspicious, and the odds against are high (if we have pretensions to accuracy and are not merely after a 'good thing' to punt on).
Note that probabilities fall the larger are the sample sizes.This is because s increases in number.However, the probability of being quite wide of the mark is much higher with smaller values of s.This can be seen in chart 2, which plots the probabilities of identifying values of s, assuming a population prevalence of 1 per cent, for samples sizes of t = 500 (s = 5), t = 2,000 (s = 20) and t = 4,000 (s = 40).
Let us say that the values of s we estimate by survey are ± 20 per cent wide of the actual mark (the peak of each curve).Note how the probabilities of being 20 per cent out, as shown roughly by the grey shaded areas, drop by greater proportions in the larger samples, which is to say that the likelihood of having a relatively larger error is greater in smaller samples.Note also, however, that in all cases with small prevalence rates the odds of getting it right or near-right are small.Consequently the prevalence rates estimated are unstable within any survey, which is to say that they could almost as easily be one figure or another, and from survey to survey.This is another reason that it was illicit for the Government to have made such a direct, unguarded comparison with prevalence rates arising from earlier studies.Chart 2. Probabilities of identifying 's' given different sample sizes 't' with p = 0.01 Unfortunately too little attention was paid to the Productivity Commission's warnings, and much more weight was given to the numbers than they could reasonably be expected to carry.Therefore, even if we ignore completely that only 30-40 per cent of 'problem gamblers' might respond truthfully to relevant survey questions, we cannot put faith in the numbers because of the low probability problem.Yet this is not the only reason to doubt the statistical efficacy of the reported results.Our second motive for challenging the statistical soundness of the reported prevalence rates -also to do with their small size -is that the various studies do not measure confidence intervals as accurately as they could do.

Vol 2, No 1
The usual formula for calculating the 95 per cent confidence interval is given in equation ( 2) below.The symbol SE(P) is the standard error of the prevalence rate, t is the sample size or number of trials (as above) and P is the percentage prevalence rate that we have estimated.The symbol p, again, is the prevalence rate in percentage form.The confidence limits and intervals in table 5 derive from this formula: P (100 -P) p limits = SE(P) ± 1.96 -----(2) t However, while applying this 'formula to risk [or prevalence] data to obtain confidence limits is straightforward, … the approach is useful only as a large number approximation' (Rothman 2002, p. 132).In particular, for prevalence rates 'that are considerably less than (or greater than 50%), the confidence limits are apt to be inaccurate'.Instead we should use the formula for Wilson's confidence limits for a binomial, which 'comes close to exact limits for a binomial distribution ' (2002, p. 132).Using the same symbols as above, the somewhat forbidding formula is: t s 1.96 2 s (t -s) 1.96 2 p limits = -----+ --± 1.96 ----+ --(3) t + 1.96 2 t 2t t 3 4t 2 As would an exact binomial distribution, Wilson's formula produces an asymmetrical distribution of confidence limits for small prevalence rates.The reason is that the number on the lower end of the interval squeezes up against zero.Table 7 offers confidence limits and intervals calculated using formula (3) with which to compare the limits and intervals reported in the seven Victorian prevalence surveys (calculated with the usual formula (2)).Each of the lower limits increases, as does each of the upper limits.In other words more accurate methods calculate higher prevalence rates.(1997)(1998)(1999)(2000) Table 7.Comparison of reported data using Wilson's confidence limits (intervals) for a binomial Now, I do not want to flog a dead horse, as it were, but it bears repeating that even the 'more accurate' results in table 7 are thoroughly inaccurate.If we can rely in any way on the Productivity Commission's reported responses from 401 clients of gambling counselling agencies (1999, p. 6.36), the data here underestimate prevalence by a factor of between 2.5 and 3.3 to one.That is, 'true' prevalence is more likely to be in the 2.5 to 7 per cent range.

Table 1 contains
, to the best of our knowledge, some of the relevant data from all contemporary 'problem-gambling' prevalence surveys in Victoria.Readers will note that the Victorian Government might have picked four alternative prevalence rates with which to compare the 2003 Victorian longitudinal community attitudes survey rate of 1.12 per cent.All had larger effective sample sizes than the Productivity Commission's Victorian sample, upon which they chose to focus.All alternatives administered the SOGS to more people.Yet the Government chose for comparison the largest rate (2.14 per cent), the smallest effective sample (725) and the lowest number of people to whom callers administered the SOGS (254): again a highly questionable use of research findings.6 Of those problem gamblers who would participate in a survey prior to seeking help, only 38 per cent believed they would answer honestly.Some 45 per cent said that they would hide their problem to some degree, and 17 per cent did not know what they would have done (table 6.9).Only 0.3 per cent said they would have exaggerated their problems.Yet the original validation exercise for the SOGS did not take into account the likely strategic behaviour by problem gamblers when answering questionnaires of this type (because it took a group of self-confessed problem gamblers in a clinical setting, rather than problem gamblers outside this setting).
most particularly, people may provide dishonest or distorted answers to questions, especially if they feel that they are engaging in stigmatised behaviour.The Commission has been told by problem gamblers that, prior to seeking help from a counsellor, Vol 2, No 1 they would not have honestly disclosed their problem.'Basedonresponses of 401 clients of counselling agencies.The survey asked problem gamblers seeking help from specialist gambling agencies whether they would have participated in a survey prior to seeking help, and whether they would have revealed the true nature of their problems.Source: PC Survey of Clients of Counselling Agencies.'(ProductivityCommission 1999, p. 6.36)

Table 2 .
Productivity Commission table 6.9 'Do genuine problem gamblers reveal they have a problem?' table 3 below.

Table 3 .
Problem-gambling prevalence rates in Victoria 1996-2003 adjusted for dishonesty and distortion*

Table 4 .
Under-reporting in the Household expenditure survey of Victorian poker-machine losses ) below stands for the probability of success in identifying the 'correct' number of 'problem gamblers' (SOGS5+) from a sample of trials (telephone calls in this instance) numbering t.The symbol p stands for the actual proportion of 'problem gamblers' in the population, and this tells us what the 'correct' value of s should be in a sample of size t.The expression in brackets represents the possible number of combinations of s within t.The formula is: