Errors and omissions in Canadian research group's bicycle helmet papers

This article highlights the main findings of a paper by Kary
Read the full article here

In a series of related works (Macpherson, Parkin and To, 2001, Macpherson, 2003, Macpherson, 2005, Macpherson et al, 2006), Macpherson and co-workers asked whether Ontario child cyclist (5 to 14 years) ridership was reduced as a result of the province's bicycle helmet law. They answered that it was not, and went on to say their work suggests that "bicycling increases after a law, because of the positive publicity related to bicycling" (Macpherson, 2003b).

The conclusion was based on a series of surveys of child cyclists in the Borough of East York, Ontario, Canada, taken between 1990 and 2001. These surveys recorded: children cycling (rate per hour) at a variety of locations (on residential streets, in parks, etc); the sex and likely social group of the children; and the rate of cycle helmet use.

Macpherson et al’s work has been influential. For instance, when the British Medical Association moved to support all-ages helmet legislation in 2005, Macpherson et al’s work was the only evidence cited in support of a claim that such legislation would not suppress cycling levels.

At the time (2005), Macpherson had not released the results of the follow-up survey in 2001 showing that helmet wearing had returned to pre-law levels. Had this been known – that non-enforced laws have no long-term effect on helmet wearing – the BMA might have been more sceptical about the claimed benefits, and so less supportive of legislation.

Full analysis of the relevant data shows Macpherson et al's conclusion – that Ontario’s law did not reduce cycling – to be incorrect. Instead the per-capita bicycle use by 5 to 14 year olds fell substantially in the years of strong helmet promotion and subsequent legislation. There are serious errors and omissions in the data presented in the published papers.

No account was taken of population growth

in East York during the years (1990-2001) in which the ridership surveys were carried out. While the authors assumed the population remained static, in fact census data show that the child population increased by about 50%. When this trend is taken in to account, it becomes clear that per-capita ridership by children fell by about one third between 1993 and 1999 (although the data are quite erratic and subject to an outlier datum for the year 1999 – see below).

For details of the different published versions of the East York ridership data, see full paper

The resurgence of cycling in 1999 is considered further below.

Note that the full decline of ridership due to helmet programmes was almost certainly much greater than this. East York was something of a laboratory for cycle helmet promotion. Significant promotional programmes began before 1993. See next point.

Full data for 1990-1992 have never been released

These omitted ridership data are crucial. They would demonstrate whether or not the study group's early helmet promotion efforts in East York during these years— comprehensively targeting East York children and parents alike with the message that helmets must be worn, because bicycling is otherwise unavoidably dangerous— had already deterred ridership. As it stands, such data as are available strongly suggest that by 1993 a steep downward trend in ridership was already established.

There are large discrepancies in rider count data as presented in various papers or even the same paper

These discrepancies are in some cases substantial, as the following table shows:

Observed Numbers of Child Cyclists, East York, Ontario, 1990-1997, 1999, 2001
One study, same events, as differently reported by:

Year	Parkin 1993 Parkin 1995	Parkin 2003	Macpherson 2001 Macpherson 2003(table 6)	Macpherson 2003 (table 7)	Macpherson 2003b	Macpherson 2005	Khambalia 2005
1990	1017	914
1991	1885	1879
1992	1861	1563
1993		984	1597
1994		1083	2355
1995		1227	763	1126		1126
1996		1202	1371	1217		1217
1997		916	1375	918		918
1999			747	1124	1128	1124
2001						614
All years							At least one year's count is 550 and at least one year's count is 1795

References: Parkin et al, 1993, Parkin et al, 1995
Parkin, Khambalia and Macarthur, 2003, Macpherson, Parkin and To, 2001
Macpherson, 2003, Macpherson, 2003b
Macpherson, 2005, Khambalia, Macarthur and Parkin, 2005

The authors have never supplied an adequate explanation for these discrepancies. The observation made in Point 1 above was based on the third column.

Outlier year 1999

The ridership value presented by Macpherson et al. for 1999 is a clear outlier. It is not explained in the published research that there were strong bicycle promotion programmes around that time. In 1998 (a non-survey year), the previously independent Borough of East York was incorporated into the megacity of Toronto. To go with that, the Cycling Ambassadors Programme was launched to boost cycling in East York and two of the other newly incorporated regions (Toronto, 2009). The programme was expanded in 1999 and 2000 but then suffered substantial budget cutbacks. This would have given a positive boost to cycling in 1999. Additionally, it is possible that the 1999 survey actually recorded particular mass events. A similar problem occurred in the Melbourne (Australia) survey of 1992 (Finch, Heiman and Neiger, 1993), when a bicycle rally passed through an observation area.

Insufficient power to detect an effect of the size of interest

The power of a study refers to how likely it is that a real effect of the size of interest will in fact be discovered. Macpherson et al. do not calculate the power of their ridership study, nor do they provide confidence intervals for their annual ridership estimates. Yet the survey to survey changes in those estimates (Macpherson, Parkin and To, 2001), beginning with 1993 to 1994, are sequentially -16%, -22%, +58%, -33%, +120%, -60%. The magnitudes of these changes are at least similar to and often far greater than the suppressive effects of helmet legislation found in other studies (typically -25% to -50%). This demonstrates the presence of confounding factors and/or randomness with larger effects than the effect that Macpherson et al. claim to be testing for. In other words, Macpherson et al.'s ridership study lacks the discriminatory power necessary to detect an effect of the size the authors claim does not exist.

Full critique

These errors and omissions (and a number of others) are detailed in the full paper available here.

References

Finch, Heiman and Neiger, 1993

Finch C, Heiman L, Neiger D, 1993. Bicycle Use and Helmet Wearing Rates in Melbourne, 1987 to 1992: the influence of the helmet wearing law. Monash University Accident Research Centre Report 45.

Khambalia, Macarthur and Parkin, 2005

Khambalia A, Macarthur C, Parkin PC, 2005. Peer and adult companion helmet use is associated with bicycle helmet use by children. Pediatrics 2005;116(4):939-942.

Macpherson et al, 2006

Macpherson AK, Macarthur C, To TM, Chipman ML, Wright JG, Parkin PC, 2006. Economic disparity in bicycle helmet use by children six years after the introduction of legislation. Injury Prevention 2006;12:231-235.

Macpherson, 2003

Macpherson AK, 2003. An evaluation of the effectiveness of bicycle helmet legislation (PhD thesis). Institute of Medical Sciences, University of Toronto .

Macpherson, 2003b

Macpherson AK, Parkin PC, To TM, . Reply to Robinson. Injury Prevention 2003;9:380-383.

Macpherson, 2005

Macpherson AK, 2005. An evaluation of the effectiveness of bicycle helmet legislation. University of Pittsburgh accessed 15 Dec 2008.

Macpherson, Parkin and To, 2001

Macpherson AK, Parkin PC, To TM, 2001. Mandatory helmet legislation and children's exposure to cycling. Injury Prevention Inj Prev 2001;7:228-230.

Parkin et al, 1993

Parkin PC, Spence LJ, Hu X, Kranz KE, Shortt LG, Wesson DE, 1993. Evaluation of a promotional strategy to increase bicycle helmet use in children. Pediatrics 1993 Apr;91(4):772-7.

Parkin et al, 1995

Parkin PC, Hu X, Spence LJ, Kranz KE, Shortt LG, Wesson DE, 1995. Evaluation of a subsidy program to increase bicycle helmet use by children of low-income families. Pediatrics 1995;96(2):283-287.

Parkin, Khambalia and Macarthur, 2003

Parkin PC, Khambalia A, Macarthur C, 2003. Influence of socioeconomic status on the effectiveness of bicycle helmet legislation for children: a prospective observational study. Pediatrics 2003;112:e192-e196.

Toronto, 2009

www.toronto.ca/cycling/ratsa/reports.htm, accessed 22 Jul 2009 but since withdrawn. .

Related News