Cochrane Review: Helmets for preventing head and facial injuries in bicyclists

Thompson RS, Rivara FP & Thompson DC. Cochrane Database of Systematic Reviews, The Cochrane Library.

Full source document (Note: Access to the Cochrane database may be denied from some countries)
Authors' abstract

This comprehensive critique was submitted to the Cochrane Review by statistician Dorothy Robinson.
Some, but not all, of the points raised were subsequently added to the published review.

For other information about this paper, see here.

Evidence from Cohort vs Helmet Law Studies

The reason for studying helmet efficacy is to predict changes in injury rates when a population of cyclists all start to wear helmets. Helmets can only be considered effective if the increased helmet wearing actually reduces injuries.

The authors of this Review discuss ways of evaluating helmet efficacy under sub heading "Background", stating that "Randomized control trials are neither feasible nor ethical towards this end. Cohort studies are not practical because of the large population required for follow-up"

Though cohort studies are not practicable, consequences of helmet laws have been studied in countries such as Australia where helmet wearing rates increased from 26% - 38% pre-law to 75% - 85% post-law. Despite these extremely large increases in helmet wearing, virtually no change was detected in head injury rates over and above the prevailing trend for other road users.

As evidence for this, a paper is enclosed containing tables and graphs of head injury rates before and after helmet laws in most Australian States. The laws in Australia were introduced at different times in different States over a two year period. In some States, such as Victoria, bicycle helmet laws coincided with the introduction of other road safety measures including speed cameras and increased random breath testing. In others, eg Western Australia (WA), there were no coincident road safety initiatives. Such data enable comparison of the efficacy of helmet laws with other road safety measures.

It is well known that most case-control studies of bicycle helmets have found substantial differences between those who chose to wear helmets and those who do not. However, the differences relate not just to head injury rates, but also many other attributes of the two groups. In Seattle, an observational study showed children wearing helmets were much more often white than black or other races, and riding in parks or bicycle paths than on city streets (DiGuiseppi, Rivara, Koepsell and Polissar, 1989). Two of the case-control studies used for the Cochrane Review found helmeted riders were less likely to be involved in vehicle/bike collisions - 12.7% of helmeted riders vs 18.0% of non helmeted riders in the study by Thompson, Rivara and Thompson, 1996; 31.6% of helmet wearers in the study by Maimaris, Summers, Browning and Palmer, 1994 compared with 43.4% for non-wearers. McDermott, Lane, Brazenore and Debney, 1993 did not report motor vehicle involvement, but found little difference in percentages with heads or helmets hitting a moving motor vehicle (16.9% vs 14.8%). The remaining two studies in the Cochrane Review do not provide enough information to compare motor vehicle involvement in wearers and non-wearers.

Several studies, eg Maimaris, Summers, Browning and Palmer, 1994 and Thomas et al, 1994, show head injury is 3 to 5 times more likely in cyclists involved in accidents with motor vehicles. The risk of head injury also varies considerably with age (Robinson, 1996), as do helmet wearing rates. Fundamental differences between populations choosing to wear or not to wear helmets make it difficult for any case-control study to separate these effects.

In contrast, a study comparing hospital admissions for all cyclists before and after a helmet law, compares the pre-law population of non-wearers with the same population of wearers post-law. Apart from those who give up, essentially the same group of cyclists is being compared. This means there should be less difficulty adjusting for bias, making the results more reliable.

Yet these latter studies show little or no benefit of increased helmet wearing from the laws, contradicting the findings of the case-control studies, perhaps due to difficulties in adjusting for fundamental differences between those choosing to wear helmets and those choosing otherwise. Another potential problems is risk compensation. Cyclists choosing to wear helmets may feel protected and so take more risks. If increased risk taking leads to more accidents, helmet wearers may have increased, rather than decreased total risk relative to the amount of cycling, even if the risk of head injury in any particular accident is decreased. Should helmet wearing be recommended if this happens?

These facts are relevant, interesting and should be an essential, if not the most important part of any Review by the Cochrane Collaboration. What would be the point in promoting helmets, if increased helmet wearing does not result in reduced injuries? Unfortunately, this Review fails to discuss this.

Several references are cited by this Review (Vulcan, Cameron and Watson, 1992; Carr, Dyte and Cameron, 1995; Pitt et al, 1994; Ekman, Schelp, Welander and Svanstrom, 1997; Rivara, Thompson, Patterson and Thompson, 1998) as providing evidence of helmet effectiveness from time series analysis before and after helmet laws. However, examination of these references shows that the evidence for this comment is very shaky, if non existent. When analysing the first three years of post helmet law data in the State of Victoria, Cameron, Newstead, Vulcan and Finch, 1994 reported that results showed "the observed proportion of head injured cases to be no different from the downward trend predicted by the model using pre-law wearing rates." A year later, Carr, Dyte and Cameron, 1995 found that numbers of head injuries had been significantly affected by the law in Victoria, but that an analysis of head injuries alone could not determine whether this was due to reductions in numbers of cyclists, improvements in road safety, or helmets. In fact, if numbers of head and non-head injuries are shown together on the same graph it is clear that non-head injuries show a similar decline to head injuries, therefore the main effect was not helmets, but other factors, such as safer roads or reduced cycling activity. These data on hospital admissions for head and other injuries in Victoria (and WA and NSW and SA and Qld) following laws which forced literally millions of cyclists to wear helmets directly contradict the case-control studies. It would be difficult if not impossible for helmets to be as effective as claimed by the Review report, yet evaluations of helmet laws be so inconclusive. The contradictions can only arise from problems in the scientific methodology such as difficulties in adjusting correctly for differences between groups choosing whether or not to wear helmets.

For Queensland, Pitt et al, 1994 reported that: "The decrease in bicycle related head injuries started before helmets became widely used and occurred against a background of unchanged admission rates for other bicycle related injuries and a decrease in head injuries due to other causes…. Our findings indicate that the reason for the decrease in bicycle related head injuries is more complex than just increased wearing of helmets."

Ekman, Schelp, Welander and Svanstrom, 1997 found, in Skaraborg County, Sweden, where helmet wearing programs resulted in a substantial increase in wearing rates, head injuries in children under 15 fell by 59% and non-head injuries 48%. For Sweden as a whole, child cyclist head injuries fell by 43% vs 32% for non-head injuries. In Sweden as a whole, head injuries fell to 84% of non-head injuries (100-43)/(100-32). For Skaraborg, the same calculation gives (100- 59)/(100-48) ie 79%. Thus as a proportion of total injuries, the fall for head injuries was little different to that for Sweden as a whole. (79% vs 84%, a difference of only 5 percentage points). A much bigger difference is seen in the fall in non-head injuries in Skaraborg vs Sweden (100-48)/(100-32) = 76%, a difference of 24%, suggesting that the helmet wearing programs had other, larger effects which reduced numbers of non-head injuries, such safer conditions or reduced cycling.

For Seattle, Rivara et al, 1994 reported that an increase in helmet use among school age children from 5.5% in 1987 to 40.2% in 1992 was accompanied by a 66.6% decrease in bicycle related head injuries in 5 to 9 year olds and a 67.6% decrease in 10 to 14 year old members of a health maintenance organisation. However, unless those wearing helmets are many times more likely to seek hospital treatment, a relatively small increase in helmet wearing (5.5% to 40%) cannot plausibly produce a two thirds reduction in head injuries. This would not be possible, even in helmets prevented all head injuries, let alone given that, for helmet wearers seeking emergency room treatment in Seattle, 20% of helmet wearers under 6 years, 12% of helmet wearers aged 6-12, and 13% of helmet wearer aged 13- 19 were treated for head injury (Thompson, Rivara and Thompson, 1996). Other factors such as trends or increased accident risk for helmet wearers must therefore be present in the data studied by Thompson, Rivara and Thompson, 1996, potentially invalidating their conclusions.

Closer consideration of the five cited reports shows, therefore, that they provide no plausible evidence for the effect of helmets from time series analysis as claimed by the authors of this Review Report. Another major time series analysis, Scuffham and Langley, 1997, not mentioned by this Review, observed a downward trend in the percentage of cyclists suffering head injuries "present before, and independent of, helmet wearing." After accounting for this trend, increased helmet wearing had "little association with serious head injuries as a percentage of all serious injuries to cyclists." Such trends are a confounding factor which the reviewed case-control studies appear not to have estimated or adjusted for. This leaves the results of such case-control studies open to question. After correctly adjusting for such trends, no published time series analysis has been able to show any noticeable decrease in head injury rates, despite the large increases in helmet wearing because of helmet laws or substantial promotion of helmets.

If, in reality, no benefit can be shown from such measures, but other road safety campaigns such as speed cameras and targeting drink-driving produce large and highly noticeable reductions in the death and injury toll for all road users (Powles and Gifford, 1993) why not spend the generally limited funds for road safety on the measures shown to be highly effective?

Risk Compensation

The differing helmet wearing rates in one of the studies in this Review could be interpreted as evidence that helmet wearing increases injury rates. In Seattle, in May and September 1987, street surveys found helmet wearing rates of only 3.1% and 3.3% in samples of 1957 and 2544 child cyclists aged 5-14 (DiGuiseppi, Rivara, Koepsell and Polissar, 1989). Survey sites were stratified according to census data on income and number of children residing in each area, in an attempt to provide an unbiased estimate of the helmet wearing rate. Coincidentally, the first study of helmet efficacy (Thompson, Rivara and Thompson, 1989) in the Cochrane Review took place around the same time (1 December 1986 to November 30 1987) at the five major teaching hospitals in Seattle. 15 out of 345, ie 4.3% of cyclists 14 years and younger whorequired treatment at the emergency department were wearing helmets (3 out of 145 = 2.1% of those with head injury). For this age group, helmet wearing in the community control of Thompson, Rivara and Thompson, 1989, consisting of cyclists who reported falling off their bikes, was 21.1%. Thus there was no significant difference between helmet wearing rates in the survey of child cyclists riding round the streets of Seattle and those requiring emergency room treatment for head injury. The largest and most significant difference was, in fact, between helmet wearing in children seen riding around Seattle and those in the community control group who fell off their bikes.

Thus one might conclude that helmet wearing in children is not significantly associated with reducing head injury, but increased risk of falling off a bike. An alternative possibility is that the community controls differed in their helmet wearing because they were members of a Group Health Cooperative, which may have promoted helmet wearing to its members. Either way, the 7-fold difference in helmet wearing rates between the community controls in this study and in cyclists of the same age group riding on Seattle streets invalidates the conclusions of this study making it ineligible for inclusion in the Cochrane Review. Whilst use of emergency room controls gives a less biased result, helmet wearing in child cyclists requiring emergency room treatment was also higher than in the street survey, suggesting that, if case-control studies are to provide a valid estimate of the total effect of helmets (including any risk compensation), helmet wearing in the 'case' group of head injured cyclists should be compared not only to controls requiring emergency room treatment for other injuries, but to population wearing rates of non-injured cyclists, or controls chosen from street surveys of cyclists not involved in accidents.

Another interesting finding of some other surveys (eg Farris et al, 1997) is that helmet wearers often appear more likely to obey traffic laws and, at least in two of the studies used in the Cochrane Review, where helmet wearing was published by motor vehicle involvement, may also be less likely to be involved in bike/motor vehicle accidents. At first sight, this appears contradictory to the risk compensation hypothesis. However, careful or risk- averse cyclists may be more likely to wear helmets than the general population. Even if helmet wearing encourages such risk-averse cyclists to take more risks, this does not necessarily mean their overall risk will increase beyond the average of those whose chose not to wear helmets. Nonetheless, any change in risk will negate the benefits of helmets for that particular group. The same applies to cyclists required by law to wear helmets, where again increased risk taking may negate any potential benefits of helmets.

Studies in which a number of different treatments are applied to the same person usually have substantially increased sensitivity compared with studies which apply different treatments to different groups of people, because the within-subject variance is generally much lower than that between subjects. For this reason, monitoring changes in accident and head injury rates after helmet laws (where non-helmeted cyclists pre-law are effectively compared with the same group of cyclists now wearing helmets because of the law) should provide a better and more realistic effect of the true effect helmet wearing has on accident and head injury rates.

Misleading Interpretation of Odds Ratios

As well as inadequate discussion of time series analyses and why little or no benefits have been found following helmet laws which required millions of cyclists in Australia to wear helmets, this Review appears to confuse odds ratios with the percentages of head injuries which might be prevented by helmets. This is a serious problem, potentially leading to significant errors of interpretation.

For example, imagine a hypothetical helmet which could prevent 50% of head injuries and a situation where 100 bareheaded cyclists had accidents, resulting in 40% of them suffering head injuries. Imagine a case-control study where 100 cyclists wearing this helmet had identical accidents to the 100 bareheaded cyclists. We would therefore expect on average 50% of 40 = 20 of these cyclists to suffer head injuries.

To calculate the odds ratio, we compute Odds(helmeted) = (No of helmeted with HI)/(No of helmeted without HI) and Odds(bareheaded) = (No of bareheaded with HI)/(No of bareheaded without HI)

The odds ratio is Odds(helmeted)/Odds(bareheaded) = 20/80 divided by 40/60 ie 0.375.

The authors of this Review have used this odds ratio interchangeably with the term "percentage reduction in head injuries" as if suggesting that odds of .375 means a 62.5% reduction in head injuries, not the 50% reduction it represented in this hypothetical example. Page 6 of this Review gives combined adjusted odds ratios from four of the studies of 0.31 for cyclists in bike/motor vehicle accidents and 0.32 for other crashes, which is translated, under the heading "Main results" into protection levels of 69% and 68%.

The fact that this can be confusing to other researchers is well illustrated by the first of the studies in the Cochrane Review. Thompson, Rivara and Thompson, 1989 found 17 out of 120 helmet wearers had head injury (14.2%), compared to 218/548 (39.8%) of non wearers. This leads to an estimate that helmets would prevent (1 - 14.2/39.8) = 64% of head injuries. The crude odds ratio was 0.25 and the adjusted odds ratio was 0.26. The summary of Thompson, Rivara and Thompson, 1989 did not present this figure, only the value obtained by comparison with the community control group of predominantly child cyclists who fell off their bikes. As already discussed, these children had an extraordinarilyhigh helmet wearing rate of 21.1%, compared to 3.2% in a stratified sample of 4501 children riding around Seattle but not falling off their bikes. The estimate, derived from the comparison with this exceptionally high helmet wearing group, was the only estimate of helmet efficacy presented in the summary of Thompson, Rivara and Thompson, 1989 "helmeted riders had an 85% reduction in their risk of head injury (odds ratio 0.15; CI 0.0-7 to 0.29)"

Sacks, Holmgreen, Smith and Sosin, 1991 interpreted the latter figures to mean that, out of 2985 head injury deaths to cyclists in the US over a 5 year period, up to 84% (2500) could be prevented if all cyclists wore helmets. An alternative estimate, based on the adjusted odds ratio of 0.26 was that 2148 (72% of total) could be prevented based. Again, Sacks, Holmgreen, Smith and Sosin, 1991 mistakenly believed that odds of 0.26 means that 74% of head injuries would be prevented, instead of the actual value of 63%, obtained by comparing injury rates in (potentially disparate) groups of helmeted and non- helmeted cyclists treated in the emergency room. In the Main Results section of their paper, Sacks, Holmgreen, Smith and Sosin, 1991 translated these figures into the claim that universal use of helmets by all bicyclists "could have prevented as many as 2500 deaths and 757 head injuries, ie one death every day and one head injury every 4 minutes."

The authors of this Review for the Cochrane Collaboration refer to the paper by Sacks, Holmgreen, Smith and Sosin, 1991, so obviously know of this misinterpretation of their results, but never appear to have attempted to correct it, or explained to the CDC why this is invalid. And they still continue to mislead other scientists by referring to odds ratios as percentage reductions in head injuries.

A final problem with this Review is that the results of McDermott, Lane, Brazenore and Debney, 1993, which happen to show the lowest benefit for helmets, are not used in the computation of odds ratios. This was claimed to be because no adjusted odds ratios were available. The authors of the Review state (p5) that "if the authors (McDermott, Lane, Brazenore and Debney, 1993) had adjusted for motor vehicle involvement the results would have indicated an even greater protective effect of helmet use." Yet there is little evidence in the paper to back up this claim. Table 2 from McDermott, Lane, Brazenore and Debney, 1993 shows 14.8% of non wearers hit their heads or faces on a moving motor vehicle, not significantly different from the 16.9% of approved helmet wearers. Any adjustment for motor vehicle involvement would be very minor. More interestingly, 50% of helmet wearers who hit their heads after collision with a motor vehicle had head injuries, compared to 71% of non helmet wearers. Clearly, if 50% of helmet wearers in who hit their heads after collision with a moving vehicle were head injured, it is implausible to suggest, as this Review does, that helmets normally prevent 69% of such head injuries. McDermott's study is a real live counter example which clearly contradicts this claim.

The post helmet-law results for Australia are also a counter-example to this claim. If helmets prevented 70% of head injuries, then in Western Australia, when helmet wearing increased from less than 39% to more than 80%, it is highly implausible there would be so little evidence for any real effect.

Conclusions and Recommendations

The Cochrane Collaboration has carried out some excellent and necessary work, addressing such questions as: "What evidence is there for the efficacy of spinal fusion?" or "After cardiac bypass surgery, is it beneficial to administer glutamate to reduce the risk of neurological complications such as stroke or seizure?" These questions can and should be answered by meta analysis of available randomised controlled trials. Results rely on the randomisation process in each trial to ensure unbiasedness. The overall findings are valid provided the evidence from the different trials is not contradictory. In data from different sources are contradictory, explanations are necessary for the different circumstances pertaining in each of the trials and the conclusions would most likely be, "In circumstances such as A, administering D is generally effective, but this is not the case for circumstances such as B".

For bicycle helmets, the main questions are likely to be, are they effective (and cost effective), and should we recommend or mandate that cyclists wear them? In this case, the act of recommending or mandating that helmets be worn may change the circumstances, eg by making cyclists feel safer and take more risks. The act of recommending or mandating helmets may also discourage people from cycling. These factors cannot be considered in isolation from the effect of helmets. They are part of the package that comes with mandating or recommending helmets.

For this reason, changes in head injury rates following helmet laws, or the results described by Scuffham and Langley, 1997 for increased helmet wearing following pre-law promotion are much more relevant than the case-control studies discussed in this Review, especially since the two sets of results contradict each other. The most likely explanations for this are the inherent problems of using self selected samples such as those choosing whether or not to wear a helmet with all that entails for differences in riding behaviour, riding styles and attitudes to risk, as well as potential for helmets to increase rotational injuries, the limited range of impact speeds for which helmets are designed, and failure to adjust for trends in the head injury rates evident in Scuffham's study, the Australian data and, it appears, also in Seattle (Rivara et al, 1994).

This Review completely fails to address these problems, nor even discusses why, in the first study in this Review (Thompson, Rivara and Thompson, 1989) helmet wearing by children in the community control sample of children was 21.1%,compared with 3.2% in a survey of child cyclists riding round the catchment area for these hospitals, nor whether this discrepancy means we should conclude that helmet wearers have a similar risk of head injury to the general population, but increased risk of falling off their bikes!

This Review also fails to discuss the effects of helmet laws and increased helmet wearing with adequate depth and apparently misquotes or misinterprets the papers cited on changes in head injuries rates after helmet laws or intensive promotion of helmets.

This Review also misleads the general reader by quoting odds ratios as percentage reductions in head injuries.

In short, this Review cannot be recommended as a valid interpretation of the existing published information on helmets.

An evaluation of helmet laws in Australia, drawing on the data from Victoria, New South Wales, South Australia, Queensland and Western Australia, where helmet laws were enacted between 1990 and 1992 has been produced. If the subject matter is considered appropriate for the Cochrane Collaboration, a new Review should be commissioned. The new review should incorporate results on head injuries following helmet laws, and discuss all aspects of helmets and helmet promotion, including possible effects of risk compensation resulting in increased accident rates, the effect of discouraging of cycling because of laws or increases in perceived danger following helmet promotion, and other effects such as potential increases in rotational injuries of the brain. It may be appropriate to invite comments from cycling groups and organisations, especially in Australia, because of their direct experience of the efficacy of helmet laws compared with other road safety initiatives. Costs and benefits of helmets in cars compared with bicycles should also be included in the review.

References

Cameron, Newstead, Vulcan and Finch, 1994

Cameron M, Newstead S, Vulcan P, Finch C, 1994. Effects of the compulsory bicycle helmet wearing law in Victoria during its first three years. Monash University Accident Research Centre .

Carr, Dyte and Cameron, 1995

Carr D, Dyte D, Cameron MH, 1995. Evaluation of the bicycle helmet wearing law in Victoria during its first four years. Monash University Accident Research Centre Report 76.

DiGuiseppi, Rivara, Koepsell and Polissar, 1989

DiGuiseppi CG, Rivara FP, Koepsell TD, Polissar L, 1989. Bicycle helmet use by children. Evaluation of a community-wide helmet campaign. J Amer Med Assoc 262:2256-61.

Ekman, Schelp, Welander and Svanstrom, 1997

Ekman R, Schelp L, Welander G, Svanstrom L, 1997. Can a combination of local, regional and national information substantially increase bicycle-helmet wearing and reduce injuries? Experiences from Sweden. Accident Analysis & Prevention 1997 May;29(3):321-8.

Farris et al, 1997

Farris C, Spaite DW, Criss EA, Valenzuela TD, Meislin HW, 1997. Observational evaluation of compliance with traffic regulations among helmeted and nonhelmeted bicyclists. Ann Emerg Med 1997 May;29(5):625-9.

Maimaris, Summers, Browning and Palmer, 1994

Maimaris C, Summers CL, Browning C, Palmer CR, 1994. Injury patterns in cyclists attending an accident and emergency department: a comparison of helmet wearers and non-wearers. BMJ 1994 Jun 11;308(6943):1537-40.

McDermott, Lane, Brazenore and Debney, 1993

McDermott FT, Lane JC, Brazenore GA, Debney EA, 1993. The effectiveness of bicycle helmets: a study of 1710 casualties. Journal of Trauma 1993; 34(6):834-845.

Pitt et al, 1994

Pitt WR, Thomas S, Nixon J, Clark R, Battistutta D, Acton C, 1994. Trends in head injuries among child bicyclists. BMJ 1994;308:177 (15 January) .

Powles and Gifford, 1993

Powles JW, Gifford S, 1993. Health of nations: lessons from Victoria, Australia. BMJ 1993 Jan 9;306(6870):125-7.

Rivara et al, 1994

Rivara FP, Thompson DC, Thompson RS, Rogers LW, Alexander B, Felix D, Bergman AB, 1994. The Seattle children's bicycle helmet campaign: changes in helmet use and head injury admissions. Pediatrics 1994 Apr;93(4):567-9.

Rivara, Thompson, Patterson and Thompson, 1998

Rivara FP, Thompson DC, Patterson MQ, Thompson RS, 1998. Prevention of bicycle related injuries: helmets, education and legislation. Annu.Rev.Public Health 1998. 19:293-318..

Robinson, 1996

Robinson DL, 1996. Head injuries and bicycle helmet laws. Accident Analysis & Prevention 1996 Jul;28(4):463-75.

Sacks, Holmgreen, Smith and Sosin, 1991

Sacks JJ, Holmgreen P, Smith SM, Sosin DM, 1991. Bicycle associated head injuries and deaths in the United States from 1984 through 1988: how many are preventable? . JAMA 1991;266:3016-3018.

Scuffham and Langley, 1997

Scuffham PA, Langley JD, 1997. Trends in cycle injury in New Zealand under voluntary helmet use. Accident Analysis and Prevention 1997 Jan;29(1):1-9.

Thomas et al, 1994

Thomas S, Acton CH, Nixon J, Battistutta D, Pitt WR, Clark R, 1994. Effectiveness of bicycle helmets in preventing head injury in children: case-control study. BMJ 1994;308:173-176.

Thompson, Rivara and Thompson, 1989

Thompson RS, Rivara FP, Thompson DC, 1989. A case control study of the effectiveness of bicycle safety helmets. New England Journal of Medicine 1989 v320 n21 p1361-7.

Thompson, Rivara and Thompson, 1996

Thompson DC, Rivara FP, Thompson RS., 1996. Effectiveness of bicycle safety helmets in preventing head injuries: a case-control study. JAMA 1996 Dec 25;276(24):1968-73.

Vulcan, Cameron and Watson, 1992

Vulcan AP, Cameron MH, Watson WL, 1992. Mandatory bicycle helmet use: experience in Victoria, Australia. World Journal of Surgery 1992;16:389-397.