Answers to Questions About Iraq Mortality Surveys

Timing of Lancet Iraq Mortality Studies

At no time did study authors Les Roberts or Gilbert Burnham say that the release of their mortality studies was timed to affect the outcome of elections. Roberts indicated that he wanted to protect the Iraqi co-authors by avoiding the perception in Iraq that the researchers withheld information until after the U.S. election. Burnham told National Journal reporter Neil Munro specifically that he was anxious that the 2006 study be released well before the election to dispel any notion of trying to influence outcomes.

Planning for the second survey began in October 2005 with the intention of completing and releasing the findings in the spring. However, the violence in Iraq was so great that it prohibited the field teams from beginning the survey until late spring. The data were released when ready. 

Questions About Riyadh Lafta’s Credentials

Riyadh Lafta, a co-author of the 2004 and 2006 Lancet mortality surveys who supervised the research teams in Iraq, has a long record as a solid partner for international research studies. His previous research includes investigations of childhood cancers, the uptake of uranium into children’s teeth, and the safety of physicians in Iraq. In late 2004, when the World Health Organization feared that there was a polio outbreak in Iraq, a disease that after billions spent has almost been eradicated under United Nations leadership, Lafta was chosen to investigate and guide the United Nations on improving polio surveillance.

Lafta asked that he not be contacted by the media out of concern for his safety and that of his family, a not unreasonable request in Iraq where doctors and academics are major assassination targets. Serious questions about study details have been relayed to him and matters resolved.

While Lafta has had a successful academic career, he did not have a career in Iraq’s Ministry of Health under Saddam Hussein’s rule, largely because he was one of a small number of doctors who refused to join the Baath party.

Use of Trained Iraqi Survey Teams

The collection of data by locally trained and supervised teams is standard for international surveys.

The Johns Hopkins data on reduction of deaths in Afghanistan, quoted both by the National Journal and President Bush, were collected in the same way—using cluster surveys managed by skilled local public health staff.

Research Protocols

The Bloomberg School of Public Health is satisfied that Burnham and his research team conducted their studies in an ethical manner and in compliance with the Bloomberg School’s policies and procedures.

In the ethical review process conducted with the School’s Institutional Review Board (IRB), Burnham and Roberts indicated that they would not record unique identifiers, such as full names, street addresses or any data (including details from death certificates) that might identify the subjects surveyed and put them at risk.

Children were not a part of the study. Onlookers—both adults and children—were told of the purpose of the project by the surveyors and asked to inform their neighbors, a common practice used by study investigators throughout the world including the U.S. Census Bureau. Since most households were located within walled compounds, conducting interviews on the doorstep was judged to be the best survey techniques from the point of security and cultural acceptability.

Release of Data

Release of mortality data were made available to academic and scientific groups in April 2007 as was planned from the inception of the study.

The researchers will not be releasing data at the household, street or neighborhood levels, which could identify and put study participants at risk.

Corroborating Evidence

Four population based studies have now shown a consistent pattern of mortality in Iraq.

A survey done by FAFO (Norway) and the Iraqi government over the first 13 months of conflict estimated 36,000 deaths, which was virtually identical to the 2004 and 2006 Lancet mortality estimates for the same period. FAFO’s crude mortality rate of 8.4/1000 years was also similar to pre-invasion 5.0 and post-invasion 7.9/1000 years rates reported in 2004 Lancet study, as was the rate of mortality for children under age 5. The original FAFO survey, which was completed by Iraq officials, was conducted twice since the first survey revealed an implausibly low death rate. The second survey revealed twice as many child deaths when researchers revisited the same households asking just about deaths in children.

All existing data suggest more people have been killed than seriously wounded in the Iraq conflict. A March 2007 poll by the BBC suggested that 17 percent of households had someone killed or wounded while a poll by the opinion firm ORB in September 2007 suggested 22 percent of households had someone killed. Both findings are consistent with the Lancet findings. ORB also estimated that 1.2 million Iraqis had died as of late 2007.

In May of 2007, the World Health Organization convened a meeting in Geneva to discuss the issue of mortality in Iraq. Three reviewers, including an author of a January 9, 2008, New England Journal of Medicine study on civilian deaths reanalyzed the Lancet data as part of their critique.  Riyadh Lafta attended to answer questions about the field activities. While many scenarios were proposed (regarding internal migration, mass exodus from the country, the possibility of excluding specific clusters) that could affect the point estimate produced, no reviewer found reason to discard or dismiss the Lancet findings. Moreover, the Lancet wrote an editorial with the release of the study emphasizing how this article had gone through a particularly rigorous peer review.

Cluster Sampling and “Main Street” Bias

Cluster sampling is a random sampling method. Sampling for the 2006 study was designed to give all households in Iraq an equal chance of being included. In this multistage cluster sampling, random selections were made at several levels ending with the “start” house being randomly chosen. From there, the house with the nearest front door was sampled until 39 consecutive houses were selected. This usually involved a chain of houses extending into two or three adjacent streets. Using two teams of two persons each, 40 houses could be surveyed in one day. Of the 47 clusters, 13 to 28 percent were rural, approximating the UN estimates for the rural population of Iraq.

In no place does the 2006 Lancet paper say that the survey team avoided small back alleys.

The National Journal suggests that the 2006 study ended on July 1 of that year and that a cluster with many deaths was added after that date to inflate figures. In reality, the study took many weeks to complete. There was no completion date set for the survey; it was finished when the last cluster was surveyed in July. Not surprisingly some clusters visited early in the survey had high mortality figures as did some visited toward the end of the clusters.

“Data-Heaping”

The National Journal makes repeated claims of alleged “data-heaping” (frequent recording of certain values) in the 2006 mortality study. As proof, the authors cite Michael Spagat, an economics professor who was not given the data to review. The article states, “Spagat noticed that the 23 missing certificates for nonviolent deaths were distributed throughout eight of the 16 surveyed provinces, while all 22 missing certificates for violent deaths were inexplicably heaped in the single province of Nineveh.” There are numerous errors in this statement.

The assertion that “all 22 missing certificates for violent deaths” came from one cluster is wrong on two counts. First, there were 3 clusters in which survey interviewers either forgot or chose not to ask for death certificates out concern for their personal safety. In total, interviewers did not ask for or households did not provide a death certificate in about 100 reported deaths.

Epidemiologists know that data-heaping is likely to come from either respondents or interviewers. In general, it is not associated with fraud but with imprecision in the reporting or recording of data.  At the conclusion of an interview in which a death had been reported, the surveyors requested to view a death certificate. Expanding this would not be a safeguard against fraud.

There are multiple points of internal consistency that point to the solidity of the data collected by our team. Examples include the following:

• Critics of the 2004 Lancet study point out that the infant mortality rate (reported as 29/1000 live births before the invasion and 57/1000 after) was far lower than the assumed rate at the time. Since the 2004 Lancet study, the United Nations Development Programme (UNDP) estimated the infant mortality rate in Iraq to be 40/1000. UNICEF’s estimate from 2001 to 2006 was 34/1000. Both estimates are extremely consistent with the unexpected finding of the 2004 Lancet study.

• The death rates over time between the UNDP estimates and the 2004 and 2006 Lancet studies are virtually identical for the periods where they overlap.

• A September 2007 survey conducted by the British polling firm, ORB, estimated 1.2 million deaths in Iraq. This finding is also consistent with the 2006 Lancet trend that mortality was increasing through 2006.

• Most deaths in the Lancet studies were confirmed with death certificates.

• The violent death rates recorded in the 2004 and 2006 surveys showed Baghdad had a violent death rate similar to the national average in spite of the impression in Iraq and the Western media that Baghdad was more violent. The Pentagon data on attacks (not deaths) in Iraq show a similar pattern.

Findings of the 2004 Lancet survey reported more deaths in the clusters in Fallujah in August and September of 2004 than during the siege, which occurred in April. Accounts from veterans who served in Fallujah in 2004 indicate that it is likely more people died in August and September because coalition forced fired more artillery into the city during that time. This finding was very inconsistent with images provided by the press and among people in Baghdad.

Likewise, the 2006 data suggested that over the preceding years, violence was greater in Diala province than in Anbar province in spite of the lack of press coverage in the former. An examination of data from the Pentagon on attacks by province showed a similar pattern. These findings strongly suggest that the data could not have been fabricated.

Funding Sources

Funding for the 2004 mortality study was provided by the Johns Hopkins Center for Refugee and Disaster Response and the Small Arms Survey in Geneva, Switzerland. Funding for the 2006 study was provided by MIT.

MIT played no role in the study design, implementation, analysis or writing of the Lancet report. The fact that some of MIT’s financial support in 2006 came from the Open Society Institute had no effect on studies, as the researchers knew nothing of funding origins.

Iraq Body Count Data

The 2006 Lancet study suggests that mortality more than doubled during the conflict due to violence. The Iraq Body Count (IBC) data, which the National Journal cited as more valid, suggest that the death rate increased by less than 10 percent during the occupation. Data from the Baghdad morgue, Najaf graveyards, and data compiled from other graveyards across Iraq consistently suggest more than a doubling of mortality.

It is important to note the IBC uses a passive method to collect data. The surveys conducted by the Johns Hopkins researchers, ORB, and others are an active surveillance method to determine deaths in Iraq.

A review of Iraq deaths reported by 4 major U.S. newspapers found that IBC missed more than 1 of every 10 deaths reported by the news media. The separate and soon to be published study from Columbia University researchers also found that the majority of violent deaths in a phone sample from Baghdad were not recorded by IBC. (Siegler A. et al. has been accepted by the journal Prehospital and Disaster Medicine. It is slated for publication in the September 2008.)

Survey Participation Rate

The National Journal suggests that the strong response rate in the Iraq mortality surveys is implausible. In 2004, 7.5 percent of households surveyed were either unoccupied or refused to participate. The number was 1.7 percent in 2006. However, the response rate is typical for surveys conducted during conflicts. In fact, a 1999 survey conducted in Kosovo had no refusals out of 1,200 participants. The low refusal rate in Iraq is likely because many people are staying home for physical safety. A new report on mortality, soon to be released by the Iraqi government, found very similar absentee and refusal rates.

Other Resources

Response to Jan. 4, 2008 National Journal articles

Response to the finding of a Sept. 2007 poll of Iraq mortality conducted by ORB Group

Response to Oct. 18, 2006 Wall Street Journal opinion article by Steven E. Moore