Growing Use in U.S., EU of Systematic Review For Safety Analyses Targets Different Issues

Turn to the nation's most objective and informative daily environmental news resource to learn how the United States and key players around the world are responding to the environmental...

By Pat Rizzuto

Dec. 11 — Research and regulatory agencies in North America and Europe are increasing their use of systematic review, but they are applying the strategies to different targets.

European regulators are using systematic review strategies to assess the safety of pesticides, genetically modified organisms and other food-related concerns, Didier Verloo, a senior official from the European Food Safety Authority said at a recent workshop.

In the U.S., the National Toxicology Program has been developing systematic review protocols for its chemical assessments. The Environmental Protection Agency is adapting techniques for its Integrated Risk Information System (IRIS), and the Food and Drug Administration is exploring how its Center for Food Safety and Applied Nutrition could use systematic review.

Meanwhile, academic toxicologists are exploring the use of systematic review methods as a means to speed the validation of new types of toxicity tests.

The origins of systematic review in clinical treatment; drivers spurring its transfer from health care to toxicology; future directions and forthcoming resources were discussed by North American and European scientists Nov. 21 in a systematic review workshop hosted by the Johns Hopkins University Center for Alternatives to Animal Testing.

Burgeoning Interest

Interest in evidence-based toxicology generally and systematic review specifically has burgeoned in the last few years, Thomas Hartung, director of the Johns Hopkins center, told Bloomberg BNA after the workshop.

Yet North American and European decision-makers seem to be implementing systematic review in different ways, Hartung said.

In the U.S. particularly, the methodology is being applied to chemicals, he said. In Europe, systematic review is being used for food-related regulatory concerns, some of which involve chemicals, he said.

The European Chemicals Agency, EU member states and chemical manufacturers that must comply with REACH—Regulation No. 1907/2006 on the registration, evaluation, and authorization of chemicals—would benefit from using systematic review to sift through the large amount of data that regulation is generating, Hartung said.

Hartung founded the Evidence-Based Toxicology Collaboration in 2011. The organization, which has a European and a North American chapter, is working to translate the ideas and approaches used for evidence-based medicine into methods that toxicologists and risk assessors can use.

Companies with representatives on the collaboration's steering committees include BASF Corp., Exxon Mobil Corp., Procter & Gamble Co., Pfizer Inc. and Unilever.

The EPA, the FDA and several U.S. and European academic institutes also have representatives serving on the steering committees.

Part of Evidence-Based Approach

The primary tool of evidence-based medicine is systematic review, according to the collaboration's website.

Workshop speakers described systematic review as including steps such as:

• framing the questions analysts will address before an analysis is launched;

• selecting search terms, databases and other tools that information specialists will use to identify relevant studies;

• developing criteria that will be used to include or exclude studies;

• assessing the quality of and potential for bias (ability of a study to find the effect it purports to find) in selected studies; and

• analyzing and synthesizing the information.


Drivers of Trend

Hartung described many factors driving interest in systematic review.

These include the increasing diversity of chemical safety data, which can include genetic, cellular, animal and human studies, he said.

Specialization within toxicology, which leads to specialists not being familiar with the full range of information that may be needed in an assessment, is another factor, he said.

The increased awareness that different forms of bias can color the conclusions of toxicity studies also has contributed to a demand for a methodical way to pull together information and assess its quality and its potential for bias, Hartung said.

There also is a growing demand for diverse parties to be able to more clearly understand the reasoning behind conclusions reached about the risks of chemicals, he said.

Traditional Assessments

Traditionally, chemical assessments primarily have been written descriptions of toxicity data, Hartung said during the workshop.

The problems with many narrative reviews, he said, include unspecified research questions, unspecified criteria for literature searches and unspecified criteria to guide the selection of those studies to be included or excluded from the analysis.

Narrative reviews also tend not to assess the quality of studies they incorporate or the potential for bias in their selected studies, he said.


By contrast, research questions for systematic reviews are specific, and they are specified before the assessment begins, Hartung said.

Literature searches are conducted based on an explicit search strategy, and scientific studies are included or excluded based on explicit criteria, he said.

The quality of those studies and their risk of bias are evaluated, and the synthesis of the information reviewed is presented both qualitatively (in words) and quantitatively (numerically), Hartung said.

This rigorous process takes time, said Roberta Scherer, a Johns Hopkins University epidemiologist.

“The systematic approach takes a whole set of arguments off the table.”Kay Dickersin, Johns Hopkins Center for Clinical Trials

Health-care researchers who use evidence-based medicine techniques established by the Cochrane Collaboration may take three to six months just to formulate a clear, answerable question, said Scherer, associate director of the U.S. Cochrane Center.

The center is part of the international Cochrane Collaboration, a not-for-profit organization, that produces systematic reviews of health-care treatments to help doctors and other health-care providers know what therapeutic approach would work best for their patients.

Regulatory agencies such as the U.S. EPA and European Food Safety Authority (EFSA), and research institutes such as the National Toxicology Program, are adapting evidence-based medicine approaches for environmental health purposes.

Food Safety Staff to Be Briefed

“The systematic approach takes a whole set of arguments off the table,” Kay Dickersin, director of the Johns Hopkins Center for Clinical Trials.

Parties no longer argue, “I don't understand what you did,” Dickersin said.

The rigorous, criteria-based process also provides analytic teams with defensible arguments when they are challenged as to why one study or another wasn't used, said Suzanne Fitzpatrick, senior adviser for toxicology at the Food and Drug Administration's Center for Food Safety and Applied Nutrition.

At Fitzpatrick's request, Hartung and other Johns Hopkins specialists in evidence-based medicine and toxicology will present information about systematic review to FDA's food safety staff in February, Hartung told Bloomberg BNA after the workshop.

EFSA: Improved Data, Analyses Follow Guidance

The EFSA's Verloo described a benefit the food safety authority has observed in response to systematic review guidance it has issued.

Once regulators describe the information they want reported about how a study was conducted and establish criteria for high quality studies, the data they receive improves, Verloo said.

The food safety authority has implemented systematic review for environmental health analyses more than any other agency that participated in the workshop.

In 2010, the authority issued guidance, “Application of Systematic Review Methodology to Food and Feed Safety Assessments to Support Decision Making.” The guidance can be used to evaluate the risks of chemical additives in animal feed, among other food-related concerns.

In 2011, the authority issued guidance showing how systematic review principles should be applied for submissions of peer-reviewed literature required for approval of pesticide active chemicals, Verloo said. The guidance was needed to implement a provision of the EU's Plant Protection regulation, (EC) No. 1107/2009, he said.

Such guidance is improving the information applicants provide and the analyses carried out by member states and authority staff, he said.

Training, Outsourcing

The authority began a staff training program in 2011, Verloo said.

About 150 staff and other experts have received training in literature search techniques, and about 160 individuals working for contractors have been trained to produce evidence-based assessments, he said.

Other tasks the authority undertook to make systematic procedures routine included obtaining and learning how to use software, called DistillerSR, to enter the information needed for systematic reviews and issuing contracts to outsource portions of its systematic review workload, Verloo said.

In 2012, the authority issued a report describing situations in which systematic reviews would be warranted and situations when they may not be needed or feasible.

These rigorous analyses take time and resources, so their use should be pursued thoughtfully, the report said.

“Important are the use of explicit systematic methods aimed at minimizing bias and maximizing transparency in order to produce the most reliable findings that can be used to inform decision making,” the report said.

“Systematic reviews should be a priority for controversial topics (which might be subject to greater scrutiny by external parties, including the public, and thereby would benefit from maximum transparency) or topics for which there was disagreement amongst experts,” the report said.

Systematic Reviews Not Always Feasible

The report discusses practical issues such as methods that can be used when resource limitations, time constraints or other situations preclude a full systematic review. It also describes the alternative methods' shortcomings.

To aid either systematic or quicker “less systematic” reviews, the EFSA has developed an inventory of databases that contain information relevant for the authority's analyses, Verloo said.

The authority has received 5 million euros ($6.2 million) to offer contracts for systematic review services from 2012 through 2016, Verloo said.

Notwithstanding EFSA's embrace of systematic review, Verloo and Scherer, with the U.S. Cochrane Center, stressed the analytic process is only one of many ways to feed information into a risk analysis.

“Systematic review is not an end in itself, but an approach to feeding information into a risk assessment,” Scherer said.

Implementation Challenges for IRIS

Vincent Cogliano, acting director of the EPA's IRIS program, said IRIS is doing a lot to implement systematic review. IRIS assessments evaluate the hazards chemicals pose and the doses at which those hazards manifest. Those are the first two steps of a risk assessment.

IRIS staff need to sort through many issues, Cogliano said.

A single IRIS assessment of one chemical may contain a dozen systematic reviews, Cogliano said. These various reviews, he said, include finding, assessing and then analyzing all the studies that would address whether a chemical presents hazards such as:

• causing cancer in humans,

• causing cancer in animals,

• causing neurotoxicity in humans,

• causing neurotoxicity in animals,

• causing reproductive toxicity in humans and

• causing reproductive toxicity in animals.


Mechanistic Studies Pose Special Challenge

Agency staff also are trying to understand how they can systematically review mechanistic information, Cogliano said. Studies on whether DNA is damaged by a chemical or how cells signal each other to undertake certain tasks are examples of mechanistic research that explores how a chemical affects the body.

For many chemicals, cellular or other in vitro studies will be the most available type of study, he said.

Yet the interpretation of these studies often is particularly contentious, Cogliano said.

The number of methods to aid the assessment of bias in certain human studies is growing, as is, to a lesser extent, the number of methods to do assess bias in animal studies, said Andrew Rooney, deputy director of the National Toxicology Program's Office of Health Assessment and Translation.

There are no methods, however, to assess bias in mechanistic studies, Rooney said. NTP is developing an approach for mechanistic studies, and it expects to circulate a draft for comment in 2015, he said.

On Dec. 10 a board of scientists that advises the program endorsed its intent to develop a 10-year contract for systematic review services that will assist four offices within the program.

Dose-Response Analysis

Cogliano said after the IRIS program has determined which health problems a chemical can cause, it conducts a series of systematic reviews to determine the exposure levels that cause each of those adverse health effects.

The use of systematic review to conduct dose-response analysis may eventually help the agency address a central challenge IRIS faces, he said.

Cogliano estimated that 20 percent of the criticism of IRIS assessments is targeted at staff's conclusions about the health hazards a chemical poses. About 80 percent of the challenges address the dose at which the agency estimates the harms could occur, he said.

Presenting an easier-to-understand rationale for the choices made in the dose-response analysis could help address those challenges, Cogliano said.

The IRIS program is developing a handbook to help its assessors understand how to carry out systematic reviews, Cogliano said. The IRIS handbook will be modeled after similar Cochrane Collaboration guidance and be consistent with similar guidance the National Toxicology Program is developing, he said.

Because systematic review methods are evolving for environmental health, the handbook will be a living document that will be updated online as new approaches are developed and peer reviewed, he said.

“IRIS has embraced and is acting to implement systematic review,” Cogliano said.

The program is mindful, however, of the time the review procedures can take and wants the implementation of these procedures to be adopted in a way that allows IRIS to continue to complete assessments so they can be used to make public health decisions, he said.

Jennifer McPartland, a health scientist at the Environmental Defense Fund, said parties representing many different perspectives on chemical risk issues see the value of systematic review principles.

The differences that may arise will be over the nuts and bolts of how to conduct such reviews, she said.

To illustrate such differences, McPartland pointed to perspectives voiced during the workshop.

Nancy Beck, a toxicologist and risk assessor with the American Chemistry Council, raised concerns about one approach used in some systematic review methods for environmental health studies. Because observational human studies—rather than carefully controlled clinical studies—are often the only ones available to environmental health researchers, that approach gives them a higher rating than do some other methods, she said. Yet observational human studies have many well-known limitations, she said.

“If you learn your study will be kicked out, you will learn how to do a better study.”Thomas Hartung,  Johns Hopkins University Center for Alternatives to Animal Testing

“Does a poor quality study design become important just because it is the only one you have? I would say no,” Beck said.

Fitzpatrick, with the FDA, said several observational studies may all point to a hazard.

“When is it enough evidence, even if it isn't a clinical study, that you have to tell the public something?”

Systematic Reviews for New Toxicity Tests

While the IRIS staff are working to adapt systematic review methods for the types of assessments the program conducts, Johns Hopkins scientists are exploring using the approach to evaluate new types of toxicity tests.

Martin Stephens, a senior research associate at the Center for Alternatives to Animal Testing, presented a pilot study he and colleagues are conducting with the zebrafish embryo toxicity test.

The embryonic zebrafish test allows researchers to test the effects chemicals have as an animal grows, while being small enough to fit into the 96-well plates commonly used for high-throughput testing systems.

Traditional validation methods take so many years and cost so much money that the Center for Alternatives to Animal Testing has decided to see whether a scientific-literature review process can help answer a central question, Stephens said.

The question is: How well does the embryonic zebrafish test predict the presence or absence of malformations known to occur in prenatally exposed rats and rabbits?

Forthcoming Resources

At the end of the workshop, Hartung described resources the Evidence-Based Toxicology Collaboration and Johns Hopkins are developing to assist in broader applications of systematic review.

The collaboration is developing guidance on toxicological systematic reviews that Hartung expects to be released in 2015.

The document will provide general guidance and serve as an introduction for interested parties, he said.

Hartung also is working with colleagues to develop instructional materials that will be used in a part-time course Johns Hopkins will offer for individuals interested in obtaining a masters in science, public health and toxicology.

The instructional materials will be available for free, most likely by 2016, for interested individuals who want the information but are not pursuing a degree, he told Bloomberg BNA.

The collaboration, meanwhile, will be exploring other ways to encourage broader uses of systematic review, Hartung said.

“I'm extremely optimistic this will help improve the quality of information agencies receive,” he told the audience.

“If you learn your study will be kicked out, you will learn how to do a better study,” Hartung said.

George Gray, director of George Washington University's Center for Risk Science and Public Health, said “I think it will improve both the quality of science and the reporting of it.”

To contact the reporter on this story: Pat Rizzuto in Washington at

To contact the editor responsible for this story: Larry Pearl at

The European Food Safety Authority's Application of Systematic Review Methodology to Food and Feed Safety Assessments to Support Decision Making is available at

EFSA's guidance on the Submission of Scientific Peer-Reviewed Open Literature for the Approval of Pesticide Active Substances is available at

EFSA's report on implementing systematic reviews is available at

Information about the National Toxicology Program's efforts to implement systematic review are available at

Information about the Evidence-Based Toxicology Collaboration is available at


Request Environment & Energy Report