FIGURE 3-2 The Society of Thoracic Surgeons evidence system model.
SOURCE: Derived from Ferguson, T. B., et al. 2000. The STS national database:The STS national database:
Current changes and challenges for the new millennium. Committee to establish a national database in cardiothoracic surgery, The Society of Thoracic Surgeons. The Annals of Thoracic Surgery 69(3):680-691.
database that is used for quality reporting, and, increasingly, for continu-ously analyzing operative issues and techniques (Figure 3-2). The STS model also allows randomized trials to be conducted within the database.
The most significant aspects of this model lie in its constantly evolving, continuously updated information base and its methods of engaging prac-titioners in this system by providing continuous education and feedback.
Many have assumed that we must wait on fully functional electronic health records (EHRs) for such a system to work. However, we need not wait for some putatively ideal EHR to emerge. Current EHRs have serious short-comings from the perspective of clinical researchers, since these records must be optimized for individual provider–patient transactions. Conse-quently, they are significantly suboptimal with respect to coded data with common vocabulary—an essential feature for the kind of clinical research enterprise we envision. This deficit severely hobbles researchers seeking to evaluate aggregated patient information in order to draw inferential conclusions about treatment effects or quality of care. While we await the
TAKING ADVANTAGE OF NEW TOOLS AND TECHNIQUES
Figure 3-3.eps
Integrated at
“enterprise level”
Disease Registries—Granular, Detailed
Primary
Care Mental Cancer
Health
Cardio-vascular Etc.
Health System A
Health System B
Etc.
Electronic Health Records Adaptable to all
FIGURE 3-3 Fundamental informatics infrastructure—matrix organizational structure.
resolution of issues regarding EHR functionality, the best approach will be to construct a matrix between the EHR and continuous professional-based registries (disease registries) that measure clinical interactions in a much more refined and structured fashion (Figure 3-3). Such a system would allow us to perform five or six times as many trials as can now be done for the same amount of money; even better, such trials would be more relevant to clinical practice. As part of our Clinical and Translational Sci-ences Award (CTSA) cooperative agreement with the National Institutes of Health (NIH), we are presently working on such a county-wide matrix in Durham County, North Carolina (Michener et al., 2008).
New Strategies for Incorporating Scientific Evidence into Clinical Practice New efficiencies can be gained through applying innovative informatics-based approaches to the broad pragmatic trials discussed above; however, we also must develop more creative methods of rapidly translating new scientific findings into early human studies. The basis for such POC clinical trials lies in applying an intervention to elucidate whether an intended bio-logical pathway is affected, while simultaneously monitoring for unantici-pated effects on unintended biological pathways (“off-target effects”). This process includes acquiring a preliminary indication of dose–response rela-tionships and of whether unintended pathways are also being perturbed (again, while providing a basic understanding of dose–response relation-ships). POC studies are performed to advance purely scientific
understand-ing or to inform a decision about whether to proceed to the next stage of clinical investigation. We used to limit ourselves by thinking that we could only perform POC studies in one institution at a time, but we now know that we can perform exactly the same trials, with the same standard operat-ing procedures and the same information systems in India and Soperat-ingapore, as well as in North Carolina. The basis for this broadened capability, as in pragmatic clinical trials, is the building of clinical research networks that enable common protocols, data structures, and sharing of information across institutions. This broadening of scope affords the ability to rethink the scale, both physical and temporal, for POC clinical trials. The wide variation in costs in these different environments also deserves careful con-sideration by U.S. researchers.
New Approaches to Old Problems: Conducting Pragmatic Clinical Trials When considering strategies for fostering innovation in clinical trials, several key points must be borne in mind. The most important is that there exists, particularly in the United States, an entrenched notion that each clinical trial, regardless of circumstances or aims, must be done under pre-cisely the same set of rules, usually codified in the form of standard oper-ating procedures (SOPs). Upon reflection, it is patently obvious that this is not (or should not be) the case; further, acting on this false assumption is impairing the overall efficiency of clinical trials. Instead, the conduct of trials should be tailored to the type of question asked by the trial, and to the circumstances of practice and patient enrollment for which the trial will best be able to answer that question. We need to cultivate environments where creative thought about the pragmatic implementation of clinical trials is encouraged and rewarded (“envelopes of innovation”), and given the existing barriers to changes in trial conduct, financial incentives may be required in order to encourage researchers and clinicians to “break the mold” of entrenched attitudes and practices.
What is the definition of a high-quality clinical trial? It is one that provides a reliable answer to the question that the trial intended to answer.
Seeking “perfection” in excess of this goal creates enormous costs while at the same time paradoxically reducing the actual quality of the trial by distracting research staff from their primary mission. Obviously, in the con-text of a trial evaluating a new molecular entity or device for the first time in humans, there are compelling reasons to measure as much as possible about the subjects and their response to the intervention, account for all details, and ensure that the intensity of data collection is at a very high level.
Pragmatic clinical trials, however, require focused data collection in large numbers of subjects; they also take place in the clinical setting where their usual medical interactions are occurring, thereby limiting the scope of detail
TAKING ADVANTAGE OF NEW TOOLS AND TECHNIQUES
for the data that can be collected on each subject. To cite a modified Institute of Medicine definition of quality, “high quality with regard to procedural, recording and analytic errors is reached when the conclusion is no different than if all of these elements had been without error” (Davis, 1999).
Efficacy trials are designed to determine whether a technology (a drug, device, biologic, well-defined behavioral intervention, or decision support algorithm) has a beneficial effect in a specific clinical context. Such inves-tigation requires carefully controlled entry criteria and precise protocols for intervention. Comparisons are often made with a placebo or a less relevant comparator (these types of studies are not sufficiently informative for clinical decision making because they do not measure the balance of risk and benefit over a clinically relevant period of time). Efficacy trials—
which speak to the fundamental question, “can the treatment work?”—still require a relatively high level of rigor, because they are intended to establish the effect of an intervention on a specific end-point in a carefully selected population.
In contrast, pragmatic clinical trials determine the balance of risk and benefit in “real world” practice; i.e., “Should this intervention be used in practice compared with relevant alternatives?” (Tunis et al., 2003). The population of such a study is allowed to be “messy” in order to simulate the actual conditions of clinical practice; operational procedures for the trial are designed with these decisions in mind. The comparator is pertinent to choices that patients, doctors, and health systems will face, and outcomes typically are death, clinical events, or quality of life. Relative cost is impor-tant and the duration of follow-up must be relevant to the duration that will be recommended for the intervention in practice.
When considering pragmatic clinical trials, I would argue we actually do not want professional clinical trialists or outstanding practitioners in the field to dominate our pool of investigators. Rather, we want to incorporate real-world conditions by recruiting typical practitioners who practice the way they usually do, with an element of randomization added to the system to provide, at minimum, an inception time and a decision point from which to begin the comparison. A series of papers recently have been published that present a detailed summary of the principles of pragmatic clinical trials (Armitage et al., 2008; Baigent et al., 2008; Cook et al., 2008; Duley et al., 2008; Eisenstein et al., 2008; Granger et al., 2008; Yusuf et al., 2008).
The Importance of Finding Balance in Assessing Data Quality If we examine the quality of clinical trials from an evidence-based perspective we might emerge with a very different system (Yusuf, 2004).
We know, for example, that an on-site monitor almost never detects fraud, largely because if someone is clever enough to think they can get away with
fraud, that person is likely to be adroit at hiding the signs of their deception from inspectors. A better way to detect fraud is through statistical process control, performed from a central location. For example, a common indica-tor of fraudulent data is that the data appear to be “too perfect.” If data appear ideal in a clinical trial, they are unlikely to be valid: That is not the way that human beings behave. Table 3-1 summarizes monitoring methods to find error in clinical trials that take advantage of a complete perspective on the design, conduct, and analysis of trials.
Recent work sheds light on how to take advantage of natural units of practice (Mazor et al., 2007). It makes sense, for example, to randomize clusters of practices rather than individuals when a policy is being evalu-ated (versus treating an individual). Several studies that have followed this approach were conducted as embedded experiments within ongoing regis-tries; the capacity to feed information back immediately within the registry resulted in improvements in practice. Although the system is not perfect, there is no question that it makes possible the rapid improvement of prac-tice and allows us to perform trials and answer questions with randomiza-tion in that setting.
Disruptive Technologies and Resistance to Change
All this, however, suggests the question: If we are identifying more efficient ways to do clinical trials, why are they not being implemented?
The problem is embedded in the issue of disruptive technology—initiating a new way of doing a clinical trial is disruptive to the old way. Such disruption upsets an industry that has become oriented, both financially and philosophically, toward doing things in the accustomed manner. In less highly regulated areas of society, technologies develop in parallel and the “winners” are chosen by the marketplace. Such economic Darwinian TABLE 3-1 Taxonomy of Clinical Errors
Error Type Monitoring Method
Design error Peer review, regulatory review, trial committee oversight Procedural error Training and mentoring during site visits; simulation technology Recording error
Random Central statistical monitoring; focused site monitoring based on performance metrics
Fraud Central statistical monitoring; focused site monitoring based on unusual data patterns
Analytical error Peer review, trial committees, independent analysis
TAKING ADVANTAGE OF NEW TOOLS AND TECHNIQUES
selection causes companies that remain wedded to old methods to go out of business when their market is captured by an innovator who offers a disruptive technology that works better. In most markets, technology and organizational innovation drive cost and quality improvement. Providing protection for innovation that will allow those factors to play out natu-rally in the context of medical research might lead to improved research practices, thereby generating more high-quality evidence and, eventually, improving outcomes.
In our strictly regulated industry, however, regulators bear the mantle of authority, and the risk that applying new methods will result in lower quality is not easily tolerated. This in turn creates a decided barrier to innovation, given the extraordinarily high stakes. There is a question that is always raised in such discussions: If you do human trials less expen-sively and more efficiently, can you prove that you are not hurting patient safety?
What effect is all of this having? A major impact is cost: Many recent cardiovascular clinical outcomes trials have cost more than $350 million dollars to perform. In large part this expense reflects procedures and proto-cols that are essentially unnecessary and unproductive, but required none-theless according to the prevailing interpretation of regulations governing clinical trials by the pharmaceutical and device companies and the global regulatory community.
Costing out the consequences of the current regulatory regime can yield staggering results. As one small example, a drug already on the mar-ket evidenced a side effect that is commonly seen in the disease for which it is prescribed. The manufacturer believed that it was required to ship by overnight express the adverse event report to all 2,000 investigators, with instructions that the investigators review it carefully, classify it, and send it to their respective IRBs for further review and classification. The cost of that exercise for a single event that contributed no new knowledge about the risk and benefit balance of the drug was estimated at $450,000.
Starting a trial in the United States can cost $14,000 per site before the first patient is enrolled simply because of current regulations and pro-cedures governing trial initiation, including IRB evaluation and contract-ing. A Cooperative Study Group funded by the National Cancer Institute recently published an analysis demonstrating that a minimum of more than 481 discrete processing steps are required for an average Phase II or Phase III cancer protocol to be developed and shepherded through various approval processes (Dilts et al., 2008). This results in a delay of more than 2 years from the time a protocol is developed until patient enrollment can begin, and means that “the steps required to develop and activate a clini-cal trial may require as much or more time than the actual completion of a trial.”
We must ask: Do the benefits conferred by documenting pre-study evaluation visits or pill counts really outweigh the costs of collecting such data, for example? Do we need 800 different IRBs reviewing protocols for large multicenter trials, or could we enact studies using central IRBs or col-laborative agreements among institutional IRBs? Is all the monitoring and safety reporting that we do really necessary (or even helpful)?
Transforming Clinical Trials
All is not dire, however. One promising new initiative is the FDA Critical Path Initiative (public/private partnership [PPP]): the Clinical Trials Transformation Initiative (CTTI), which is intended to map ways to better trials (www.trialstransformation.org). A collaboration among the FDA, industry, academia, patient advocates, and nonacademic clinical research-ers, CTTI is designed to conduct empirical studies that will provide evi-dence to support redesign of the overall framework of clinical trials and to eliminate practices that increase costs but provide no additional value. The explicit mission of CTTI is to identify practices that through adoption will increase the quality and efficiency of clinical trials.
Another model that we could adapt from the business world is the concept of establishing “envelopes of creativity.” In short, we need to create spaces within organizations where people can innovate with a cer-tain degree of creative freedom, and where financial incentives reward this creativity. Pediatric clinical trials offer a good example of this approach.
Twenty years ago, clinical trials were rarely undertaken in children; many companies argued that they simply could not be done. Pediatricians led the charge to point out that the end result of such an attitude was a shock-ing lack of knowledge about the risks and benefits of drugs and devices in children. Congress was persuaded to require pediatric clinical trials and grant patent extensions for companies that performed appropriate trials in children (Benjamin et al., 2006). The result was a significant increase in the number of pediatric trials and a corresponding growth in knowledge about the effects of therapeutics in children (Li et al., 2007).
Conclusions
If we all agree that clinical research must be improved in order to provide society with answers to critical questions about medical tech-nologies and best practices, a significant transformation is needed in the way we conduct the clinical trials that provide us with the most reliable medical evidence. We need not assume that trials must be expensive, slow, noninclusive, and irrelevant to the measurement of important outcomes that matter most to patients and clinicians. Instead, smarter trials will
TAKING ADVANTAGE OF NEW TOOLS AND TECHNIQUES
become an integral part of practice in learning health systems as they are embedded into the information systems that form the basis for clinical practice; over time, these trials will increasingly provide the foundation for integrating modern genomics and molecular medicine into the frame-work of clinical care.
INNOVATIVE ANALYTIC TOOLS FOR LARGE CLINICAL AND ADMINISTRATIVE DATABASES
Sebastian Schneeweiss, M.D., Sc.D.
Harvard Medical School
BWH DEcIDE Research Center on Comparative Effectiveness Research Instrumental Variable Analyses for Comparative Effectiveness
Research Using Clinical and Administrative Databases
Physicians and insurers need to weigh the effectiveness of new drugs against existing therapeutics in routine care to make decisions about treat-ment and formularies. Because FDA approval of most new drugs requires demonstrating efficacy and safety against placebo, there is limited interest by manufacturers in conducting such head-to-head trials. Comparative effectiveness research seeks to provide head-to-head comparisons of treat-ment outcomes in routine care. Because healthcare utilization databases record drug use and selected health outcomes for large populations in a timely way and reflect routine care, they may be the preferred data source for comparative effectiveness research.
Confounding caused by selective prescribing based on indication, severity, and prognosis threatens validity of nonrandomized database studies that often have limited details on clinical information. Several recent developments may bring the field closer to acceptable validity, including approaches that exploit the concepts of proxy variables using high-dimensional propensity scores and exploiting provider variation in prescribing preference using instrumental variable analysis. The paper provides a brief overview of those two approaches and discusses their strengths, weaknesses, and future developments.
Very briefly, what is confounding? Patient factors become confounders (“C” in Figure 3-4) if they are associated with treatment choice and are also independent predictors of the outcome. When researchers are inter-ested in the causal effect of a treatment on an outcome, factors that are independently predicting the study outcome, such as severity of the under-lying condition, prognosis, co-morbidity, are at the same time also driving the treatment decision. Once these two conditions are fulfilled, you have a confounding situation and you get biased results. In large-claims database