How Manipulated Clinical Evidence Could Distort Guidelines - the Case of Statins for Primary Prevention

The new American College of Cardiology (ACC)/ American Heart Association (AHA) guidelines on the primary prevention of cardiovascular disease, which we discussed here, continue to generate controversy. 

Articles in the media and online-first in medical journals underscored some of the issues we discussed before.  Jeanne Lenzer, in a news article in the British Medical Journal, found that the chair of the guideline panel had important past conflicts of interest that were not previously disclosed.(1)  The chair denied their significance (look here).   Guideline panel members continued to justify their efforts, but in my humble opinion, raised no new evidence or logic to support it (look here  and here)

Bigger Questions about the Validity of the Clinical Evidence

However, several new articles suggested the need for a deeper look at one particular aspect of this case, the validity of the evidence from clinical research about the benefits and harms of statins for primary prevention.

JAMA published a summary of the Cochrane review that provided a basis for the guideline developers' confidence in the worth of statin therapy in primary prevention.(2)

results suggest that the benefits of statin therapy outweigh serious life-threatening hazards.

However, almost as an aside, it noted,

Some trials included participants with CVD, but rather than exclude these trials, we included trials that contained 10% or fewer participants with documented CVD.

Primary prevention in this case is defined as prevention for patients without existing cardiovascular disease.  There is evidence that statins may well have benefits that outweigh harms when given to patients with known cardiovascular disease, particularly documented coronary artery disease (CAD).  Mixing such patients in any numbers into "primary prevention trials" would likely exaggerate the benefits of statins.  Yet such not quite primary prevention studies were included in a systematic review of primary prevention. 

In addition, a commentary by one of the guideline developers defending the group's work also underscored the fact that many of the supposedly primary prevention trials they used as evidence were not pure primary prevention trials.(3)

Notably, the 2013 cholesterol guideline cut points were derived from the placebo rates for myocardial infarction, stroke, and cardiovascular disease death observed in the 3 exclusively primary prevention statin trials, Air Force/Texas Coronary Atherosclerosis Prevention Study (AFCAPS/TexCAPS), Management of Elevated Cholesterol in the Primary Prevention Group of Adult Japanese (MEGA) study, and the Justification for the Use of Statins in Primary Prevention: an Intervention Trial Evaluating Rosuvastatin (JUPITER).[4-6]

The wording again suggests that all the other trials used as evidence about primary prevention were NOT "exclusively primary prevention studies," and hence, as I noted above, may have been biased so that they would likely exaggerate the apparent benefits of statins in primary prevention.

This suggested to me that the systematic review that provided a basis for the guidelines' aggressive recommendations about statins for primary prevention, and the trials on which it was based needed further skeptical, critical reviews.

The Cochrane Review: Was Evidence about Statin Benefits vs Harms Manipulated?

We have frequently discussed the manipulation of clinical research.  By that we have meant design, implementation, analysis or dissemination of research in ways likely to further vested interests.  In particular, when drug, biotechnology and device companies sponsor and control clinical research on their own products, they may set up the research in ways likely to make their products look better than they actually are.  

We looked through the most recent Cochrane review of statins for primary prevention(7) with a skeptical eye out for suggestions that the trials reviewed could have been so manipulated.  We found quite a bit.

First, we found that not all trials reported on the "hard" outcomes that one needs to consider when evaluating benefit vs harm of statins in primary prevention.

Data on all-cause mortality were provided in 11 trials.[of 19, and hence missing in 8].

And,

 Twelve trials provided data on adverse events.[and hence 7 did not.]

 Also, data on specific adverse events was often not reported: myalgia and rhabdomyolysis were reported in only 9/19 trials; new onset diabetes mellitus in only 2/18; hemmorhagic stroke in only 2/19; abnormal liver tests in only 10/19; kidney dysfunction in only 4/19; arthritis in only 2/19, and by implication, cognitive dysfunction in 0/19.  

  Failure to look for all the possible bad outcomes of treatment could obviously bias the study in the direction of minimizing the harms of the treatment.

A substantial number of trials either failed to report on crucial aspects of their methods, or admitted to flaws that could have induced important biases.: 3/19 did not described randomization methods; 4/19 did not use double-blinding; 6/19 did not use intention to treat analysis; 7/19 did not report their drop-out rates.

So it is very surprising to me that the authors concluded,

In general there was low risk of bias ... though all trials were either fully or partially funded by pharmaceutical companies.

In my humble opinion, the Cochrane review showed many trials that had flaws could have biased their outcomes, and hence the outcomes of the overall review.  Some of the flaws clearly could have lead to biases that would have made statins look more efficacious, or less dangerous than they might actually be.  I do not understand the conclusion that the risk of bias was slow, and the lack of discussion about the direction the bias could have taken.

 Review of the "3 Exclusively Primary Prevention Statin Trials"

Given the Cochrane review's apparent lack of skepticism about methodologic problems in the industry funded statin prevention trials, I endeavored to take a closer look at the three trials that Dr Robinson held out as the real primary prevention trials.  Instances of manipulation, as we defined it above, for each trial are described below

AFCAPS/ TexCAPS(4)

Narrow Patient Population - This study excluded many patient for whom the statins were not contraindicated or warned against: uncontrolled hypertension; type 1 or type 2 diabetes mellitus on insulin or with a HgBA1C at least 10%; and body weight more than 50% "desirable limit for height."  (Based on the official contraindications and warnings for commonly used statins, e.g., see contraindications for Lipitor here, active liver disease, pregnancy for likely to become pregnant, nursing mothers, hypersensitivity to the medicine; and warnings: use of cyclosprine or strong CYP3A4 inhibitors, uncontrolled hypothyroidism, renal impairment.)  Thus the results may not generalize to many patients who would otherwise be considered statin candidates.  By excluding such patients, the results may bias the study towards minimizing the probabilities of harms that might occur were statins used on a wide population for primary prevention.

Unknown Randomization and Allocation Concealment Procedures - According to the Cochrane Review, the study report did not explain how randomization or allocation concealment were accomplished.  

Early Termination/ Multiple Comparisons - The study was terminated early based on an early look at the number of outcome events.  Two such early or interim analyses were planned.  Taking multiple looks at the data over time raises a multiple comparisons problem, and may lead to exaggerating the benefits of the treatment.(8). Furthermore, stopping early decreases the sample size and hence the power to find adverse effects of treatment.

Implausible Dropout Rate, Missing Data - According to the Cochrane Review, the study reported no dropouts.  This seems somewhat improbable, suggesting skepticism about the accuracy and completeness of the data collection.  On the other hand, a study chronology suggests that of 6605 patients who started the study, 6540 had data on complete endpoint status, suggesting missing data.  Since dropouts and missing data may be due to different reasons in different arms of the study, they threaten the validity of data about benefits and harms.  

Adverse Effects not Reported - The study provided no data about development of diabetes, hemmorhagic stroke, kidney dysfunction, arthritis, or cognitive dysfunction, suggesting incomplete data about harms, and hence bias towards minimizing harms.    

MEGA (5)

Narrow Patient Population -  [The patient population was not described in the main report, but only in an earlier methods article.](9)  The study excluded patients with congenital or rheumatic heart disease; chronic atrial fibrillation, current diagnosis of malignancy; poorly controlled hypertension or diabetes mellitus; current use of oral or parenteral corticosteroids; and other conditions at the discretion of the physician.  These exclusions seem unrelated to the contraindications or warnings on the stain label.  Again, such a narrow patient population reduces the generalizability of the study results, and may bias the study to minimizing the harms of statins.

Only Single Blind - This was an open-label study, so patients and physicians knew who got statins and who got placebo.  Such knowledge could have biased patient management, including how diligently particular diagnoses and outcomes were pursued, and biased data collected from patients or physicians.

Adverse Effects Not Reported - The study provided no data about diabetes, hemmorhagic stroke, kidney dysfunction, arthritis, or cognitive dysfunction, again suggesting bias towards minimizing harms.  

JUPITER(6)

Narrow, Unusual Patient Population - The study was limited to patients without hyperlipidemia but with an increased C-reactive protein.  Thus it is not clear that its results would generalize to a more typically defined primary prevention population.  The study excluded patients receiving post-menopausal hormone-replacement; with diabetes; uncontrolled hyertension; cancer other than non-melanoma skin cancer within 5 years; recent history of drug or alcohol abuse; "another medical condition that might compromise safety or the successful completion of the study;" also patients with "inflammatory conditions such as severe arthritis, lupus, or inflammatory bowel disease...;" and also "patients taking immunosuppressant agents such as cyclosporine, tacrolimus, azathioprine, or long-term oral glucocorticoids."  Again, this narrow patient population would reduce generalizability and bias towards minimizing harms.

Early Termination/ Multiple Comparisons - This study was terminated early after an early look at the data.  Allowing for multiple looks at the data may exaggerate efficacy.  

Implausible Dropout Rate - According to the Cochrane Review, the study had no dropouts.  This value seems implausible, again suggesting data collection problems.  

Adverse Effects Not Reported - The study provided no data about hemmorhagic stroke, arthritis, or cognitive dysfunction, again suggesting bias towards minimizing harms.  [Revised December 9, 2013 - see comment below by Marilyn Mann.]

Summary

Aspects of the continuing controversy over the new ACC/ AHA guidelines for statins in the primary prevention of cardiovascular disease hinted that the clinical trials which provided the evidentiary basis for the guidelines, and for the use of statins in primary prevention in general, was more flawed than is widely appreciated.  The latest Cochrane Collaboration review of this data acknowledged multiple, important flaws affecting most of the studies.  Our more detailed review of the three studies held out as the purest found additional flaws.  Many of these flaws seemed likely to bias the studies towards exaggerating the efficacy and/or minimizing the harms of statins in primary prevention.  Since all these trials were funded, and presumably influenced by pharmaceutical companies that make statins, these flaws seem to be examples of manipulation of the clinical evidence.  Rather than being the result of simple mistakes, or inevitable trade-offs, they seem to be study features intended the support the vested interests of the study sponsors.

It is not clear why the Cochrane review did not temper its conclusions based on the flaws in the studies, and particularly by the possibility that these flaws represented study manipulation.  

The multiple flaws, possibly due to study manipulation, in the clinical evidence about statins in primary prevention suggest that we should be extremely skeptical about whether the benefits of such treatment outweighs its harms, and hence about whether the recommendations in the latest guidelines to give statins to all patients predicted (but perhaps not accurately) to be at even slightly elevated risk are warranted.

The flaws in multiple large studies of a very common clinical problem, and their effects on systematic reviews and clinical practice guidelines suggest that suppression and manipulation of research are rife in medicine and health care, presumably fueled by the pervasive web of conflicts of interest that spans health care.  We need extreme skepticism about the integrity of clinical research, especially research sponsored by those whose products and services are being studied, and who thus have vested interests in the research turning out to make their products and services look good.  

The good news is that we may not have to look too far to find ways to improve the trustworthiness of guidelines and the soundness of medical decision making.  Implementation of the Institute of Medicine's recommendations on reducing conflicts of interest (look here), and developing trustworthy guidelines (look here) might lead to the development of sound guidelines in the future.  

Furthermore, while endless discoveries of manipulated and suppressed research may have lead some evidence-based medicine advocates to despair, our latest exercise suggests that the principles of evidence-based medicine, unflinchingly applied, could really do good.  Review of the three statin studies above based on standard principles of critical review readily spotted the multiple signs of manipulation.  The problem with the Cochrane review was not that it missed these signs.  Rather, the reviewers for some reason noted most of them, but then did not react.  If systematic reviews were done with sufficient skepticism about the possibility of manipulation of clinical research, and were willing to call out when the emperor seemed short on fabric, then a lot of mischief could be avoided.  


References

1.  Lenzer J. Majority of panelists on controversial new cholesterol guideline have current or recent ties to drug manufacturers.  Brit Med J 2013.  Link here. 

2.  Taylor FC, Huffman M, Shah E. Statin therapy for primary prevention of cardiovascular disease.  JAMA 2013.  Link here.  

3.  Robinson JG.  Accumulating evidence for statins in primary prevention.  JAMA 2013;  Link here.

4.    Downs  JR, Clearfield  M, Weis  S,  et al; for the AFCAPS/TexCAPS Research Group.  Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. JAMA. 1998;279(20):1615-1622. Link here. 
5.  Nakamura  H, Arakawa  K, Itakura  H,  et al; MEGA Study Group.  Primary prevention of cardiovascular disease with pravastatin in Japan (MEGA study): a prospective randomised controlled trial. Lancet. 2006;368(9542):1155-1163. Link here.

6.  Ridker  PM, Danielson  E, Fonseca  FA,  et al; JUPITER Study Group.  Rosuvastatin to prevent vascular events in men and women with elevated C-reactive protein. N Engl J Med. 2008;359(21):2195-2207. Link here. 

7.  Taylor F,  Huffman MD, Macedo AF et al.  Statins for the prevention of cardiovascular disease. Cochrane Library 2013.  Link here.

8. Mueller PS, Montori VM, Bassler D et al.  Ethical issues in stopping randomized trials early because of apparent benefit.  Ann Intern Med 2007; 146: 878-881.  Link here.  

9.  Management of Elevated Cholesterol in the Primary Prevention Group of the Adult Japanese (MEGA) Study Group.  Design and baseline characteristics of a study of primary prevention of coronary events with pravastatin among Japanese with mildly elevated cholesterol levels.  Circ J  2004; 68: 860-867.  Link here.

"The Scent ... of a Casino" - Clinical Research Results as Fodder for Insider Trading

"The scent and smoke and sweat of a casino are nauseating at three in the morning. Then the soul-erosion produced by high gambling - a compost of greed and fear and nervous tension - becomes unbearable and the senses awake and revolt from it." - Ian Fleming.  Casino Royale.  1953

A major theme of this blog has been threats to the integrity of clinical research.  When I started out as a young naive academician in medicine, I viewed clinical research through the lens of evidence-based medicine, as primarily a means to develop better evidence to aid in the care of patients, and secondarily a way to advance science and public health.  Yet in the past 20 to 30 years, clinical research has become commercialized.  Now most randomized controlled clinical trials are done to support the licensing of drugs and devices for particular indications.  Furthermore, clinical research has become de facto a primary avenue of marketing of drugs and devices.  So we have discussed endlessly how clinical research has been manipulated by those with vested interests in selling drugs and devices, and may be suppressed when even such manipulation fails to produce results desired by commercial sponsors.

If that was not bad enough - and it is - now there is evidence that clinical research has become the roulette wheel in an investment casino.  This is the lesson provided by new US government charges that giant hedge fund SAC Capital used insider knowledge to trade drug company stocks.


Let me try to provide a narrative of the key elements of the case.

The Key Players

Matthew Martoma was a " former student at Harvard Law School, ... [who] co-wrote papers on medical ethics before seeking a business degree at Stanford University and joining a little-known Boston hedge fund."  Then, "in 2006, at age 32, Martoma made it to SAC Capital Advisors LP and gained the attention of the firm’s billionaire owner Steven A. Cohen."(1)

Dr Sidney Gilman is

an 80-year-old neurologist with expertise in neurodegenerative disorders, including Alzheimer’s disease. According to his biography on the University of Michigan website, Gilman first served on the faculty at Harvard and Columbia and then had a long and distinguished career at the University of Michigan, where he was the chair of the department of neurology for many years. He is a member of the Institute of Medicine of the National Academy of Sciences and a past president of the American Neurological Association. In other words, he’s a bigshot.

Gilman moonlighted as a consultant, working for an expert networking firm, where he provided advice to the financial industry (and which eventually led to the insider trading case), and for Elan Pharmaceuticals. In addition to his consulting for Elan, he also served as the chair of the Safety Monitoring Committee for a phase II clinical trial of a highly promising (at the time) Alzheimer’s drug, bapineuzumab, under development by Elan and Wyeth.(2)

The owner of SAC Capital, a hedge fund worth $14 billion, is Steven A Cohen, "a prodigious art collector, an investor in the New York Mets, a supporter of Mitt Romney’s presidential campaign."  His "career ... has reached mythic status on Wall Street. Over the last 20 years, Mr. Cohen has amassed a multibillion-dollar fortune by posting returns averaging 30 percent a year. SAC has grown into a firm with about 1,000 employees around the world."(3)

The Maneuver


Per the NY Times(4), 

The doctor met Martoma as a consultant for an expert- networking firm based in Manhattan and had sessions with Martoma from mid-2006 to July 2008, according to the government.

Gilman worked for Gerson Lehrman Group’s Scientific Advisory Board starting in 2002,... 

Also according to the NY Times(5)
 

Dr. Gilman’s consulting work for Mr. Martoma earned him about $108,000, according to court filings. Based in part on Dr. Gilman’s leaks about positive developments related to the clinical trials of a new Alzheimer’s drug, SAC accumulated a roughly $700 million position in the stocks of Wyeth and Elan, according to the government.

The S.E.C. said that the fund’s owner, Mr. Cohen, took a large position in Wyeth and Elan in his personal portfolio based on Mr. Martoma’s recommendation. Mr. Cohen maintained his holdings even though there was significant internal debate about the wisdom of such a large position in the drug makers, the government said.

Furthermore, per Larry Husten writing in Cardiobrief(2),

 While serving on the Safety Monitoring Committee of the trial, from the summer of 2006 through mid-July 2008, Gilman had access to the safety (but not the efficacy) data from the trial. Throughout this period he leaked the positive safety information to his contact at SAC (the enormous hedge fund), which then began to accumulate a large position in both Elan and Wyeth.

So, allegedly based on information from Dr Gilman, the high-ranking academic physician doubling as a data monitoring committee chair for a drug trial, paid for these services by Elan, SAC Capital and Mr Cohen made a large investment in Elan and Wyeth, the companies sponsoring the trial.

But then, again per Husten(2),

 Later in June, Elan chose Gilman to present the full trial data at ICAD. Gilman did not become privy to this data until the middle of July when Elan gave him the full results of the trial. The results, in sharp contrast to the earlier view in the press release, were decidedly negative, offering little hope that the drug would be considered effective. Gilman apparently understood this, because he immediately gave the results to the hedge fund, which then rapidly and dramatically reversed its long position on Wyeth and Elan.

Specifically, per the New York Times(5), after

Dr. Gilman told Mr. Martoma that patients were experiencing serious side effects, prosecutors say. Afterward, Mr. Martoma e-mailed Mr. Cohen, telling him 'it’s important' that they speak. They spoke on the phone for nearly 20 minutes, the government says, and Mr. Martoma told his boss that he was no longer 'comfortable' with the investments.

The following day, SAC reversed course. Mr. Cohen’s head trader sold the firm’s entire inventory of roughly 10.5 million shares of Elan and about seven million shares of Wyeth, the government said. Once it had dumped the shares, SAC built a short position in the two stocks, betting their value would drop.

According to the S.E.C., the trader, Mr. Cohen and Mr. Martoma kept the sales confidential. The trade, wrote the head trader in an e-mail to Mr. Cohen, 'was executed quietly and efficiently over a four-day period through algos and darkpools' — referring to trades using algorithms and to trading platforms that do not have the same reporting requirements as the stock exchanges — 'and booked into two firm accounts that have very limited viewing access.'

After the companies announced the results of the trials, Elan’s stock fell about 42 percent and Wyeth’s about 12 percent.

The trading allowed SAC to avoid about $194 million in losses and earn about $83 million in profits on Elan and Wyeth, according to prosecutors.

At the end of 2008, Mr. Martoma received a bonus of about $9.3 million, the S.E.C. said.

So, having bet heavily on Elan and Wyeth based on Dr Gilman's initial information that the trial was showing positive results, SAC Capital and Mr Cohen then bet heavily against these companies based on Dr Gilman's new information that the trial would end up not showing such results.  Because they had this information allegedly before it was made public, these bets yielded huge profits.  Mr Martoma, the conduit between SAC Capital and allegedly Mr Cohen, made millions.  Dr Gilman made over a hundred thousand.

The Charges

Per Bloomberg(4),

Martoma, 38, was accused by prosecutors in Manhattan federal court with playing a lead role in what they called the most lucrative insider-trading scheme in history, given the $276 million profit he allegedly helped the hedge fund achieve.

The government started off taking a tough approach, per the NY Times(5),

FBI agents arrested Mr. Martoma, 38, early Tuesday morning at his home in Boca Raton, Fla. He was released on bail after making an appearance in Federal District Court in West Palm Beach. Mr. Martoma, who has been unemployed since leaving SAC in 2010, is expected to appear in federal court in Manhattan on Monday and enter a plea.

 In addition, Dr Gilman, per Bloomberg(1), "has entered into a non- prosecution agreement with prosecutors in which he agreed to testify before a federal grand jury and to forfeit $186,761, money which he was paid by Elan for his consulting work."

Mr Cohen has not been charged.  However, according to John Cassidy writing in the New Yorker(6),

the complaints do assert that Cohen, identified as 'Portfolio Manager A,' personally authorized many of Martoma's trades, and pocketed the bulk of the profits they generated.  In a statement accompanying the indictment [US Attorney Preet] Bharara was careful to state that S.AC., and by extension Cohen, reaped enormous rewards from the criminal wrongdoing, even though they weren't being charged.

the complaints do assert that Cohen, identified as “Portfolio Manager A,” personally authorized many of Martoma’s trades, and pocketed the bulk of the profits they generated. In a statement accompanying the indictment, Bharara was careful to state that S.A.C., and by extension Cohen, reaped enormous rewards from the criminal wrongdoing, even though they weren’t being charged.

Read more: http://www.newyorker.com/online/blogs/johncassidy/2012/11/sharks-circle-around-hedge-fund-big.html#ixzz2DdLAchwv

Read more: http://www.newyorker.com/online/blogs/johncassidy/2012/11/sharks-circle-around-hedge-fund-big.html#ixzz2DdKdZcAY

In addition, there is now some likelihood that there will be legal actions against SAC Capital.  The company has reportedly received a "Wells notice" that the SEC is considering pursuing such action.  In addition, according to the New York Times(7), "an additional action against SAC, or even Mr. Cohen, could involve accusations of fraud based on the so-called control-person liability theory, meaning that it was in 'control' of Mr. Martoma when he engaged in insider trading."

the complaints do assert that Cohen, identified as “Portfolio Manager A,” personally authorized many of Martoma’s trades, and pocketed the bulk of the profits they generated. In a statement accompanying the indictment, Bharara was careful to state that S.A.C., and by extension Cohen, reaped enormous rewards from the criminal wrongdoing, even though they weren’t being charged.

Read more: http://www.newyorker.com/online/blogs/johncassidy/2012/11/sharks-circle-around-hedge-fund-big.html#ixzz2DdKdZcAY

Summary

At one point, I would have simply regarded the trial of bapineuzumab as a clinical scientific experiment meant to determine if a promising new therapy would work for the important clinical problem of Alzheimer's disease, and incidentally as means to learn more about the biology of that disease.  More recently, I would have regarded the trial as primarily a means for Elan and Wyeth to persuade the US Food and Drug Administration to approve this drug as a treatment for this disease, and then if so, a means for these companies to market it.  I would have been skeptical about the value of the clinical evidence supplied by the trial to support use of this drug, but would not have dismissed this evidence out of hand.

Now it appears that a major role of this trial was to be a roulette wheel in a Wall Street casino.  Big bettors on this wheel included people who may have had advance knowledge about the slot in which the ball would land.

Ideally, clinical research ought to be done by people who have no particular interest in the results turning out one way or the other.  Obviously, clinical investigators as humans may have opinions about how studies might turn out.  However, they should not be in positions to gain or lose money according to the results, or to be intimidated by those with interests in having the studies obtain particular results. Our current system in which many clinical studies are funded by organizations with interests in how the results turn out has lead to numerous cases of manipulation of study results, and sometimes suppression of studies whose results could not be manipulated successfully in the desired direction.  Such threats to study integrity make it difficult to make the best clinical decisions for individual patients, distort health policy, and break the trust of patients who volunteered to participate in the studies.

Now it appears that short-term stock trading that bets on the results of clinical research, sometimes informed by insider information about these results, could be another powerful external influence on clinical research that could additionally threaten its integrity.  Furthermore, as we speculated seven years ago (see this post and links backward), fear of insider trading may be making clinical research more opaque, and this opacity may allow even more control of research by sponsors (companies funding it) rather than investigators.

We have previously suggested that real consideration ought to be given to taking clinical research out of the hands of organizations, particularly health care corporations, that have vested interests in the direction of the results.  Knowledge that clinical research is increasingly becoming a casino for stock traders and hedge funds, and that research results are becoming the stuff of insider trading should further prompt this consideration. 


References
1.  Burton K, Kishan S, Van Voris B. Cohen's 'Elan Guy' Martoma dropped ethics for hedge fund.  Bloomberg, Nov 23, 2012.  Link here.  
2.  Husten L. The doctor and the center of the insider trading scandal.  Cardiobrief, Nov 27, 2012.  Link here.
3.  Lattman P. S.E.C. weighs suit against SAC Capital.  New York Times, Nov 28, 2012.  Link here.
4.  Van Voris B, Hurtado P. Insider crackdown uses SAC manager in health-care pivot.  Bloomberg, Nov 21, 2012.  Link here.
5.  Lattman P. Insider inquiry inching closer to a billionaire.  New York Times, Nov 20, 2012.  Link here.
6.  Cassidy J. It's time for Obama to bite the hedge-fund sharks.  New Yorker, Nov 21, 2012.  Link here.
7.  Lattman P. S.E.C. weighs suit against SAC Capital.  New York Times, Nov 28, 2012.  Link here. 

When Clinical Trials are Meant for Marketing, not Science

The development of the randomized controlled clinical trial (RCT) was one of the major scientific advances in clinical medicine.  RCTs provide a major part of the evidence underlying evidence based medicine.  RCTs provide a major source of data used by the US Food and Drug Administration, and similar agencies in other countries, to decide whether to approve drugs or devices to manage particular clinical problems.  Unfortunately, with the rise of the RCT came a rise in attempts to suppress and manipulate clinical trials by those with vested interests, often in selling the products and services the trials can evaluate.

Many of the examples we have discussed were of attempts at manipulation or suppression of clinical trials originally done to provide evidence for product approval, or even ostensibly to advance clinical science.  Yet a relatively new article covering evidence revealed from litigation about the drug gabapentin (Neurontin) originally made by Parke-Davis, first a part of Warner Lambert, now part of Pfizer suggests that many clinical trials may not be done to advance science, or even just to provide data to regulators, but only to market products.

The article is Vedula SS, Goldman PS, Rona IJ, Greene TM, and Dickersin K. Implementation of a publication strategy in the context of reporting biases: a case study based on new documents from the Neurontin litigation.  Trials 2012; 13: 136.  Link here.

Methods Summary

The article described a case study based on documents revealed in 2008 litigation.  The purpose of the study was:

to describe the implementation of a publication strategy for off-label marketing of gabapentin, within the context of reporting biases and spin of Pfizer and Parke-Davis’s clinical trial findings, for four off-label uses: migraine prophylaxis, treatment of bipolar disorders, neuropathic pain, and nociceptive pain. 

The study examined documents revealed through the 2008 litigation.  It concentrated on documents relating to the marketing of Neurontin for four off-label indications, that is, for four potential reasons to use the drug which had not yet been approved by the US FDA.  It focused on documents discussing strategies for marketing for these indications, and then documents about particular company sponsored RCTs done for these indications.

Strategies to Sell Drugs

The documents examined by the investigators showed that Parke-Davis prepared "marketing assessments" for four possible indications for Neurontin.  The assessments discussed two possible strategies.  The "indication strategy" would be to conduct trials for the purposes of providing data to the FDA in the hope that the FDA would then approve the new indication for the drug.  The "marketing strategy" would be, in the words of an internal company memo quoted in the study, to conduct clinical trials and

to disseminate the information as widely as possible through the world’s medical literature

In other words, the publication strategy involved using RCTs first for marketing, rather than scientific or regulatory purposes.

How the Trials were Controlled by Marketers 

The publication strategy involved not simply doing trials and publicizing their results, but controlling the messages conveyed by the trials to make sure they primarily supported marketing.  This was done by having the company's marketing department control the content of the trial reports.

 A Neurontin Publications Subcommittee (NTN PSC) was formed within Pfizer and Parke-Davis to implement a publication plan. Minutes from meetings between the NTN PSC and Medical Action Communications (MAC), a medical writing company, indicate that a list of key messages, guiding the content of published reports related to the trials of gabapentin for off-label indications, was developed based on a branding guide

Thus it appeared to be that marketers, not physicians or scientists were in control of supposedly scholarly research publications appearing in medical journals.

The marketers controlled content by controlling those who wrote it.

 A standard operating procedure related to publication of affiliate-driven manuscripts was identified in internal company documents dated October 16, 2002, and it sheds further light on the publication planning process (see Figure 3). (The term affiliate in this context refers to Pfizer’s foreign affiliates, that is, corporations related to Pfizer by either shareholdings or other means of control, including subsidiary, parent, or sibling corporations). According to the internal company documents, 'affiliate-driven manuscripts' were written for Pfizer and Parke-Davis by MAC and sent to the authors for approval. Each article was coordinated by a manuscript team, consisting of representatives from the medical and marketing divisions of the company. The documents also indicate that all affiliate-driven manuscripts should be forwarded to the NTN PSC for review. One of the objectives of manuscripts being reviewed by the NTN PSC was to ‘ensure that they are in-line with current product messages and areas of interest’

Manipulating the Dissemination (and Analysis) of Research Results

Note that the control of publication by marketers describe above involved ghost-writing of articles by medical education and communications companies (MECCs).  Later the article explained that these ghost writers were "not appropriately acknowledged" in the resulting published articles.

The Vedula et al study article provided examples of "spin" in the writing of again supposedly scholarly research publications used in fact to support marketing, and controlled if not composed by marketers.

 We identified spin in publications related to 8/12 trials included in our analysis.... We classified the following as spin: emphasis in the published report on outcomes that were not specified in the study protocol (Study 879–201) ...; conclusions that did not match study findings described in the internal company research report (Study 945–220) ...; extensive rationale to explain away statistically non-significant (unfavorable to the sponsor) findings (Study 945–209; 945–291; No study number - Gorson) ...; conclusion of treatment effectiveness from an uncontrolled study (Study 945–250) ...; emphasis on statistically significant secondary outcomes despite negative findings for the primary outcome (Study 945–271) ...; and an explicit description of an attempt to spin study findings (as described in internal company emails) (Study 945–306).

Note that while most of these examples of spin involved manipulation of the dissemination of study results, that is, doubtful, biased or fallacious arguments based on apparently unbiased data, they also involved manipulation of the analyses (in italics). 

The article also described manipulation of dissemination involving the timing of publication, including delaying publication of an article despite the wishes of the study's investigators because the results were not statistically significant, and hence not favorable for marketing. 

Summary

Documents revealed by litigation about Neurontin in 2004 provided insights about how pharmaceutical and presumably other kinds of health care corporations may conduct systematic, deceptive stealth marketing campaigns to promote their products and services (look here).  We noted that initial media coverage of documents revealed in the 2008 litigation also suggested the existence of a systematic "publication strategy" to control dissemination of results of particular trials, while suppressing trials whose results could not easily be spun to provide support of marketing objectives (look here).

Now the new paper by Verdula et al fill out our knowledge of this case.  The paper suggests that randomized controlled clinical trials may be done not to advance science, or even convince regulators, but primarily to market their sponsors' products.  Thus some significant proportion of the clinical research literature, the literature that physicians and other health professionals have relied upon to make evidence-based decisions for their patients, may exist mainly for marketing purposes.  Even the most rigorous methods used by clinical epidemiologists to review research are meant to discover problems that arose from human error or the inevitable trade-offs made when research is done in the real world, but not deliberately introduced biases and defects meant to promote vested interests.  Thus it is not clear that even the most "evidence-based" medical decisions are based on real scientific evidence rather than the spinning of marketers.

Thus health care professionals, policy makers, researchers, and the interested public need to be even more skeptical about arguments made to promote innovative treatments and other clinical interventions.  However, it is not clear that even rigorous skepticism can defend the integrity of evidence based medicine from marketing disguised as clinical research.

Going forward, we must consider erecting an impregnable barrier between clinical research and those whose primary interest is to make money by selling health care goods and services.  If we do not do that, we will forever need to worry that we really have no idea what "works in medicine," and whether any particular test, treatment, or program provides benefits that outweigh its harms. 

"MetroHealth, Explorys use huge patient database to revolutionize medical research"

As I have written in the past, beware claims of "revolutions" where healthcare or medical research are involved.  As in a slide in my recent presentation to the Health Informatics Society of Australia on healthcare IT trust, I asked:

In that same talk, I pointed out the "revolutions" usually have downsides, and IT always produces winners and losers (per the empirical research of Social Informatics). 

I also asked "have we suffered a complete breakdown in the scientific method with regard to EHR and clinical IT?" in a 2009 post on uncontrolled EHR data and comparative effectiveness studies at this link.  

In the Aug. 29, 2012 article "MetroHealth, Explorys use huge patient database to revolutionize medical research"  in The Plain Dealer (Ohio), the following claims are proffered.  I am assuming regarding this article, as in newspaper articles I make contributions to, that the researchers were involved in its provenance and content:

Large databases of electronic medical records hold great promise for medical research. In theory they can provide doctors access to huge amounts of anonymous patient data, allowing large-scale population studies without the cost and hassle of patient recruitment, review boards and staff training.

Now, a team of data experts at MetroHealth Medical Center and the Cleveland Clinic Innovations spinoff company Explorys has shown just how powerful such medical records can be: In three months, they've replicated a major medical study that took a Norwegian team 14 years to research and report. And they've done it at a fraction of the cost, with a sample about 40 times as large.

The local effort, led by MetroHealth Chief Medical Informatics Officer Dr. David Kaelber, was possible because of Explorys' database of 14 million electronic medical records gathered from 12 major health systems.

14 million records gathered from 12 major health systems, which may in fact be using disparate EHR systems or versions, is almost indisputably not controlled data.

A comparison to a more rigorous study was conceptualized and performed:

Most of all though, he [Kaelber] needed to prove that it worked -- that he could get exactly the same results as traditional research studies that required more time and resources.

Enter the Norwegian study. In 1994, a group of researchers started registering 26,714 people in the northern region's largest city, Tromso. It was part of a much longer study on the population's heart disease risk. They recorded height, weight and other measures of obesity, then followed participants for 13 years, recording any blood clots they had.

Their conclusion: the combination of obesity and a tall stature significantly increases risk of blood clots, especially in men.

The same hypothesis was tested with the uncontrolled EHR data, with claims that (obviously) the time and expense were lower, at least in the instant sense (costs of EHR implementation and maintenance in 12 major health systems could be in the billions of dollars at a time when a health system can easily spend $100 million each over just a few years as Bob Wachter points our here - one hundred million dollars for UCSF):

... The sample: 959,030 patients with medical records in the Explorys platform. The method: using software to search for blood clots in the health and claims records contained in the database, and then looking for patterns in those patients' height and weight. The result: Exactly the same as the Tromso study.

"And actually the statistical significance of our study was much, much higher because our sample size was 40 times greater," Kaelber said. Kaelber and his team published their results online in July in the Journal of the American Medical Informatics Association.

The conclusion to the article here (subscription required) was:

With the right clinical research informatics tools and EHR data, some types of very large cohort studies can be completed [and, by implication, trusted and acted upon at local, regional or societal levels - ed.] with minimal resources.

This data was, and I don't think this can be refuted, poorly controlled.  There are limits to what "data cleansing" and standardization can accomplish with such data. In methodologies like this, confounders are too numerous to list comprehensively, but just a few - missing data; variability in observations (inter- and intra-observer variability); data originating from different vendor systems with different terminology and definitions, input by myriad people of different backgrounds with differing interpretations of terminologies (students/MD's/RN's etc); different pressures creating bias (time, reimbursement maximization, litigation avoidance); to name just a few.

Several questions: 

• What is the likelihood these results were themselves a chance outcome? Correlation is not proof, especially where n=1. 

• How do we reliably identify ante hoc the "types of very large cohort studies" that we can complete?

• What is the likelihood that such techniques are generalizable to all studies (such as comparative effectiveness research, currently a governmental initiative), especially where issues and confounders might be subtle?

• Related to generalizability supra, where are the scientific studies that prove methodologies like this are valid and reliable for other than, say, low-granularity epidemiological purposes?  Put more colloquially - how do we know what these efforts attempt to do, in what is almost indisputably a radical deviation from controlled-trial norms, is not "computational alchemy" (i.e., attempting to turn lead into gold)?

• Is there rigorous data on attempted studies like this that failed to correlate with more traditional research?  Unless the latter is studied and/or attempted and some sense gained of whether a radical approach using uncontrolled data such as this does not fail to a concerning extent, then claims of "revolutions" need to be postponed.

Worst case scenario: large dataset, radical analytics and a political agenda can lead to undesired outcomes.  We need to know what we are doing.

More on these issues in a 2009 article I authored here. 

I have invited Dr. Kaelber to comment on the questions I raise. 

Note:  I am not challenging this specific study per se, which is an experiment, but am challenging the generalizability of this type of "from data rags to information riches" methodology, as I've done in other posts on similar topics. Not to mention, the overconfidence as expressed in the press, which is what the public including our lawmakers and people responsible for resource allocation read.  I think it's highly premature to write of "revolutionizing medical research."

Sep. 4, 2012

Here is the reply from Dr. Kaelber -

We actually cover many of the points you raise in the discussion of the article.

I think it is important if we are focusing on a scientific discussion of a scientific article to focus on the primary source - the scientific article itself, and not a secondary source - the reporting of the scientific article.

One of the major points in the discussion (quoted from the published manuscript) is that we feel that there are three keys to using pooled, standardized, normalized, and de-identified EHR data, including:

1. Understanding data sources - understanding of the characteristics of the underlying EHR data sources, including a data dictionary and ontologies used.

2. Corroborating data findings - internal and/or external methods to corroborate retrospective EHR cohort study data and/or findings. For example, manual chart review of a sample of the EHR data (internal validation), or other studies demonstrating similar results (external validation), ideally coupled with a biologically plausible hypothesis.

3. Clinical data versus research data - recognizing that retrospective EHR data were typically collected for clinical and not research purposes. Therefore, depending on the type of data, the quality may not meet typical research standards. In some cases, the large quantity of clinical data may help mitigate the fact that it was not collected with the higher precision and accuracy that can occur as part of a prospective research study.

As I was also quoted in the Plain Dealer article as saying this is like a new type of power drill for those who have been used to using hand drills for retrospective (electronic or paper) chart reviews. To be used appropriately, researchers need to understand the opportunities and disadvantages of using of the new tool and be trained on how to use it.

I would also add that, again, as was stated in the introduction for the published paper that clinical research informatics as a field is in its infancy and we need more people involved in the field, more tools for people in the field, and more experience of people using tools in the field to really understand both the contributions and limitations for electronic health record data to advance clinical research.

I hope these comments are of some interest/use.

-David

My thoughts are that use of of experimental EHR-based methodologies like this, that could have significant social implications, requires significant caution and validation.  I think we are in agreement on that point.  The methodology of use of EHR data from myriad sources needs to be shown valid, as well as proven not to be invalid, before the findings of "new" studies that are not attempting to duplicate prior ones are translated to the bedside or beyond.

The disadvantages include, among other issues (and perhaps most important) the spread of premature over-optimism about health IT such as in newspapers.  That problem has, I believe, contributed greatly to the prevalent hyper-enthusiasm and "there's little supportive data - nevertheless, we believe - so let's spend $15+ billion dollars" attitudes about current health IT as I wrote about, for instance, here (ONC's 'Data Palooza') and here (on MU Stage 2 justifications).

Finally, I would question whether increasing the quantity of data could ever reliably compensate for lack of quality (unless, perhaps  the additions were themselves of high quality).  Adding low quality data to already low quality data should not produce better results, it seems to me.

-- SS