Is this new treatment any good?

When a new drug or treatment is launched, we have to ask if it is one that we should use. Does it represent a real advance? For a new drug, a great deal of money has already been invested in this new product and the marketing team will ascertain that it is presented in the most favourable light to enhance sales.¹
Amongst the questions to be answered are:

• Is it relevant to my patients or my practice?
• Is this drug a new concept or a variation on a theme? Perhaps it is a "me too" drug in a lucrative but highly packed niche such as another ACE inhibitor.
• How effective is it claimed to be?
• How safe is it - and how much of its safety is unknown?
• Does it represent a significant advance on current options?
  • Is it more effective?
  • Has it fewer adverse effects?
  • Is it cheaper?
  • Is the dosage or route of administration more manageable?
• How good is the evidence about it, and is any information missing?
• Do those promoting it have a conflict of interest?
• Are the benefits worth the risk of adverse effects (known and unknown)?
• Would I take this treatment myself?

Drugs

New medicines in the UK require a licence from the MHRA.² They have undergone clinical trials, but this does not mean that the product is safe. The number of patients tested at this stage is not enough to ensure that all adverse effects are discovered.

• For example: the diabetes drug troglitazone was withdrawn 3 months after it was released because of hepatic side-effects.³ Varenicline is a new anti-smoking drug: at the time of its release, only minor side-effects were known; one year later, warnings about possible depression and suicidal thoughts have been issued.⁴

Safety monitoring

Licencing decisions take into account the seriousness of the condition being treated. For example, more uncertainties or adverse effects will be “allowed” for an anti-cancer drug than for a painkiller used in minor illness.

New drugs carry a “black triangle”, indicating that they are intensively monitored medicines. Monitoring is continued until the MHRA is satisfied that the drug “works safely in large numbers of people”.²

Non-drug treatments

• Non-drug treatments also require evaluation, although blind or double-blind protocols may not be feasible.
• Treatments that appear intuitively supportive, are not necessarily either beneficial or neutral. For example, in post-traumatic stress disorder, negative outcomes were reported for some forms critical incident debriefing.⁵

Informed consent⁶

Good medical practice requires doctors should tell patients about:

• Options for treatment, including the option not to treat and uncertainties
• Details of the treatment
• Adverse effects:
  • Common adverse effects
  • Serious adverse effects (even if rare)
• Share information in a way that the patient can understand

New drugs and treatments have not usually been tested on pregnant or breastfeeding women or children, so there will be a lack of safety information for this group. In general, new drugs are not licensed for this group, and older treatments are the first choice. If a new treatment seems necessary, consult an expert, the National Teratology Information Service (for pregnancy)⁷ or the BNF for children (for children and breastfeeding).⁸

The authors and setting

• Who is presenting the evidence; do they have competing interests?
• Is this a reputable journal or conference?
• Is there peer review?
• Who are the authors? Bear in mind that “ghost writing” occurs,¹¹ and that "independent" research companies may also be vulnerable to commercial pressure.¹²

Study design

What type of study was done? The "gold standard" for measuring the effectiveness of a drug is the randomised controlled study (RCT).
This type of study may be:

• Single blind - the patient does not know which medication is being taken.
• Double blind - neither patient nor researcher know which medication is being taken.
• Double blind, crossover - the patient will be taking active compound A and a placebo B, or new compound A and existing compound B, and then cross over to receive the other compound or placebo. Neither the patient nor the researcher will be aware which compound the patient is taking at any one time, and the subject is, in effect, his own control.

Look at the methods section:

• Is the design biased in terms of patient selection, inclusion or withdrawal?
• Was there true randomisation of the patients?
• Is the study adequately powered? This means, is there a large enough number of subjects in the study? It is possible to calculate the number required before embarking on the study. The person reading the paper is unlikely to check such calculations but will simply ask the question, "Do the numbers look reasonable or are they rather small?"
• What outcome (end point) is used? Be cautious about:
  • Surrogate outcome measures, e.g. the use of lipid values as an outcome, rather than actual clinical events such as myocardial infarction. If surrogate end points are used, there should be good evidence that they are valid for the condition being studied.
  • Composite end points, e.g. combining non-fatal myocardial infarction and death. This can make a treatment appear more effective, but the two outcomes are clinically different and are best considered separately.

Validity of results

When assessing the validity of the results it is important to answer several questions:

• Were the groups made up of similar patients? Were the groups treated equally in all ways except the intervention?
• Were all the patients entered into the trial accounted for at the end?
• Have withdrawals adverse events been recorded for both treatment groups?
• Were the patients analysed in the groups to which they were randomised? This is an important concept. For example, if a cancer trial puts patients who were too ill to receive the active intervention into the control group, this unfairly favours the intervention group.

Interpreting the results

Are the results clinically significant?

For example, a trial of a hypotensive agent may be able to demonstrate a statistically significant fall in blood pressure, but that fall may be so small that the clinical benefit is negligible.

Are the results expressed in a meaningful way?

Results are often quoted as a relative risk reduction, which makes the effect appear more dramatic. However, absolute risk reduction and numbers needed to treat (NNT) are more useful in clinical practice, because they take into account how common the outcomes are, and therefore the size of the treatment effect in practice.

For example, here are the results of a hypothetical trial of a treatment (T) for a condition (C):

These results can be expressed in various ways:

• Treatment T halved the rate of death from C (relative risk reduction).
• Treatment T reduced deaths from C by 10%, or increased the survival rate from 80% to 90% (absolute risk reduction).
• For 10 people treated with T, one death from C will be avoided (NNT).

Note: the NNT is easy to calculate: it is the reciprocal of the absolute risk reduction, i.e:
NNT = 1/absolute risk reduction as a fraction.

Significance and confidence intervals

Statistical significance is often taken as P≤0.05, meaning there is a 1 in 20 probability that the difference between the two groups was due purely to chance.

Confidence intervals: these express the degree of uncertainly about the "true" result.

For example, suppose two trials appear, superficially, to have identical results. One is a small trial comparing A with B and the respective scores were 40% and 50%. Calculation of confidence intervals on the small sample show that there is a 95% probability that the "true" result lies somewhere between ±7% for each group. This means that there is a 95% chance that the "true" result for A is between 33 and 47% and the "true" result for B is between 43 and 57%. There is also a 5% chance that the actual figure lies further out. In this case the result for A may be as high as 47% and the result for B may be as low as 43%. Hence there is overlap between the confidence ranges and so the difference between A and B is unproven. Another trial has apparently similar results with A and B at 40 and 50% respectively but, being larger, the confidence interval is only ±3%. This means that there is a 95% chance that A lies between 37 and 43% while B lies between 47 and 53%. There is no overlap and so the difference between A and B has been demonstrated.

Example using risk values, NNT and cost

A hypothetical scenario: Mrs Z requests your help to encourage the PCT to fund a treatment for her cancer. She says that it halves her chance of dying and adds that £20,000 does not seem a high price to save a life. You find that the cost of treatment is actually £20,000 a year and the duration of treatment is usually 2 or 3 years. This makes the cost of a course of treatment £40,000 or £60,000. Let us call it £50,000. The risk of recurrence of her cancer is reduced from 20% to 10%. On this basis the NNT to prevent one recurrence is 10. This makes the cost of preventing one recurrence £500,000. Not everyone with a recurrence will die but let us assume a poor outcome with a 50% mortality from recurrence. This means that two recurrences must be prevented to save one life. It would appear that the cost of saving a life is not £20,000 but nearer £1 million.

New formulations of older drugs are often promoted by manufacturers, sometimes around the time that the patent on the older drug has expired. Bear in mind that some of these new drugs may benefit the manufacturer more than the patient.

"New for old" drugs include:

• "Me too" drugs which are similar to an existing product by another manufacturer.
• "Second generation" products such as escitalopram or desloratidine. These are isomers or metabolites of an existing drug, and may be marketed as "more effective" (because a lower dose is required on a weight for weight basis). In practice there is seldom any clinical benefit.
• New formulations such as slow-release preparations - these may sound useful, but in practice the advantage may be minimal.

A review of these products in the Drug and Therapeutics Bulletin concluded that "new for old" drugs are seldom worth the extra expense.¹³