Is this new treatment any good?

When a new drug is launched we have to ask if it is one that we should add to our personal formulary. Does it represent a real advance?¹ A great deal of money has already been invested in this new product and the marketing team will ascertain that it is presented in the most favourable light to enhance sales. This is not to reject the possibility that it may represent a very substantial therapeutic advance, or a more moderate one, nor is it to condemn the pharmaceutical industry for commercial perspective, but "all that glistens is not gold".²

Some people like to wait until a new drug is recommended by consultant colleagues or more widely used before they accept it. This is really passing the buck as until that time, patients are possibly being denied a better treatment. A substantial amount of evidence will already have been presented before the drug can have a licence but it is the duty of the doctor to assess the evidence to decide if this appears to be an innovation that is of benefit to patients.

Amongst the questions to be answered are:

• Is this drug a new concept or a variation on a theme? Perhaps it is a "me too" in a lucrative but highly packed niche such as yet another NSAID, another ACE inhibitor or a new PPI.
• How effective is it claimed to be?
• Does it represent a significant advance on current options?
  • Is it more effective?
  • Has it less adverse effects?
  • Is it cheaper?
  • Is the dosage or route of administration more manageable?
• Does it represent value for money?

Any new drug will be promoted as a significant advance but it is important to remember that at an early stage, evidence will be limited. A number of new drugs have been found to have significant adverse effects after launch and have had to be withdrawn and most new drugs are more expensive than older ones. Sometimes it may appear that a new product is being launched by a company to replace one of its own existing products as the patent is about to expire.

Some degree of regulation of pharmaceutical products dates back to the time of Henry VIII but it may be surprising to learn that a more rigid system of control was introduced as recently as 1971.³ There has been The Committee on the Safety of Medicines (CSM), the Medicines Control Agency (MCA) and the Medical Devices Agency (MDA) and in April 2003 both were replaced by the Medicines and Healthcare products Regulatory Agency (MHRA). Regulation of medicines is now on an EEC basis but substances that are not classified or licenced as medicines have little or no control.

Before a medicine can be sold in the UK, a number of licences are essential.⁴

The product must have a licence called a "marketing authorisation". This was formerly called a product licence. The companies that are involved in all stages of the manufacture and distribution of the product need to have manufacturer's and wholesale dealer's licences. New products which are still in development require a licence before they can be tested on human subjects. This is a clinical trial authorisation.

Applications for clinical trial and marketing authorisations are assessed by medical, pharmaceutical and scientific staff at the MHRA. Applications for marketing authorisations come mainly from the pharmaceutical industry but anyone with the necessary supporting data may apply.

It is a complex task to evaluate the benefits versus possible adverse effects of a medicine. Evaluation takes into account:

• The nature of the active ingredients
• The dosage form, such as tablet or liquid
• The nature of the disease or condition to be treated
• The effective dose that needs to be given
• The type of patient, for example, age, sex
• The duration of treatment

A high risk to benefit ratio may be acceptable in the treatment of terminally ill patients where the quality of life might be enhanced but a very low risk to benefit ratio is expected in the treatment of patients with self-limiting diseases, for the purpose of prophylaxis including vaccines and for those requiring life-long treatment for their illnesses.

The assessors frequently take independent expert advice on matters of safety, quality and efficacy from advisory bodies or independent experts.

It is clear that before a product can be awarded a licence, that there has already been a considerable amount of questioning by very erudite people. However, the questions that they can ask are limited in nature.

There must be evidence of efficacy. This will almost certainly involve double blind randomised controlled trials. They may not be placebo controlled unless the substance is to treat a condition that currently has no effective treatment. It is more relevant to compare it with current treatments, especially if there is a gold standard treatment. For example, we do not want to know if a new drug for asthma is better than nothing. We want to know how it compares with existing treatments. The size of trials will vary according to the potential size of the market. It may not be possible to recruit many subjects for a rare disease but for treatments that are given to fundamentally healthy people, such as antibiotics and vaccines, larger trials will be required.

There will have been testing for toxicity and perhaps for common drug interactions but the rarity of these may be such that they are not apparent before the launch of the product and it is only when it is used on a large number of people outside of a controlled situation that these problems become apparent. It should be assumed that a new product is unsafe in pregnancy and lactation as there will not be positive evidence to the contrary.

A product licence will be issued if the substance appears to be effective and safe. It does not have to be better than existing treatments and cost is not an issue.

It is still up to the individual doctor to look at the evidence and to assess the potential value of the new product. This requires a critical examination of published papers. For a new product, systematic reviews and meta-analyses are unlikely to be available.

Sources of Evidence

• Where are the studies published? Most research of any value will be published in peer reviewed journals and will have been refereed by other experts in the field who will hopefully have selected only worthwhile studies for publication.
• Any data that is referenced as "in house data" or "data on file" or "personal communication" should be given much less weight as a source of evidence, as the data will not necessarily have been scrutinised objectively by any other party.
• Who are the authors? Less weight may be given to a paper published by the company's own employees, although to publish anything that was frankly fraudulent would be most unwise. More credence may be given to independent authors although pharmaceutical companies can "ghost write" papers. Also look at the declaration of interest in terms of funding. Those with the funds can exert a degree of influence.

These points may add a degree of scepticism to the analysis but even the most cynical must accept that a firm that falsifies data is courting disaster. "A cynic is a man who knows the price of everything and the value of nothing."⁵ Any "massage" is more likely to be a matter of how the data is presented. As Benjamin Disraeli once said, "There are 3 types of lies. Lies, damned lies and statistics."⁶

Study Design

What type of study was done? The "gold standard" for measuring the effectiveness of a drug is the randomised controlled study (RCT).
These types of study may be:

• Single blind when the patient does not know which medication is being taken
• Double blind when the patient and the researcher do not know which medication is being taken
• Double blind, crossover when the patient will be taking active compound A and a placebo B, or new compound A and existing compound B, and then cross over to receive the other compound or placebo. Neither the patient nor the researcher will be aware which compound the patient is taking at any one time. In a typical design, subjects may be taking either A or B in 3 months blocks over a year and half of them will have drugs in order ABBA and the other half in the order BAAB. Crossover studies are very useful because the subject is, in effect, his own control.

The design of the study can influence the results and it is important to consider several points when looking at study design. Unfortunately, not all papers are entirely forthcoming about the details of study design. On the other hand, any study involving human subjects will require approval by an ethics committee, and they will ask some searching questions about methodology. A poorly designed study is unethical as it asks patients to participate in a study that is useless or worse, in that it may mislead. Sometimes research may be "outsourced" to other countries with less vigorous ethical requirements.

• Is the design biased in terms of patient selection, inclusion or withdrawal?
• Are all withdrawals accounted for and adverse reactions compared between the two groups?
• Is the study adequately powered? This means, are there a large enough number of subjects in the study? It is possible to calculate the number required before embarking on the study. The person reading the paper is unlikely to check such calculations but will simply ask the question, "Do the numbers look reasonable or are they rather small?"
• Is the primary outcome that has been chosen biased in favour of either treatment?
• Has the method used been validated in other studies?

When calculating the necessary number of subjects for a trial, there ar two questions to ask:

• What is clinical significance?
• What is statistical significance?

A small clinical benefit is easier to demonstrate than a large one although the numbers needed in each group to demonstrate a smaller benefit are larger than for a more ambitious difference. Statistical significance is often taken as P≤0.05 that means that there is a 1 in 20 chance that the difference between the two group was purely chance. P<0.01 means a 1 in 100 chance and P<0.001 a 1 in 1000 chance. More impressive P values need larger groups.

When assessing the validity of the results it is important to answer several questions:

• Was there a true randomisation of the patients?
• Were all the patients entered into the trial accounted for at the end?
• Were the patients analysed in the groups to which they were randomised?
• Were the groups made up of similar patients?
• Were the two groups treated equally in all ways except medication?
• Have adverse events been recorded for both treatment groups?
• Are the results significant?

Some of these questions are summed up by the term "analysis by intention to treat". This is a very important concept. Where it does not occur it may not be apparent from the paper. There are examples of cancer trials in which patients who were too ill to receive the active intervention were put in the control group. It is unsurprising that they fared worse. There was a paper that examined the effect of an education package for young people to reduce the risk of unwanted pregnancy. Teenagers were split into 2 groups. One was offered the education package, the other was not. However, those who were offered the education but refused it were allocated to the control group. This is unacceptable.

Accounting for those who started but failed to finish is important as is a comparison of adverse effects. Otherwise the issue of adverse events is not being adequately addressed.

Are the results statistically significant? Are the results clinically significant? A very large trial of a hypotensive agent may be able to demonstrate a statistically significant fall in blood pressure but that fall may be so small that the clinical benefit is negligible. A trial may have shown that a drug is better the placebo but it has not been compared with existing treatment. The benefit over placebo for the new drug may be better than for the existing one but that is an invalid comparison. Only a "head-to-head" comparison of the two treatments may be used to assess which is better.

Confidence Intervals

Although results are given as an absolute number, there is a degree of uncertainly about the "true" result and this is where confidence intervals⁷ (CI) are so important. Let us examine two trials that appear, superficially, to have identical results. One is a small trial comparing A with B and the respective scores were 40% and 50%. Calculation of confidence intervals on the small sample show that there is a 95% probability that the "true" result lies somewhere between ±7% for each group. This means that there is a 95% chance that the "true" result for A is between 33 and 47% and the "true" result for B is between 43 and 57%. There is also a 5% chance that the actual figure lies further out. In this case the result for A may be as high as 47% and the result for B may be as low as 43%. Hence there is overlap between the confidence ranges and so the difference between A and B is unproven. Another trial has apparently similar results with A and B at 40 and 50% respectively but, being larger, the CI is only ±3%. This means that there is a 95% chance that A lies between 37 and 43% while B lies between 47 and 53%. There is no overlap and so the difference between A and B has been demonstrated.

As stewards of the resources of the NHS, clinicians are required to ascertain that treatments give value and do not squander resources. The whole of the NHS, including primary care, has a finite budget and money that is spent in one direction is not available to be spent elsewhere. Value for money is not an issue that is addressed by the licensing authority but it is a matter that may be considered by NICE.

Assessing the value for money that a treatment provides asks a number of questions:

• Does the treatment work? If it does not achieve its objective it is a waste. A better question would be, is there a reasonable chance that it will achieve its objective?
• No treatment is without any risk and so the question is whether the ratio of risk to benefit is acceptable.
• Is the outcome worth the outlay? Suppose that an antiviral agent slightly shortened the duration of chicken pox in children. Is the cost worth the marginal benefit? The value of that same agent in ophthalmic shingles may be an entirely different matter.

Hence it is not simply a matter of price. It is a matter of cost, degree of benefit, risks and the value of the benefit achieved.

A tool that health economists sometimes use is quality adjusted life years, usually abbreviated to QALY. This indicates the improvement in quality of life over a number of years as a result of an intervention. It tends to be used for expensive but well established procedures such as total hip replacement or coronary artery bypass grafting and will not be available for a new drug.

Few, if any drugs will have the desired result of improving outcome for every patient. A calculation can be made of the number needed to treat (NNT)⁸ to improve the outcome for one patient. Where adverse events are being considered the same calculation can be applied to numbers needed to harm (NNH). NNT is often used for drugs to treat cancer or heart failure as it is important to have a very "hard" outcome such as death. This is far more valuable than "soundbites" for public consumption such as "it halves the death rate".

Let us compare the NNT for two drugs, one of which cuts the death rate by 50% whilst the other reduces it by quarter. Which is better?

In the first case the death rate falls from 2% to 1% with the new treatment. This means that of 100 people given the new treatment, 98 will survive but they would have survived without it. One will still die and one who would otherwise have died will survive. Hence the NNT is 100.

In the second scenario, the new products reduces the death rate by just 25% but from 80% to 60%. This means that of every 10 people given the new treatment, 6 will still die, 2 will survive who would have survived before and 2 who would have died will now survive. Hence the outcome is changed in 2 of 10 or 1 in 5, giving a NNT of 5.

NNT gives a good objective assessment where it is appropriate to use it and even NNT should really be accompanied by confidence intervals.⁹ Many people are surprised at the size of numbers, even for very useful interventions. Whether either, both or neither of these hypothetical drugs represents value for money depends upon a number of subjective assessments including the age of the patient and the expected quality and quantity of life to be won.

Let us look at another hypothetical scenario. A patient requests your help to encourage the PCT to fund a treatment for her cancer. She says that it halves her chance of dying and adds that £20,000 does not seem a high price to save a life. You find that the cost of treatment is actually £20,000 a year and the duration of treatment is usually 2 or 3 years. This makes the cost of a course of treatment £40,000 or £60,000. Let us call it £50,000. The risk of recurrence of her cancer is reduced from just under 20% to just over 10%. Let us call this a reduction from 20 to 10%. On this basis the NNT to prevent one recurrence is 10. This makes the cost of preventing one recurrence £500,000. Not everyone with a recurrence will die but let us assume a poor outcome with a 50% mortality from recurrence. This means that two recurrences must be prevented to save one life. It would appear that the cost of saving a life is not £20,000 but more like £1 million.

If the patient gets her drug and has no recurrence, she will probably attribute it to the drug but there is 8 times the chance that the drug had no effect on outcome. If she does not get it and dies, failure to prescribe the drug will be held to blame although there is only a 50% chance that the drug would have affected the outcome.

In most case, and especially in general practice, the issue is not one of such an emotive nature as cancer therapy but the questions to be answered is not, "does this product work?" but "does this product provide value for money in terms of offering significant advantages over existing products? If it is more expensive, is it worth the extra money?" This involves the balance of many factors and an ability to make some clarity from data that is presented. It is important to be objective about claims which might not be false, but they may mislead. The final decision rests upon whether you think that this drug is right for your patient.¹⁰