The Lack of Clinical Quality Guidelines Causes Underestimation of Efficacy in Sham Controlled Acupuncture Trials
A serious flaw in the design of acupuncture research studies has been contaminating “sham controlled” clinical trials for decades. Unfortunately, this problem has been largely unacknowledged having only recently been questioned by researchers in China (1) even though it should be readily seen as rendering such trials incapable of answering the fundamental question they are designed to answer: How much of acupuncture’s clinical effects are due to placebo effects or the real but “nonspecific” effects of needling anywhere?
The flaw I am referring to is not the one that has been widely (and rightly) criticized – that the sham controls are not inactive and may in fact provide a therapeutic effect similar to the traditional acupuncture points being studied(2-5). Nor am I referring to the more recently raised question of trials being “underpowered” by small sample sizes although this may also be a valid concern (6). What I am referring to is a serious oversight in the design of many trials’ “real/verum/active” arms. Simply put – the lack of vetted clinical quality guidelines to guide the active arms’ clinical protocols means there is no way of knowing if the real acupuncture is being afforded the chance to be as effective as it could be. To further illustrate this problem, it is helpful to consider a term used in some legal circles: Maximum Therapeutic Benefit (MTB).
MTB is a term used in the medical insurance industry and in systems such as the U.S. Social Security Administration. Its meaning is just as its name implies – a state where a patient/client has achieved the maximum benefit that can be reasonably expected for a given therapy. Determining if MTB has been achieved is an important factor as most medical insurance plans are obligated by law to cover medical services until MTB is reached and establishing monetary awards for permanent disability is typically delayed until MTB in achieved in the treatment of the disability.
Unless reasonable steps are taken to insure MTB can be achieved in the active arm of a sham controlled trial it is impossible to estimate the real/verum acupuncture’s efficacy even if one could insure the sham control is inactive and the sample size is adequate. The lack of clinical quality guidelines can lead to the real acupuncture being administered in a suboptimal manner lowering its effectiveness rates below its true potential in a manner similar to using a suboptimal dosage of a drug in a placebo controlled drug trial. There is good reason to believe that some, perhaps many, of the trials that found real acupuncture to be only as effective as sham may well be invalid because the real acupuncture was carried out in a suboptimal clinical manner that did not to allow it to achieve MTB.
Why do I say there is “good reason” to believe this is a serious flaw in these trials? Because of two unresolved, glaring discrepancies seen across different trials; the first involving effectiveness rates and the second involving the number and frequency of treatments given. These two marked discrepancies are especially (but not exclusively) seen in trials conducted in East Asia vs. those conducted in the West.
Trials conducted in countries such as China, Korea, Japan, and Taiwan consistently show real acupuncture having high effectiveness rates in the 75%-90+% range and to clearly outperform sham controls (7). Many acupuncturists claim similar effectiveness rates within their practices and a recently published patient satisfaction survey taken of 89,000 U.S. acupuncture patients found similar rates of success in the treatment of their primary complaint (8). Sham controlled trails done in the West, however, tend to show a mixed set of results. Some trials find higher effectiveness rates similar to trials conducted in East Asia with real acupuncture outperforming sham while others report effectiveness rates in the 40%-50% range with real acupuncture not outperforming sham. The higher the effectiveness rates, the more likely the real acupuncture will outperform the sham as sham rates are rarely found to go beyond the 50%-60% range and are usually in the 40%-50% range. The critical question to ponder is what factors are responsible for the much higher effectiveness rates or, conversely, what factors are responsible for the comparatively low effectiveness rates seen in a significant portion of Western trials?
A common refrain from acupuncture critics is that “higher quality” sham controlled RCTs tend to find real acupuncture to not outperform sham while trials that show real acupuncture to outperform sham are of lower quality. Some have gone so far as to say that the consistently high positive outcomes (effectiveness rates) of Chinese trials mean those trials cannot be trusted (9-10). Yet the trial “quality” these critics are referring to is limited to the methodological quality of the trial design/implementation not the clinical quality. Of course, few, if any of these critics have the expertise to judge the clinical quality of the active arms of acupuncture trials.
When it comes to clinical quality for acupuncture therapy, while there can be many factors involved, experienced acupuncturists agree that effectiveness rates will be impacted by the frequency and numbers of treatments given over time, otherwise, why bother to ever do more than just one treatment? It is also common knowledge within the acupuncture community that in China, acupuncture treatments are typically done on a daily or every other day basis in the first stage of the treatment process and that dozens of treatments may be utilized in total, especially in the treatment of chronic conditions. Yet the vast majority of acupuncture trials conducted in the West use far fewer treatments in total and at a substantially less frequent interval. It seems quite reasonable to wonder if this might be a factor in the relatively lower effectiveness rates that plague Western trials.
Trials in the West often begin treatment at once a week and may only undertake 4 to 10 treatments in total. Some trials only use 1-3 treatments! Rarely will Western trials employ more than 2 treatments a week or beyond 20 treatments in total. The higher treatment frequencies/total numbers employed in East Asian trials are utilized by highly experienced acupuncturists to help achieve the highest effectiveness rates (MTB). Yet, despite this great disparity in treatment frequency/total numbers, no clinical rational is offered to explain why Western researchers have confidence that their treatment frequency/total numbers are clinically justified. Researchers often go to great lengths to explain why their trial design is adequate – blinding methods, sample sizes, control protocols, etc., but the rationale behind clinical protocols is given far less consideration. It would be hard to imagine such a great disparity in drug dosages being accepted in drug trials without researchers offering some clinical rational for using those different dosages.
There are some rare exceptions. Two trials on osteoarthritis of the knee, one done in the U.S.A. and the other in Israel (11-12), both utilized a higher treatment frequency (twice a week for 8 weeks) and total treatment numbers than is usually the case in Western acupuncture trials . Both of these trials found that the effectiveness rates for the real acupuncture did not start to significantly surpass the sham until 12 weeks – a longer timeframe than most Western studies allow for. Other studies have found similar trends with higher numbers of treatments leading to higher effectiveness rates (13-14).One has to wonder how many trials that did not find real acupuncture to outperform sham may have done so if they had utilized more frequent treatments carried-out over longer periods of time.
Until it becomes the norm to place as much emphasis on clinical quality as is placed on trial methodology quality, research on acupuncture will not give an accurate picture of that therapy’s efficacy and true clinical potential.
Some researchers may feel justified in the use of these lower treatment frequency/total numbers in their trials because many acupuncturists in the West treat their patients with similar frequency/total numbers. This type of research design, it could be reasoned, is mimicking actual practice. That may (or may not) be true. However, sham controlled acupuncture trials are not being promoted as a means to test acupuncture’s efficacy based on how it tends to be practiced by some acupuncturists in the West where cost concerns often constrain optimal treatment numbers. They are being promoted as a test of acupuncture’s actual efficacy and the stigma of trials showing real acupuncture not outperforming sham has tainted acupuncture’s legitimacy and slowed its acceptance in the West.
I understand that some involved with the acupuncture research community may take the view that the focus for acupuncture research moving forward should be on directly comparing acupuncture to other therapies (comparative effectiveness) rather than sham controlled studies. I agree. In fact, I believe the focus should have been on comparative effectiveness all along. Still others would argue that even with the poor clinical quality trials having been done, acupuncture’s efficacy has been shown in several conditions to be at least equal to and sometimes better than other widely used therapies and that acupuncture is being unfairly held to a higher standard than other therapies. I agree with that also. However, there is a large body of research in the record, both sham controlled and otherwise, that has been conducted with poor clinical quality standards that very likely lowered true acupuncture’s actual effectiveness profile. This flaw needs to be acknowledged, implications debated, and corrections made to both help interested parties set the existing record straight and guide acupuncture research moving forward whether involving sham controlled trails or not.
Considering that sham controlled acupuncture trials have caused a significant negative impact on acupuncture’s acceptance, it is incumbent upon the acupuncture community – both researchers and clinicians – to address the issue the absence of clinical quality guidelines is having on such trials. Until it becomes the norm to place as much emphasis on clinical quality as is placed on trial methodology quality, research on acupuncture will not give an accurate picture of that therapy’s efficacy and true clinical potential. The following steps will help address this issue and the Acupuncture Now Foundation is very interested in working with those who wish to pursue dialog and action on these steps or similar steps.
Matthew Bauer began his full-time practice of acupuncture and Chinese Medicine in 1986 after several years of studying Taoist history and philosophy with a 74th generation Taoist Master. In addition to working with several acupuncture/Chinese Medical organizations, Matthew founded the Acupuncture Now Foundation in 2014. Serving as a consultant in the managed care industry since 1998, Matthew helped to create the first managed care acupuncture-based credentialing and utilization management guidelines and now serves on the Board of Directors of American Specialty Health Group, Inc. As a managed care consultant, Matthew took part in a consensus-building think-tank with a dozen experienced acupuncturists from the U.S., Mainland China, Taiwan, and Korea that developed clinical quality guidelines. That experience convinced Matthew of the need for experienced acupuncturists to work at building consensus on clinical issues to guide clinicians and researchers and further our understanding of this ancient healing system. Matthew has authored dozens of articles and two books and he has a passion for educating people about acupuncture believing it to be a responsibility of those who understand how it works to share this knowledge with those who don’t.