Editor’s Note: This article was originally made available to our Pα+ subscribers. Given the interest in this topic, we’re briefly lifting the paywall to allow a wider readership. If you would like to read all of our content, please consider subscribing to Pα+. ∎
Yesterday, the Institute for Clinical and Economic Review (ICER) published its draft evidence report on MDMA-assisted psychotherapy (which it abbreviates to ‘MDMA-AP’) for the treatment of PTSD. ICER is a nonprofit organisation that, in its own words, collaborates “with stakeholders across the US healthcare system [to] assess how much better a health intervention is, and what a fair price could be.”
The organisation was keen to remind readers that the preliminary draft “marks the midpoint of ICER’s eight-month process of assessing these treatments.” As such, its findings are not final.
While ICER Chief Medical Officer David Rind noted that “PTSD can be a severe condition affecting nearly all aspects of an individual’s life” and that “current therapeutic options are insufficient for many people with PTSD”, he tempered expectations. “While MDMA-AP may be a promising therapy for PTSD,” he continued, “functional unblinding in the clinical trials and additional concerns around trial design and conduct leave many uncertainties about the balance of benefits and harms.”
“It will be incumbent on regulators with complete access to primary data”, he continued, “ to carefully assess whether MDMA-AP has been proven safe and effective.”
Our editor, Josh Hardman, reviewed the report overnight to pull out some key messages. If you have the time, however, you can take a look yourself (PDF).
In short: ICER is pessimistic regarding the level of clinical evidence provided by Lykos Therapeutics, highlighting various concerns that range from trial design and conduct issues through to potential ethical and safety problems. While the organisation is generally relaying concerns it has heard from parties with ‘firsthand’ and ‘secondhand’ experience or knowledge of the trials, it’s hard not to see certain sections as scathing.
ICER goes on to deem the clinical evidence to be of ‘low certainty’, which means it’s ‘insufficient’ in terms of feeding into a determination of comparative clinical effectiveness. That means that ICER’s cost effectiveness analyses are all exploratory. While that exploratory analysis does find MDMA-AP to be ‘less costly, more effective’ versus placebo, ICER is keen to remind readers that its analysis is somewhat hamstrung by its low certainty in the clinical evidence.
ICER's Clinical & Ethical Concerns re: Lykos’ Studies
If the results reported by Lykos in its two Phase 3 studies “are reflective of the expected outcomes if MDMA-AP is administered broadly to people with PTSD”, the executive summary reads, “it would be an important addition to treatment options for PTSD”.
However, and this is the kicker, ICER has “substantial concerns about the validity of the results.”
These concerns, ICER claims, surfaced through its engagement with stakeholders. Given that it felt the aforementioned concerns could “potentially affect the interpretation of the evidence for MDMA-AP”, ICER conducted interviews with those having “firsthand or secondhand knowledge of the trials and related events.”1 The organisation says it has received “relatively little input” from Lykos to date, and so does “not have direct information on their reflections on these concerns.”
Broadly, they can be bucketed into trial design and delivery-related concerns and ethical or misconduct concerns.
Trial Design, Validity
One of the first things that ICER picks up on, which it “heard from multiple people” with regard to, is the choice of CAPS-5 as the primary outcome measure.
Where PTSD patients have experienced more than one traumatic event, CAPS-5 has them pick an ‘index traumatic event’. Put simply, they are instructed to choose the most traumatic single event and their symptoms are measured against that trauma.
The issue with this, according to people ICER spoke with, is that it might have “failed to capture participants[’] overall response to MDMA-AP.” ICER “repeatedly heard about participants experiencing improvement or resolution in the single trauma … while new issues became overwhelming”. “We heard this from multiple people in ways that leave us with no doubt that this occurred”, ICER states, clarifying: “that is, that there were participants who improved on the CAPS-5 outcome while worsening overall”.
ICER further notes that the scale “was developed and validated using a predominantly male military population”, adding that it is therefore “unclear how well it captures changes in PTSD in a population of women and those who may have experienced repeated physical and/or sexual traumas.”
ICER does acknowledge, however, that the measure “is considered the gold standard for PTSD”.
Aside from concerns about the primary outcome measure employed in the study, ICER also presents questions about the standardisation of the ‘assisted psychotherapy’. “We heard that therapy was not well standardised in the MAPP trials”, ICER noted, meaning “it is hard to be certain how to generalize from the results.” The organisation did accept that this problem “exists in many trials of psychotherapies”, however.
ICER also notes that the MAPP1 study protocol permitted patients to “request additional integrative visits”, with an “uneven distribution in the number of participants who opted” to take those up: i.e., patients in the MDMA arm were more likely to opt-in to these additional sessions than those in the placebo arm.
Perhaps unsurprisingly, ICER also reported concerns around functional unblinding.
94% of participants in the MDMA arm of MAPP2, ICER notes, correctly guessed as such; while 75% in the “Lykos-specific non-assisted psychotherapy” (LSNAP) arm also guessed their assignment correctly.
“Because of the effects of MDMA, the trials were, essentially, unblinded”, ICER writes.
Elliot Marseille, Director of the Collaborative for the Economics of Psychedelics told Psychedelic Alpha that it’s “perfectly appropriate to mention the blinding problem”. But, if it becomes standard to deem it a “disqualifying concern”, Marseille added, “no psychedelic therapy could ever be endorsed by ICER”. Nor could chemotherapy for cancer, “let alone hip replacement surgery”, Marseille quipped.
Ethical and Safety Concerns
Elsewhere in the Patient and Caregiver Perspectives chapter of the report, ICER recounts various other “concerns about trials of MDMA-AP”. Such concerns are noticeably absent from ICER’s reports on other mental health interventions we reviewed such as esketamine for treatment-resistant depression.
However, a key element of the organisation’s Value Assessment Framework is to incorporate patient experiences and values in its reports. While that’s likely to primarily focus on patients with the disease at hand, it’s fair to extend that to those in the trials.
One of the key thrusts of ICER’s report here is that many individuals involved in Lykos’ studies are true believers in psychedelics’ therapeutic potential.
“The pool of therapists and, in some cases, trial participants appears to have pulled heavily from the existing community of those interested and involved in the use of psychedelics for possible psychological benefits”, ICER writes.
This leads to at least half a dozen issues, the report claims. To summarise what ICER says it heard from stakeholders:
- The community engages with psychedelics “more like a religious movement than like pharmaceutical products”, with these feelings “sometimes inculcated in patients participating in the trials.”
- Unblinding of therapists “was particularly likely given their experience” with psychedelics. What’s more, 40% of patients had prior MDMA experience.
- ICER says it heard “repeatedly” that there was pressure “to have the results of the MAPP trials be favorable.” This led, according to ICER’s respondents, “to some participants feeling pressured to report good outcomes and suppress bad outcomes when they were in the MDMA arms of trials.” Some participants felt they “could be shunned if they reported bad outcomes”, if they were ‘in the community’, or that it could deny future patients the option of MDMA-AP. This is a “particular problem in people receiving MDMA”, ICER heard, “as it makes them particularly suggestible and susceptible to context.”
- “Patients in the trials included therapists who had worked in the space”, ICER writes, which is “unusual and heightens concerns about pressures to tailor reported results.”
- ICER says it heard, firsthand and secondhand, about “extremely severe negative outcomes” that “do not seem to have been attributed to the treatment by the trial researchers.” “Some patients were told…their negative outcomes were evidence they were responding appropriately”, while others were allegedly “prevented from entering the long-term follow-up study and felt this was done to keep these negative outcomes out of the data set.”
- Finally, ICER recounts an event it heard of whereby, “after the trial was completed and a participant was struggling, that they were told to take their own supply of MDMA at home.” If this were just one isolated event, ICER argues, “it shows the clear breakdown of blinding, the inclusion of participants who were anticipated to have access to their own supply of MDMA, and a disregard of good clinical trial practices.”
Beyond these alleged trial conduct-related issues, ICER also recounts other ethical and safety concerns. On the topic of safety, ICER notes that “there is no question that … sexual boundaries … were severely crossed with at least one patient.” They also note that some experts emphasised that “certain cardiovascular and long-term neurological impacts may be challenging to detect within the confines of short-term studies”.
ICER acknowledges that this “disconnect” between reports it heard from patients and the published results of the studies could be “due to the timing of evaluation measures rather than deliberate attempts to suppress these reports”, however.
In short, however, ICER paints a pessimistic picture of the design but also conduct of Lykos’ studies, which sees issues like functional unblinding compounded by the potential biases of those involved.
“Insufficient” Clinical Evidence, According to ICER
“Ultimately”, ICER said in concluding its section on patient and caregiver perspectives, “based on our limited sample of participants, we are left very uncertain about the frequency of harms and benefits, the reliability of reports of benefits, and the generalizability of MDMA-AP to those outside the community.”
The organisation notes that there are strong beliefs all-round: both in the apparent MDMA-advocate camp, but also among “those with concerns about the MAPP trials”. That should be considered when reviewing the information they present here, they note.
These “very strong feelings” among a subset of both “proponents and skeptics” of MDMA-assisted therapy “are unusual in most assessments of medical interventions.”
Given that “ICER is not able to assess the frequency of misreporting of benefits and/or harms and thus the overall balance of net benefit with MDMA-AP”, the organisation concludes (in its preliminary report, at least) that the level of certainty in the evidence is “insufficient”.
That level of certainty rating was given to both MDMA-AP itself, as well as comparing it with trauma-focused psychotherapies.
ICER’s method, like that of similar organisations, is grounded in a review of clinical evidence. Of course, that forms the basis of the most important comparative inputs to a cost-effectiveness analysis. This hamstrings ICER’s ability to produce more confident analyses.
Given that fact, ICER only presents exploratory analyses of MDMA-AP, which take the results of the two Phase 3 studies at face value.
Exploratory Cost-Effectiveness Analysis
As mentioned, ICER only presents an exploratory analysis, as it deemed the clinical evidence to be “insufficient”. What’s more, the comparator group (LSNAP) was “an unproven therapy”, making it tricky to know how it compares to trauma-focused therapies.
ICER puts the cost of intervention at $23,117, which is around double that seen in the only previously published cost-effectiveness estimates of MDMA-assisted therapy (Marseille et al., 2022; 2020; as well as a more recent analysis of group psychedelic therapy, 2023).
The reason ICER’s estimate is much higher than Marseille et al.’s 2022 and 2020 studies is because it attempts to include drug costs, which have not yet been disclosed by Lykos. ICER used IPD Analytics’ range of $5,000-$15,000 per three-session treatment course, landing on the midpoint of $10,000 for its exploratory analysis.
The exploratory analysis does find MDMA-AP less costly and more effective than placebo.
However, given the fact that it’s exploratory, little further information—such as threshold prices—is presented.
Marseille told Psychedelic Alpha that this is “the same basic conclusion” as reported in the work he conducted with colleagues. That is: “given results as reported in the trials, MDMA-assisted therapy both costs less and is more effective than the comparator.”
ICER also points to potential logistical challenges in delivering MDMA-AP. “Implementing MDMA-AP would be challenging for health care systems like the VA, which already faces increasing mental health demand and staff shortages”, it noted. “The substantial therapist time and specialized training required could make widescale MDMA-AP adoption difficult”, ICER continued.
Later, the report states that the multiple prep, dosing and integration sessions with “at least two therapists leaves questions about feasibility of MDMA-AP administration.”
ICER’s Influence
Unlike many other countries, the U.S. (“largely for political and ideological reasons that the government should not be involved in drug coverage decisions”, according to Cherla et al., 2020; see also Mulligan et al., 2020) doesn’t have a national health technology assessment (HTA) program. HTAs seek to apply a transparent and scientific method to the appraisal of health technologies such as drugs and tests. This appraisal is often used to inform decision-makers, such as state-funded healthcare systems, regarding coverage.
One such example is the National Institute for Health and Care Excellence (NICE) in the UK, which effectively informs which health technologies are used within the country’s National Health Service (NHS).
While ICER “does not have a formal role for evaluation or reimbursement”, its analyses on value-based pricing and budget impact “are being increasingly used by private payers to inform price negotiations with pharmaceutical companies” (Cherla et al., 2020).
According to a 2017 survey of 99 U.S. payors by Lising et al., 59% of payors indicated that they or their organisation had used ICER reports as part of their evaluation process. A more recent survey, conducted in 2022, saw that number jump to three-quarters of respondents, suggesting ICER’s increasing salience among payors (see also Faraci et al., 2022).
All this to say: there’s substantial evidence that ICER is influential in coverage decisions and pricing negotiation for payors, even instead of a formal, national role.
Beyond payors, ICER’s reports are also widely read by practitioners, researchers, patient advocacy groups, and so on. Given that we’re the first to publish an article on this report, there’s little reaction to comment on at the moment; though some researchers and commentators have weighed in on Twitter.
Stanford researcher Michael Ostacher wrote on Twitter, “I don’t know what bearing this will have on FDA.” “Hopefully it will give them pause”, replied UCSF’s Josh Woolley, who added that the report “matches my experiences as well”.
Jules Evans, Director of the Challenging Psychedelic Experiences Project and prolific writer on such matters, shared a thread on the platform. According to Evans, one therapist told him that “the critique of being unable to maintain the blind due to the subjective effects of MDMA is completely fair”, adding that “this limitation is shared by all psychotherapies, not just psychedelics”. “But”, the therapist continued, speaking to Evans, “the claim ‘that therapists encouraged favorable reports by patients and discouraged negative reports’, based on completely biased sources, undermines the decades of effort that we all as therapists have put into trying to fairly evaluate this potential treatment’”.
Next Steps
As we have reported previously, Lykos secured a Special Protocol Assessment (SPA) with the FDA in 2017. As such, key aspects of the Phase 3 program were agreed in advance with the regulator. (As ICER points out, for example, there is “no agreed upon definition of a clinically meaningful response” on CAPS-5, but Lykos collaborated with FDA via an SPA to agree that a reduction of 10 points or more would be clinically meaningful.)
If the FDA proceeds to approve MDMA-assisted therapy, this ICER report offers a look at how other bodies might view the evidence base for the intervention, and thus its acceptability among providers and payors.
Both the Draft Evidence Report and the Draft Voting Questions are now in a period of public comment. Formal comments may be emailed to publiccomments@icer.org no later than 5 pm ET on April 22nd, 2024. See the webpage for more information on the appropriate format.
Note: We reached out to Lykos for comment, but none was received at the time of publication. On Friday, 29th March, Lykos provided the following statement:
Our Phase 3 MAPP1 and MAPP2 double-blind, placebo-controlled studies were designed in partnership with the FDA through a Special Protocol Assessment Process (SPA) and executed with independent raters to mitigate the risk of functional unblinding and ensure the reliability and validity of efficacy endpoints. We stand behind the design of the clinical trials, the integrity of the investigators, and are grateful to the patients who participated in the clinical trials. We are working with the FDA as they review our new drug application (NDA) for investigational midomafetamine capsules (MDMA) used in combination with psychological intervention (MDMA-assisted therapy).
- ICER specifically mentions a “podcast” that discussed concerns around the trials, and thereafter spoke with “a small number of people that included people involved with the podcast, subjects in the trials, and a therapist who had been involved in one of the trials.” The podcast in question is the Power Trip series which appeared on the New York Magazine’s Cover Story podcast; co-produced with Psymposia.