In late March, the Institute for Clinical and Economic Review (ICER) published its draft evidence report on MDMA-assisted psychotherapy (which it abbreviates to ‘MDMA-AP’) for the treatment of PTSD. For more background on ICER and the original draft evidence report, please see our contemporary coverage: ICER Draft Evidence Report on Lykos’ MDMA-Assisted Psychotherapy Deems Clinical Evidence “Insufficient”.
Following the draft evidence report’s publication, a public comment period opened. Today, ICER published those public comments as well as its responses and a revised evidence report.
Here, we review some of the public comments, as well as ICER’s responses, before discussing notable changes between the draft and revised evidence report.
Lykos Submits Brief Comment, ICER Emphasises Lack of Engagement
Chief among the public comments was a brief submission from Lykos Therapeutics itself1 (page 1).
“Lykos is disappointed by the ICER draft report as it discounts the significant unmet medical need for patients living with PTSD, relies on a limited number of stakeholder perspectives, and focuses on areas squarely within the FDA’s purview on the evaluation of safety and efficacy of drugs in the United States,” the letter states.
Much of the short letter then emphasises the significant unmet need for patients with PTSD, arguing that the report “neglected to include meaningful input from veterans, first responders, survivors of domestic violence and physical trauma”, and so on, who “have valuable perspectives to share on the needs of those living with PTSD.”
This substantive focus is largely because Lykos appears unwilling to enter discussion with ICER on its study design choices—both via the public comment process, but also throughout ICER’s research and drafting of the report, according to the nonprofit. Given its investigational therapy is under review by FDA, “the company will refrain from discussing details of the application in public comments on the ICER report”, Lykos wrote.
ICER pointed out that this is a common situation for a sponsor to find themselves in, given it regularly carries out reviews right ahead of FDA approval. “Nearly every review that ICER conducts involves therapies under review by FDA”, it said in its response to Lykos’ comments. “In the vast majority of those cases, companies find that they are able to engage with ICER and discuss reasons for specific design decisions.”
While keeping schtum on specific details regarding its study design and conduct, Lykos sought to emphasise its engagement with the FDA, which has included the agreement of a Special Protocol Assessment and a Breakthrough Therapy Designation, both in 2017.
“Given the novel nature of midomafetamine treatment, Lykos has consistently sought and received feedback from the FDA throughout the development program”, the letter signed by CEO Amy Emerson reads.
Perhaps most contentiously, however, Lykos claims that “the issues in the ICER report were not raised during the three meetings held between ICER and the Lykos management team prior to issuance of the draft report.”
ICER disagrees. “We want to emphasize that ICER did reach out to Lykos throughout the review”, it said in its response to Lykos’ letter. “Unfortunately, Lykos declined to collaborate with us,” the response continues, “and did not provide us with any relevant contacts to engage with (clinicians, patient advocates, researchers in the MAPP trials, etc.).”
This doesn’t chime well with a complaint made in a letter submitted by a group of around seventy therapists and trialists involved in Lykos’ Phase 3 studies (page 6), a version of which was originally published on Psychedelic Alpha (see Exclusive: Therapists and Trialists from Lykos’ Phase 3 MDMA-Assisted Therapy Studies Push Back on ICER’s Critical Draft Report).
“One hundred and nine therapists and principal/co-investigators contributed to the Phase 3 trials of MDMA-AP for PTSD”, the letter submitted as a public comment reads, continuing: “To our knowledge, none of them were consulted before the preliminary report was issued.”
Responding to the authors’ dismay at the lack of inclusion of investigators in the drafting of its evidence report, ICER again pointed to Lykos’ alleged lack of engagement in the process. “It is typically the case that investigators in manufacturer-sponsored trials will only engage in ICER reviews when the manufacturer gives permission and connects those investigators with ICER”, it said.
“Lykos was informed of the concerns ICER was hearing”, it continued, adding that it “would have been happy to engage with any investigators or therapists that Lykos had connected us with as a way to work through these concerns.”
In an apparent effort to further clarify its side of the story, ICER took issue with Lykos’ claim that “issues in the ICER report were not raised” in any of three meetings that took place between the two groups.
In its response, ICER explains that the first meeting was in August 2023, before the start of ICER’s review, when MAPS PBC said they felt the ICER review was early. ICER thus delayed the review by a month.
A second meeting took place in September 2023, and a third in March 2024, according to ICER. In that latter meeting, the nonprofit claims “the only engagement Lykos was willing to provide was to suggest that the price for MDMA used in a prior MAPS cost-effectiveness analysis should not be assumed as a placeholder price.”
According to ICER, Lykos “chose not to engage” over this time period.
In its reply to Lykos’ comment, ICER also reproduced an email it says it sent to the sponsor in early December 2023, notifying it of its line of inquiry into the Cover Story podcast and follow-up interviews with its producers, offering Lykos an opportunity to respond to the allegations. “As with other outreaches from ICER to Lykos, we received no substantive response.”
“Given this, it seems disingenuous to suggest that Lykos was unaware that ICER would be trying to evaluate concerns about the MAPP trials or that ICER did not try to provide Lykos an opportunity to engage”, ICER added.
On the topic of cost-effectiveness analysis, Lykos said in its comment that it was conducting its own that is “leveraging real-world cost estimates from a retrospective claims analysis along with data from its Phase 3 and long-term follow-up studies, the latter for which data will be published in the coming months.”
Otsuka Weighs In, Echoes ICER’s Concerns
The only other drug developer to weigh on in the draft evidence report was Otsuka, with a two-page comment submitted by Kaan Tunceli, VP Global Value & Real World Evidence (page 4).
The comment sought to echo ‘concerns about trials’, calling out four “potential threats to the validity of the trials” that ICER discusses in its report:
- “The community of therapists and patients involved in the trial holding very positive beliefs about the therapy prior to participation
- Lack of standardization of the trial
- Possible pressure to suppress negative outcomes of the trial
- Safety concerns regarding follow-up care”
“Otsuka agrees that all these issues are concerning and may have skewed the results of the trial and efficacy/safety data of the drug”, it continues. “[W]e are hoping that some of these issues become more apparent as this process continues”, it added.
The letter goes on to agree with ICER on certain methodological challenges in its economic model, before providing three “longer-term considerations”.
The first argues that it will be “critical to determine if the harms of potential addiction outweigh the risks of this therapy in people with co-occurring substance abuse issues”, which may also have implications for the economic model.
The second argues that the follow-up period “was likely not long enough to fully understand the benefit/risk profile in the longer term”, while the third points to “the larger question of treatment durability.”
This isn’t the first time Otsuka has weighed in on the Draft Evidence Review for MDMA-assisted therapy. In November 2023, Otsuka was the only entity to comment on the Draft Scoping Document for the review, for example.
It’s worth noting that Otsuka is currently working with Lundbeck to seek approval for brexpiprazole (Rexulti) in combination with sertraline (Zoloft) for PTSD.
Study Therapists and Investigators Go on the Defensive
Two individuals who played prominent roles in Lykos’ studies, as well as the group of therapists and trialists mentioned above, sought to defend how the trials were conducted and reported.
Jennifer Mitchell, Professor in the Departments of Neurology and Psychiatry & Behavioural Sciences at UCSF, was involved in the Phase 3 trials and submitted a public comment (page 15) that attempted to address some of ICER’s concerns around trial design and conduct, such as with functional unblinding, research oversight, and so on.
Mitchell explained, for example, that she had “no knowledge of such events”, referring to a claim in the draft evidence report that “sexual boundaries were severely crossed with at least one patient”, and that “when a ‘patient was struggling, they were told to take their own supply of MDMA at home.’”
“That you are unaware of an event of a patient being told to take their own MDMA does not alter our evaluation of the event”, ICER said in response to Mitchell’s comment, but it did clarify that the “mention of sexual boundaries being crossed refers to an event in a phase II trial.”, adding that it has updated wording in the report to make it clearer (see page 9 of the revised report).
ICER further acknowledged that its decision to maintain anonymity among certain sources “does somewhat decrease the transparency we would typically want in an ICER report.”
When it comes to other methodological elements, ICER acknowledges that attempts were made to mitigate things like functional unblinding. But, it remains concerned with these issues in spite of these efforts.
Regarding the ICER draft evidence report stating that “Some patients were prevented from entering the long-term follow-up study and felt this was done to keep these negative outcomes out of the data set”, Mitchell says “This is untrue”, and said they “went through great pains to locate and include all phase 3 study participants”, with approximately 70% making it into the data set.
ICER, however, said that it had heard first-hand reports of such instances. “We cannot be certain that these reports were accurate,” it added, “but we do wonder whether Dr. Mitchell is certain that this could not have occurred.”
Scott Shannon, a principal investigator on the Phase 2 and 3 studies and a founding member of the Psychedelic Research and Training Institute, opened his comment by stating: “I find the summary to be unreflective of the current science and the nature of our psychiatric research” (page 32).
Shannon took issue with two key topics of ICER’s analysis: functional unblinding and investigator bias.
On the topic of functional unblinding, he argued that it “is a well-known element of anti-depressant trials”, where it “accounts for as much as 75% of the medication effect documented in SSRI trials” (citing Mora et al., 2011).
He further notes that “chronic PTSD … has much lower spontaneous remission rate”, so given that the “average participant in our studies suffered with PTSD for an average of 14 years … we would expect a very low spontaneous remission rate and a much more limited placebo response.”
“If these results are called into question because of functional unblinding then all of the results for FDA approved psychiatric medications must be challenged as well”, Shannon continued, adding that “[t]he large effect size found in these results would more than compensate for the minor shifts that could be attributed to unblinding.”
“The comments found in your report on unblinding are biased in nature and unconvincing,” he concluded.
In its reply, ICER said that “[i]mportant levels of functional unblinding are relatively uncommon in drug trials, but when it may have occurred, ICER discusses it as a concern.” It shared its review of therapies for Alzheimer’s Disease as an example of where it has done so.
On the topic of investigator bias, Shannon explained that all the study therapists and physicians “underwent thorough instruction in investigator bias and this was a constant intention for all of our staff who participated in the trial.”
“It is interesting to note that many to most of the psychiatric medication trials run by the pharmaceutical industry are led by investigators with deep commercial ties to that industry”, he continued, adding: “I suspect that is not called into question.”
In its reply, ICER pointed out that “[c]oncerns around investigator bias were pervasive when we spoke with experts in the field.” These complaints “came from many more people than the small number who raised many of the other issues discussed in section 2.1.”, it added.
Others Double-Down on ICER’s Concerns
Other commenters sought to validate or corroborate the concerns ICER raised in its draft report.
Joar Øveraas Halvorsen, associate professor of clinical psychology at the Norwegian University of Science and Technology, submitted a twelve-point comment (page 17) which corroborated many of ICER’s conclusions, including their determination that the evidence for MDMA-AT is “insufficient”; a conclusion that was reached, he said, by the American Psychiatric Association, an expert committee convened by Australia’s Therapeutic Goods Administration, and other “independent, non-partisan organizations.”
He also pointed out that the lack of comparative trials hamstrings the relative efficacy and cost-effectiveness estimates, to which ICER responded that it agrees regarding “a pressing need for head-to-head comparative clinical trials.”
Neşe Devenot, meanwhile, submitted a short comment that included the text of a petition she co-authored, which calls for the FDA to host an extended open public hearing2 at the forthcoming MDMA-AT advisory committee meeting (page 24).
Joe Welker also sought to corroborate some of the concerns raised by sources tapped in the draft evidence report (page 41). “I share the opinion of other industry critics who have been disturbed by the religious fervor, dogma, and social punishments in this field that evoke the worst of my own religion”, he wrote, adding that he is concerned “that spiritual and religious motives have significantly impacted the quality of Lykos’ data”.
He added that “researchers with close ties to Lykos” have admitted to “a decades-long strategy of using science as a vehicle to promote their spiritual beliefs”, linking to his own writing on the subject.
“While one may argue the legitimacy of these spiritual beliefs on their own merits and the legal issues involved in prohibition,” he continued, “this presents a unique problem: how do we trust data when a company not only has financial conflicts of interest that may impact research quality, but open spiritual and religious conflicts of interest?”
One of the longer submissions came from Sasha Sisko (page 48), who summarised their even longer manuscript that details cases of alleged abuse, boundary violations, omission of adverse events, and so on. “ICER does not have any regulatory, investigative or law enforcement authority”, it said in response to Sisko’s tome, instead encouraging them to report the allegations to relevant organisations.
Calls for More Representation from Those with PTSD
Several comments primarily sought to emphasise the toll of PTSD and push for greater representation of those with lived experience in ICER’s report.
One such group is Disabled American Veterans (DAV), which submitted a two-page letter (page 34). “Veterans are desperate for new treatment options and DAV believes more research is needed to verify effectiveness and ensure the safety of any new treatment options”, it wrote.
But, the “glacial pace” of the drug development and approval process, while ‘necessary’, “often results in veterans taking it upon themselves to seek symptom relief with other drugs and substances.”
“DAV has spoken to veterans who travel overseas searching for treatments that are currently unavailable in the United States”, it continued, adding that “they are turning to compounds that are often deemed dangerous, illegal or untested for efficacy.”
In its response, ICER has noted that it has included additional material in the Patient and Caregivers Perspectives section and invited DAV to participate in the May 30th meeting.
Another organisation that encouraged ICER to “glean additional input from those with lived experience of PTSD” was Mental Health America (page 38). ICER said that it has invited multiple patient organisations to share stories and lived experiences at the forthcoming meeting.
Methodological Critiques
Other public comments took broader aim at ICER’s economic model, methodology, and timing.
The Partnership to Improve Patient Care (PIPC), which counts the Pharmaceutical Research and Manufacturers of America (PhRMA) and the Biotechnology Industry Organization among its Steering Committee members, made several recommendations in its public comment (page 42).
Among its suggestions, PIPC encouraged ICER to adopt a societal perspective3 in its economic model. “As PIPC has commented to ICER previously,” it writes, “for diseases that have a considerable caregiver burden and high societal costs, like PTSD, the societal perspective presents a clearer picture than only using the health care perspective.”
“We did include productivity changes and other indirect costs in a separate modified societal perspective analysis”, ICER responded, pointing to Table E10 in the report.
PIPC also urges ICER to produce a range of estimates that apply to different patient types, such as PTSD severity or subgroups, as opposed to just the “average” patient, given heterogeneity of PTSD. “[w]e look at the average price for these drugs in our analyses”, ICER responded.
The last of the public comments came from a group of four graduate students (page 53), who argue—among other things—that ICER “should wait to release the evidence report until an FDA decision has been made”, adding that ICER’s releasing of this report based on ‘incomplete data’ prior to FDA evaluation “could jeapordize patient treatment access”.
“The founder of MAPS had contacted ICER previously to suggest that MDMA-AP might be an appropriate topic for an ICER review”, ICER explained in response to the comment, presumably referring to Rick Doblin.
“When ICER was about to begin its review we again connected with the founder of MAPS to share our interest in a near-term review of this topic”, Doblin reportedly “felt that it was ‘good to learn’ of [ICER’s] near-term interest in this topic and he did not convey any concerns at all about the timing of our interest.”
Anyhow, as aforementioned, it’s common practice for ICER to review drugs immediately prior to an FDA appraisal. As ICER points out in its response, “this is when an independent assessment of value is most helpful in informing the critical decisions that stakeholders across the US health system need to make around pricing, coverage, and prescribing.”
“While it may seem premature to assess the value of a treatment before it has been given to a population in a real-world environment,” it continued, “this is precisely what pharmaceutical companies do when they set the treatment’s initial price.”
The authors go on to critique other elements of ICER’s economic model in a seven-page commentary.
In an interesting response to one such critique, ICER pointed out that “[e]ven therapies that, in the long run, save costs and provider time can be infeasible in the near term if there are inadequate resources to implement those therapies.”
In a more general response to the four commenters, ICER stated: “While we appreciate your inquiry into these aspects of our work,” ICER responded, “it appears that all of the authors of this public comment are affiliated with No Patient Left Behind, but this organization is not listed as an author in your comment.”4
“We invite you to clarify whether these comments were drafted in collaboration with No Patient Left Behind”, ICER continued, adding that they “believe that transparency is important when participating in public discussion on these topics.”
Changes Reflected in the Revised Evidence Report
Reading the draft and revised evidence reports side-by-side, it’s clear that little has changed.
One more prominent change is the title of the report, which has gone from “3,4-Methylenedioxymethamphetamine Assisted Psychotherapy for Post-Traumatic Stress Disorder” to “Midomafetamine-Assisted Psychotherapy for Post-Traumatic Stress Disorder”, with ICER now opting to use the international nonproprietary name for the drug. They are still skirting Lykos’ preferred terminology, however, which skips “psychotherapy” in favour of “psychological intervention”.
As alluded to above, other changes include the addition of two paragraphs to section 2, Patient and Caregiver Perspectives, which discuss common comorbidities of PTSD and the VA’s efforts to investigate psychedelics, among other things. Page 28 discusses changes made to the economic evaluation between the draft report and the revised report, while page 36 shares some areas for future research.
Ultimately, however, the revised evidence report is substantially similar to the draft. And, it still arrives at the conclusion that current evidence is “insufficient” in terms of feeding into a comparative cost-effectiveness estimate.
While ICER accepts that attempts were made to mitigate issues such as functional unblinding, it remains concerned about the potential effects of these factors on the quality of Lykos’ evidence; especially as levels of unblinding, for example, were greater than in other cases it’s reviewed.
Similarly, ICER claims that concerns around investigator bias were “pervasive”, and echoed by many other people than reflected in its report.
ICER Emphasises Its Independence
On the topic of bias, ICER reports that it “heard concerns from those who felt the Report was unbalanced by focusing mainly on possible biases among those who are proponents of MDMA-AP and also heard concerns from those who felt the Report was inappropriately trying to appear balanced by also highlighting possible biases in those who have raised concerns about the MAPP trials.”
Receiving post-publication expressions of concerns from ‘both sides’ suggests, according to ICER, “that some perceptions of bias may be in the eye of the beholder”
Again maintaining its lack of bias, ICER wrote: “It seems common today, and perhaps in the past as well, to assume that those who disagree with us must have corrupt motives. ICER and its funders have no stake in the results of our reports.”
Next Steps
Now that the public comments, ICER’s responses to said comments, and the revised evidence report have been published, a public meeting will be held on May 30th. Members of the public may register via Zoom to attend the meeting.
Editor’s Note: Given the interest in this topic, we’re lifting the paywall to allow a wider readership. If you would like to read all of our content, please consider subscribing to Pα+. ∎
- “[A]lthough the ICER report uses the term ‘MIDMA-AP,’ [sic] the Lykos-sponsored research and submission currently under consideration with the U.S. Food and Drug Administration (FDA) is for midomafetamine (the U.S. established name for the substance) used in conjunction with psychological intervention (MDMA-assisted therapy)”, the first footnote in the letter states.
- Yesterday, Devenot revealed that the FDA had asked her to withdraw her petition. Read more via her Substack, Chemical Poetics.
- As we have covered previously, some cost-effectiveness analyses of psychedelic interventions, such as McCrone et al. (2023), have only found them to be cost-effective when a societal perspective is adopted.
- This author thought he was smart in twigging that these students appear to have one thing in common: they’re Fellows at No Patient Left Behind, a group founded in part by healthcare investment firm RA Capital that Patients for Affordable Drugs Now says would be better named “No Pharma Investor Left Behind”. However, he was deflated upon reading ICER’s responses, which had already drawn the link!
