In the spirit of bringing experts to the table (or, your screen/inbox), we asked Boris D. Heifets, MD, PhD to provide a more in-depth commentary on the paper. This commentary is reproduced below.

Overall

First off, the authors should be congratulated on pulling off the first randomized, controlled study of psilocybin for the treatment of major depressive disorder. These studies are expensive and difficult (we are starting a similarly designed trial of psilocybin for chronic pain with colleagues at UCSF & UC Berkeley) and despite all the caveats attached to their findings, the fact that it’s appearing in NEJM speaks to the incredible importance of finding better treatments for severe psychiatric disease. However, this study has some very unfortunate (and unlucky) design features that make it difficult to interpret.

Strengths

• Largest study yet, and the first randomized study with an appropriate comparison group to study how effective psilocybin might be;
• psilocybin shown to be safe with a side effect profile comparable, or maybe better than, the leading SSRI;
• authors used several different scales to assess symptoms of depression, including measures that the study participants reported and standardized ratings by the study clinician.

Weakness

• Despite many strong positive signals that psilocybin actually did out-perform the SSRI, the study was designed in a way that prevents drawing any conclusion about their results. Deciding on your primary outcome measure is something clinical researchers routinely do before a study (and post publicly), to prevent cherry-picking data that favors a particular outcome. Out of the nearly dozen clinical scales they used to measure depression symptoms, they somehow happened to pick the one scale that showed no difference between the two treatments as their major (primary) outcome measure. Despite having favorable comparisons in the secondary outcomes, In the authors’ own words: “Because of the absence of a prespecified plan for adjustment of confidence intervals for multiple comparisons of secondary outcomes…no clinical conclusions can be drawn from these data.”
• Like virtually every study of psychedelics, it is incredibly difficult to keep the subjects “blind” to their treatment condition – the effects of the drug are too obvious. This study used an active treatment (SSRI) as a comparison group, so the hope was that may have kept the expectations of treatment benefit somewhat balanced between the groups, but unfortunately this study did not test how well their blinding procedure worked.

The Weird

• What I can’t wrap my head around is the way this study is written, with the authors’ perplexing claim in print (Section S2. Methods, end of paragraph) and on Twitter that they expected psilocybin to be no better than a standard SSRI (both with psychotherapy). The statistical analysis plan and calculation of how many subjects to recruit reads exactly as one would expect for a study that expects a massive difference between the two treatments. The authors (and a huge number of academics, myself included, and journalists) have described psilocybin as a “game changer” and potential “breakthrough”. The palpable excitement around this emerging area of medicine is all predicated on the idea that this new class of medicines and these new ideas about psychotherapy would up-end our dated, ineffective ways of thinking about and treating depression and other psychiatric disorders. The entire motivation for doing a trial like this, in my mind, would be to show the world, in a well-executed, rigorous fashion, that the new treatment beats the old treatment on all the gold-standard metrics. To my eye, that’s what this trial looks like it was designed to do, and unfortunately it missed the mark. If the authors really believed from the get-go that these treatments would be similar, they could have designed the trial to test for equivalence, and they would have analyzed it accordingly. What we are left with is very difficult to interpret, except to say that we need to do it again with many more subjects.  
• It’s more likely than not that existing metrics for depression are just not well suited to studying psilocybin and measuring its impact, e.g. emotional breakthroughs, meaning-making, spirituality. Hopefully the field will evolve and broaden its horizons.

Outlook for medicalization and commercialization

• It is formally speaking a negative study, despite all the promising signals, which is a rather surprising outcome for the field of psychedelic medicine. I don’t know what impact this study will have on psilocybin’s FDA approval (which I am generally in favor of), though I suppose the study design and lack of interpretability really doesn’t address the basic question “does it work for depression”. The title of this paper could just as easily have been “Serious people take psilocybin seriously because, hey, the NEJM is publishing a study on it”.
• This study may have some unwelcome implications for the industry developing around psilocybin for medical use. It will be harder to make a case that psilocybin is the game changing panacea some had hoped for – and harder to convince insurers to pay an order of magnitude more for a therapy that’s no better than an SSRI. What’s more, the therapeutic model incorporated in this study strongly hints at what’s to come – repeat dosing sessions. I already have very mixed feelings about cash ketamine clinics catering to patients with crippling levels of depression and chronic pain… the price tag on this may be much higher.