Recent Successes in Correcting THR Science

Efforts to correct flawed science often go nowhere, but there are a few recent successes: a couple post-publication criticisms and a retraction - though not before flawed papers had caused real damage

In theory, science should be self-correcting. In practice, unfortunately, it’s often not. In tobacco harm reduction (THR) research, a combination of factors work against the science being self-correcting: funding incentives that focus the research on harm rather than harm reduction, polarization in the field and ostracization of researchers with actual or even perceived ties to industry, and confirmation bias in the interpretation of ambiguous results and throughout peer review. However, these problems are not unique to THR.

Depending on my workload, when I see an especially flawed paper, I will sometimes take a formal action to try to correct it. My options are:

1. Contact the authors (if I think there’s a reasonable chance of response) and see if they are willing to answer my questions or provide the data or code they used. There’s always a chance their response might alleviate my concerns. If there is a serious flaw and the authors acknowledge it, then I can try to convince the authors to go to the journal to request a correction or retraction. (Each stage of this sequence gets successively less likely, but I and colleagues have been able to do it).

2. Write a Letter to the Editor critiquing the study and submit it to the journal that published the original article. Most times I’ve tried this, my letter is swiftly rejected. (I get the sense that if the criticism is solely related to methodology, rather than fraud, editors consider it a statistical judgment call that’s not worth a formal action). If I’m lucky and the journal does decide to publish my criticism, they almost always offer the authors of the original article the opportunity to respond. This is usually unsatisfying because the authors often dismiss the criticisms and double down on their original conclusions, but at least there’s a formal publication of the criticism.

3. Depending on the journal, there might be a format for posting a comment that involves less gatekeeping, such as JAMA’s “comment” section which is moderated only for relevance. The downside of this is that the comment section is hard to see from the main page and you have to be intentionally looking for comments. Also, there is no expectation or pressure for the authors to respond.

4. Failing the above, I can always post on PubPeer or Qeios, which are platforms for open post-publication review (though the offending authors are under no pressure or expectation to take corrective action). By the way, one benefit of PubPeer in particular is that if you install the browser plugin, it detects whether the website you’re on contains any references to papers with PubPeer comments and displays it in a very obvious banner at the top of the page:

I ordered the above list from least likely to happen but most impactful, to easier to accomplish but less impactful. I’ve written several post-publication criticisms over the last several years (when I have time to do so) and it’s unusual that the journals or editors are interested. Many of my comments end up on PubPeer for that reason. But since there have been several notable successes recently, by others as well as myself, it’s worth highlighting when things go right.

1. Retracted Paper on E-Cigarettes & Stroke

In 2022, Patel et al. published an article analyzing NHANES data, claiming that e-cigarette use was associated with stroke. This article did incredible damage with respect to misperceptions of harm; as the Retraction Watch article notes:

The paper, published in Neurology International in 2022, reported e-cigarette users had a higher risk of early stroke than traditional tobacco users. It has been cited 22 times, according to Clarivate’s Web of Science, and was covered in the media, featured in a public campaign against vaping and included in a contested meta-analysis.

Colleagues Floe Foxon and Gal Cohen were skeptical about some of the numbers and decided to replicate the analysis using NHANES data, which are public. The problems were more severe than they expected; for example, the NHANES data that Patel et al. claimed to use has ~20,000 participants, but Patel et al. reported a sample size of >200,000. This is not a simple rounding error, forgetting to list an exclusion criterion, or missing data issue — there are over 10 times more participants than there should be.

With such a glaring difference in the starting point, it’s not clear whether any of the results in the original article reflect anything real. As bad as that is, the problems go beyond the data analysis and methodology. Lead author Urvish Patel falsified his affiliation (i.e., said he worked at a school of medicine that he was not employed by or affiliated with). He also runs a company that is essentially a paper mill, in which aspiring students (usually medical students) pay a few thousand dollars essentially for having their name added to the author list of a scientific article (to pad their resumes) under the guise of participating in a research training program. (One of the “tells” is that there are many more authors than are needed for a basic data analysis, and each is from a different institution.)

This paper had just about everything wrong with it that could be wrong in a paper. It seemed like if any paper deserves to be retracted, surely this is one. Thankfully it was retracted in the end, though it took nearly two years and has done damage (as I mentioned above) which also has done downstream further damage (e.g. how many people who could have benefited from e-cigarettes were instead scared away from them by the public health campaign)?

2. Letter to the Editor on Smokeless Tobacco & Macular Degeneration

In 2025, Ochoa et al. published an article analyzing NHIS data and reporting:

“smokeless tobacco as a novel factor associated with AMD [age-related macular degeneration], suggesting its avoidance may reduce the risk of developing AMD.”

Notice the overt causal language in that statement, which is not even close to being supported by the cross-sectional, observational data. The article examined e-cigarette use, too, which was not significantly associated with AMD but the authors seem to believe it is anyway, calling for:

“additional studies… to better understand the long-term effects of e-cigarettes on AMD.”

I submitted a letter to the editors, after working with Brad Rodu on an earlier version, including some helpful analyses he ran on sample size and a breakdown by smoking history. To my surprise, they responded immediately and took our criticisms seriously. They were especially interested in a general commentary on analyzing national survey rather than a specific one on Ochoa et al. (though of course I used that as an example).

My letter (journal link here; unformatted full text here) highlights 5 common flaws in such papers (of which there are hundreds by now):

• Exposure is defined overly broadly. That is, Ochoa and most similar papers use “ever use” or “past month use” as the possible cause of the health outcome. But only a fraction of these go on to prolonged use, and even for much-more harmful cigarette smoking it takes decades to develop health effects.

• The number of cases is insufficient. There were only 7 cases of AMD among participants who used smokeless tobacco, and only 8 among those who used e-cigarettes. This is far too small to get reliable results. Worse, these numbers weren’t reported in the original article (this is from Brad’s NHIS tabulation).

• Analyses did not rule out cumulative and/or lingering effects of prior smoking. This is nearly pervasive in all studies of e-cigarette use and a health outcome. There’s a large overlap between e-cigarette use and cigarette smoking and it’s not sufficient to control for smoking status (i.e. whether they smoked or not). What matters for health outcomes is cumulative exposure (e.g. pack-years) or duration of smoking, but Ochoa (and all but a handful of other such studies) don’t do.

• Temporality of the association is unclear and possibly backwards. NHIS is cross-sectional and didn’t contain information on age of AMD diagnosis. There might be people who developed health problems from smoking, and in response switched to e-cigarettes. This would produce the same association in the dataset but for the opposite reason.

• Interpretation assumes a one-way causal relationship. The authors interpret the results as smokeless tobacco causing AMD and don’t consider alternative explanations. Again, note the causal language in the abstract’s conclusions (“its avoidance may reduce the risk”).

This is speculation, but I think the difference in interest from this journal vs. the ones I’ve approached in the past is that this is specific to eye diseases (Journal of VitreoRetinal Diseases) and they (a) might not be aware of the toxicity in tobacco research and (b) might have fewer of these types of articles (i.e. analyses of national survey data) and may be more receptive to hear the flaws.

The authors replied here and had two main responses (vs. our five):

• Justifying “ever-use” as the exposure as being “a well-recognized and widely accepted definition in tobacco epidemiology” and (being an overly broad measure), “helps capture designs of experimentation and admission, especially in the youth and young adult population.” (This seems in opposition to the paper’s focus on AMD which is, by definition, age-related. E-cigarettes were not invented when the people with AMD in this dataset were youth and young adults, and there’s no information on what age these participants started using smokeless tobacco).

• Reiterating that they analyzed current, former, and never-smoking separately (which ignores the criticism that adjusting for detailed smoking history is necessary) and then going into a non-sequitur about 99.3% of AMD patients being diagnosed by a doctor and dropping participants with missing data from the analysis.

The response is unsatisfying and doesn’t address any of the five root issues I identified in my article. But at least there’s a formal published criticism of the article that appears in the “related articles” section.

3. Letter to the Editor on E-Cigarettes & Cancer

In 2025, de Oliveira et al. published a systematic review concluding that “e-cigarette may e associated with an increased risk of certain cancers, including cervical and breast cancer.”

Miguel & Steffensen, who consult to a subsidiary of British American Tobacco, published a letter to the editor pointing out numerous flaws in the de Oliveira et al. review, namely:

• Substantial deviations from the preregistered protocol for the systematic review, including additional outcomes, widening their search criteria to include lower-quality cross-sectional studies and not-peer-reviewed conference abstracts, and additional statistical comparisons

• Including retracted studies in the review; in one place, a retracted study is listed with the highest possible quality rating.

• Inconsistencies in numbers reported in different parts of the manuscript, including orders-of-magnitude differences in sample size (e.g., 1555 vs. 98,000).

• Key limitations are not acknowledged, including the issue of confounding by prior smoking history, lack of detailed data on duration/cumulative exposure, and unknown temporality (i.e., not knowing whether cancer preceded e-cigarette use).

At the time of writing this post, I do not see a reply correspondence from the original authors.

4. Replication Study on E-Cigarettes & Chronic Kidney Disease (CKD)

In 2025, Li et al. published a paper analyzing NHANES data concluding:

“E-cigarette use is independently associated with CKD in a dose-dependent manner, particularly among non-diabetic individuals. Notably, NHANES data limitations include lack of past e-cigarette use or smoking history records, which may introduce residual confounding. These findings highlight vaping as a modifiable risk factor for kidney disease, urging targeted public health interventions.”

Credit where credit is due: the authors note the possibility of residual confounding in the abstract, which is a more prominent place than it’s usually mentioned (buried in the Limitations section at the end of the Discussion).

Unfortunately, their claim that “smoking history records” are not available in NHANES is simply not true. There’s a very detailed smoking history questionnaire which has information on heaviness of use, duration of use, and cumulative history. While there will still be some degree of residual confounding after accounting for these, smoking history is the main confounding factor that needs to be accounted for here.

Yet it’s not, and despite the authors acknowledging this problem (not to mention being able to do something about it but choosing not to), they nevertheless conclude that vaping “is a modifiable risk factor” and call for public health efforts, all of which assume that it is a causal relationship.

Colleagues Gal Cohen, Sooyong Kim, and I re-analyzed NHANES to see if we could replicate their findings, and improve on them (since we knew that detailed smoking history was available). Our replication study, which was just published, points out the following issues:

• We could not come close to replicating basic starting points in their analysis such as sample size (7139 vs. 872) or the prevalence of CKD among e-cigarette users (8.4% vs. 33.5%). Of course, this was after closely following their description of the analysis in their Methods section.

• The definition of CKD wasn’t consistent (Li et al. provided two separate definitions) and neither was the standard way of defining CKD risk.

• Insufficient sample size: only one e-cigarette user had CKD according to one of their definitions. It is not possible to get reliable statistics from so few cases.

• The original analysis adjusts for smoking status (i.e. controls for current/former/never smoking) but this is too crude and insufficient to account for the health effects of cigarette smoking, which scale with duration and cumulative exposure.

• We improved upon the original analysis by segmenting or stratifying by smoking history — i.e. examining the associations among the current smoking group, then the former smoking group, then the never smoking group (all separately). The idea is that if e-cigarettes have a causal effect on CKD, this should be evident at all levels of smoking. But we didn’t find robust evidence for this. (The results varied widely across definitions of CKD; one showed a higher rate among e-cigarette users, but another showed a lower rate, and using the standard definition there was no difference).

The figure below sums up our main points (kudos to Gal for fitting so much information into a single figure):

While we are happy this is finally published, there are several less favorable aspects of this situation that limit the feedback loop to actually correct the incorrect science. Since this is published in a different journal than Li et al. was published in, there’s no obvious way for an unknowing reader to find out that there’s a criticism of this article.

The reason why it’s published in a different journal is because the original journal (BMC Public Health), which was the first stop for our re-analysis, swiftly desk-rejected it (i.e. the editor rejected it without even sending it out for peer review). In an ideal world, journals would want to keep their own house in order and ensure they publish only the best-quality science.

We then submitted our re-analysis paper to three other journals, some of whom explicitly list replication studies in their Aims & Scope. All of them were desk-rejected. But we were successful in the end in getting this published.

Conclusions

There is a lot of flawed research in peer-reviewed journals. Over the years, I’ve made many attempts to correct especially flawed research. It is time-consuming to do this, not only to write up the criticism but to go through the official channels to try to pursue a formal action by the journal.

It is also often demoralizing:

• Full retraction is rare.

• When a retraction happens it can take years, during which time the paper is being cited and doing damage in the scientific and public spheres.

• If the editor doesn’t think the flaws warrant a full retraction, a lesser option is often a letter to the editor published in the same journal. But those usually allow the original authors the final word and they use it to dismiss the criticisms and double down on their original conclusions.

• Most likely is that the editor / original journal are not interested at all in the criticism. There’s always PubPeer or Qeios, but unless readers of the original paper are specifically interested in science integrity and have the PubPeer browser plugin, they won’t be aware of the criticisms.

Against this backdrop, it’s especially exciting when these usually-thankless efforts succeed. There have been a cluster of four success stories just in the past two months and that’s worth taking a moment to appreciate.