Response to
Little and Lakens: a Comment on The Vindication of
Magnitude-Based Inference Alan M Batterham, Will G Hopkins Sportscience 22,
sportsci.org/2018/CommentsOnMBI/ambwgh.htm, 2018 Summary: The responses of Roderick Little and Daniël
Lakens to our rebuttal of Dristin Sainani's critique of magnitude-based
inference (MBI) have highlighted the need for MBI to be promoted and accepted
as a valid form of Bayesian inference, in which probabilistic statements
about the true magnitude of an effect are not modified by any prior belief or
information about the effect. We respond here to the comments of Rod Little and Daniël Lakens on our rebuttal of Kristen Sainani's critique
of magnitude-based inference (Sainani, 2018), as well as summarising briefly
and integrating the recent positive interactions on social media (see e.g., this Twitter thread). Rod Little’s helpful
comment reinforces our assertion that MBI is indeed Bayesian with a
least-informative prior. We are very happy to view MBI as a special case of
calibrated Bayes inference and to describe the prior distribution as
dispersed uniform. We also agree that
this "objective" form of Bayesian inference has a long history,
countering recent claims on Twitter (by Harrell and Althouse) that we
have "made up" a "new" method. That said, we have
supplemented the simple presentation of posterior probabilities of attaining
various effect sizes of interest with decision guidelines anchored to
clinical, practical, or mechanistic relevance. Some kind of guidance is
needed for researchers and practitioners making decisions, especially about
potential implementation of an effect, and also for journal editors deciding
whether an outcome has sufficient precision for publishability. Recently, we
considered renaming MBI as minimalist
Bayesian inference. On reflection, however, we feel that we have extended
the simple derivation of posterior probabilities of various effect sizes sufficiently,
with a focus on size of effects in relation to clinical/practical/mechanistic
importance, to justify the continued use of the term magnitude based inference. We are grateful to Daniël Lakens for helping us to fully grasp
the central plank of the critique of MBI–that MBI
has no firm theoretical basis and requires a formal, principled footing. As
stated, MBI is Bayesian. Bayesians do not typically concern themselves with
error control, but we did so for two reasons. First, Welsh and Knight (2015) reported
that error rates were high for MBI. Second, and more importantly, researchers
and practitioners have to make decisions, and decisions have attendant
errors. We have elaborated on our definitions and we believe firmly that our
error rates are acceptable compared with those of null-hypothesis
significance testing. Therefore, we contend, as stated by Rod Little, that
MBI is a form of calibrated Bayesian inference with a dispersed uniform prior
giving a posterior distribution with reasonable frequentist properties. We
acknowledge that in our rebuttal to Sainani our use of the term hybrid to describe MBI did not
characterise the approach properly; calibrated
Bayes is the more appropriate and explicit term. As discussed in a Twitter exchange
with Daniël Lakens, we do not expect researchers to quantify error rates
(based on some assumptions) on a study-by-study basis. It is clear therefore
that the above articulation resolves the main point of contention in the
recent debate; MBI indeed has a firm, explicit epistemological basis. MBI is
calibrated Bayesian inference, and the prior is not buried, in Rod Little’s
terms; we are up-front about the prior being minimally informative (dispersed
uniform). Rod Little cites the well-known “screening paradox” for a rare
disease to illustrate that prior distributions play an important role in
inference in situations where prior information is strongly informative. In
many instances, however, we contend that beliefs based on existing data or experience
are not sufficient to form a robust prior distribution, and we prefer to
specify a dispersed uniform prior and to let the posterior be determined
entirely by the data. We agree fully, of course, that the specification of
the prior distribution needs to be transparent and subject to criticism, and
this requirement goes for both "objective" MBI and "subjective"
Bayesian approaches with informative prior distributions. The cornerstone of a fully transparent presentation of results
with MBI uses our qualitative probabilistic terms based on the posterior
probabilities of substantial and/or trivial effect magnitudes. It is
important to note at this juncture that the Bayesian ROPE procedure
(Kruschke, 2018) advocated by Daniël Lakens, using its default broad prior, gives
posterior probabilities of benefit and harm that are practically equivalent
to those from MBI, as both approaches use minimally informative priors. We have
always advocated complementing the results with the presentation of the credibility
interval, which in MBI is congruent with the standard confidence interval. Anyone
of an equivalence-test persuasion may then use the disposition of the
credibility interval to their specification of the region of practical
equivalence. This suggestion is pragmatic, and should not be taken as us
slipping into frequentist language. As we have argued, we prefer estimation
to "testimation", so we wish to avoid hypothesis tests of any kind,
involving either the nil (zero) hypotheses or non-zero hypotheses, as in
equivalence testing. Anyone using MBI still concerned with error rates may
supplement the results with the presentation of the second-generation p
value, staying true to an estimation approach anchored to clinical or practical
relevance. We hope
that we have shown clearly that we are not purveyors of "shoddy
statistics", distorting the scientific record in sports medicine and
exercise science. Bayesians of other persuasions may take issue with our
choice of prior, and frequentists might argue with our estimation approach,
but there is no doubt that MBI is on a firm epistemological foundation as a
form of calibrated Bayesian inference with a least-informative prior. Kruschke, J. K. (2018). Rejecting or Accepting Parameter Values in
Bayesian Estimation. Advances in Methods and Practices in Psychological
Science, 2515245918771304. https://doi.org/10.1177/2515245918771304 Sainani KL (2018). The problem with
"magnitude-based inference". Medicine and Science in Sports and
Exercise (in press) Back to index of comments. Back to The Vindication of Magnitude-Based Inference. First published 3 June
2018. |
