-
The "weak evidence" argument is unfounded and highlights strength of DA-BOLD link
Submit responseWe appreciate the interest in our work by the author of the "weakness of evidence" comment. Further, we also thank the commenter for allowing us the opportunity to revisit the strength of our evidence. The commenter suggests that inappropriate statistical analyses were used in this study and that there were undue influences from outliers on the association between fusiform gyrus BOLD and D1 binding potential (BP) reported in our study, as described in the article entitled Dopamine D1 Binding Potential Predicts Fusiform BOLD Activity during Face-Recognition Performance. Although the commenter argues that the use of his or her recommended statistical methods are more appropriate than those utilized in our study, the commenter fails to give evidence to demonstrate this assertion. Here we give a point-by-point response to the "weakness of evidence" argument. We conclude that the commenter's argument hinges upon misleading assertions. In fact, the contention that our results were in some way "weak" does not stand when using the commenter's suggested methods.
We do agree with the commenter that use of large sample sizes is optimal for drawing the most accurate conclusions about natural phenomena. Our sample size was not large (N = 18); although we confidently liken this size to many other studies within multimodal imaging research. We are also confident in the robustness of the D1-BOLD relationship we obtained, given the large proportion of shared variance we demonstrated between these two variables (see Cohen, 1988).
The commenter claimed that our original ordinary least squares (OLS) regression model had "overfit" our data. The commenter used leave-one-out cross-validation modeling to which he or she suggested that the use of this modeling technique showed a subjectively marked change in model fit between FFG D1 binding potential (BP) and BOLD. There are at least two problems with the commenter's approach and interpretation. The first and most obvious is that cross-validation modeling is not appropriate in this case because we are neither attempting to fit contending models nor are we assessing models for their ability to extrapolate to novel data. Second, if one were to use this approach, as the commenter did, one would see that the commenter's claim of a "weak" association between D1 BP and FFG BOLD is conventionally unfounded (e.g., Cohen, 1982). The model that the commenter used showed a subjectively reduced fit compared to our original model (i.e., r2 = .38 to r2 = ".20"). However, the commenter mistakenly elides over the fact that the effect size of his or her model is neither trivial nor weak (i.e., it is at least a medium effect size; see Cohen, 1988).
The commenter's suggestion to use a Spearman's rho is not supported given that the original, parametric model is not affected by potential outliers (see below). It is also important to note the commenter's calculation error of Spearman's rho (.0939). The actual value of this calculation is .3891.
Although our original work showed that possible outliers did not have any influence on the FFG D1 BP and BOLD model (see manuscript p. 14704), the commenter insinuated that the outlier detection methods used in our paper were antiquated. When he or she used "modern outlier detection techniques," these, the commenter claims, revealed several bivariate outliers. Although the commenter did not provide any context for his or her insinuation that our methods are antiquated, we must point out that several multivariate outlier methods and point influence diagnostics show that our model was not unduly affected by anomalous observations. First, a residual plot against fitted values reveals symmetry about zero with no noticeable slope that would depend upon magnitude (see http://www.utdallas.edu/research/nprlab/figure.htm). Second, Mahalanobis distance measures indicate that even the more extreme points of our model occur at a rate of about 8% on random bivariate normal distributions with our sample means and covariances. Third, influence statistics, such as Cook's distance measure, do not show any point that has greater influence on the FFG DA-BOLD parameter estimate compared to any other point within the model. Lastly, subjecting our model to a Monte Carlo (B = 1000) bootstrapping procedure, we find that the significance of our model remains (? = .615, 95% CI = .01-.96, p = .03). Such random resampling methods allow for equal probability of representation of each observation, minimizing the influence of outliers (see Efron, 1982). By these and many other measures, our model was not unduly influenced by potential outliers.
Regarding the commenter's criticism of Figure 3 within the article, we agree that one cannot claim a difference between correlations without testing them directly. We were surprised that the commenter would charge us with making such a claim. We did not say that D1 binding potential in ROIs outside of FFG have no correlation with BOLD in FFG, or for that matter are unimportant compared to D1 BP in FFG (p. 14705). Actually reading our manuscript would have assuaged any such concerns and revealed that our focus was on the importance of FFG in the context of DA intensive regions. Our emphasis on the relationship between FFG D1 and BOLD was specifically highlighted because it represents a unique contribution to the literature.
Our work demonstrated a novel finding, that FFG D1 BP was predictive of FFG BOLD signal during face recognition. As with any empirical findings, especially novel ones, our results must be subjected to further testing to confirm or negate their veracity. This should be done via rigorous replication efforts (e.g., Open Science Collaboration, 2015). The commenter's remarks did not offer replication data to support an alternative hypothesis, nor did it offer sound statistical reasoning for why our methodology was somehow flawed. Rather, the comments were unsupported by the commenter's own evidence and represent a flawed attempt to impugn our work.
References Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd Edition). Hillsdale, NJ: Earlbaum.
Efron, B. (1982). The Jackknife, Bootstrap, and other Resampling Plans (1st Edition). Philadelphia, PA: Society for Industrial and Applied Mathematics.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349 (6251).
Conflict of Interest:
None declared
-
Weak evidence for a link between dopamine D1 binding potential and fusiform BOLD activity
Submit responseThis study is very interesting but the sample size and the results are not sufficient to draw firm conclusions. A version of this comment with Matlab code is available on PubPeer: [http://bit.ly/1PBz2dA]
In the section on "D1-BOLD relationships", the authors write that, "The standardized correlation coefficient revealed a positive relationship between these variables (B = 0.62; R2 = 0.38, p < 0.007; Fig. 2"
I used GraphClick version 3.0 to retrieve the data points from figure 2. A new analysis in Matlab reproduces the results, with an R2 of 0.38. However, R2 is inflated because the same data are used to fit the model and to predict. To avoid overfitting, I used a leave-one-out cross- validation, which reveals an R2 of 20%, almost half the original value. But that's using Pearson's correlation. Remember that the R2 of a linear model is the correlation between obtained and predicted values. Using Spearman's correlation reveals a much weaker value of 0.0939. A plot of predicted observations as a function of obtained observations shows that the OLS model is inaccurate: it overestimates the smallest BOLD responses, and underestimates the largest BOLD responses. Very few observations fall on the reference line where the observed and predicted values would be equal.
The authors continue, writing, "regression diagnostics did not identify any unduly influential data points; Belsley et al., 1980)."
Modern outlier detection techniques suggest the presence of two bivariate outliers:
robust_correlation(data(:,1),data(:,2))
The code is available in Pernet et al., (2012)
Finally, the authors state, "No such relationships were observed in any other face-specific network regions"
This analysis does not demonstrate that the significant correlation differs from the non-significant ones. This common error is described in Nieuwenhuis et al. (2011).
More information about linear regression can be found in the excellent books by Kuhn and Johnson (2013) and Wilcox (2012).
References
Kuhn, M. and Johnson, K. (2013) Applied predictive modeling. Springer, New York.
Nieuwenhuis, S., Forstmann, B.U. & Wagenmakers, E.J. (2011) Erroneous analyses of interactions in neuroscience: a problem of significance. Nat Neurosci, 14, 1105-1107.
Pernet, C.R., Wilcox, R. and Rousselet, G.A. (2012) Robust correlation analyses: false positive and power validation using a new open source matlab toolbox. Front Psychol, 3, 606.
Wilcox, R.R. (2012) Introduction to robust estimation and hypothesis testing. Academic Press, Amsterdam ; Boston.
Conflict of Interest:
None declared
- Print ISSN: 0270-6474
- Online ISSN: 1529-2401
