2.4. Permutation test for correlation coefficients
The classic approach to multiple hypothesis testing, for example, the Bonferroni correction, is very conservative in some cases, increasing the risk of Type 2 errors. In order to adequately test the six correlations for which we formulated hypotheses, we applied a simultaneous permutation test as described in the following. Let us denote X1 (N250 targets), X2 (N250 low-distinct non-targets), X3 (N250 high-distinct non-targets), X4 (recognition accuracy), X5(recognition speed). We were interested in six correlation coefficients\(\rho_{i}\) i = 1,…,6 (between each of the three variables X 1, X 2,X 3 and each of the two variables:X 4, X 5). The problem can be addressed by verifying the hypothesis:
\(H_{0}:\ \operatorname{}{|\rho_{i}}|=\ 0\)
\(H_{1}:\ \operatorname{|}\rho_{i}|>\ 0\)
Hence, we took the maximum from the (absolute) values of the six mentioned correlation coefficients as a test statistic:
\(T=\operatorname{}{|r}_{i}|\) i = 1,…, 6
where ri is a sample Pearson correlation coefficient. To perform the permutation test we calculated the value T0of statistics for our tested sample from the experiment. Assuming that the null hypothesis is true, we performed N = 10,000 random permutations (shuffles) of the data in each of the five variables, calculated the value of the test statistic for each permutation and created the empirical distribution of Tj ,j =1, 2,…,N .
If the calculated ASL (Achieved Significance Level) (Good, 1994):
\begin{equation} \text{ASL}\approx\frac{\text{card}\{T_{i}\geq T_{0})}{N}\nonumber \\ \end{equation}
is less than the significance level p = 0.05, the null hypothesis is rejected. Please note that apart from being less conservative than Bonferroni correction, the permutation test of statistical significance has the advantage of not assuming normal distribution of the data.