The significance fallacy in inferential statistics

BMC Research Notes

Table 1 Results for ‘thermometer’-study

	Actual ^a	‘Significant’ (n = 53)	‘Non-significant’ (n = 73)	(z-value) *p-value*	Effect size
N	33	76	50	(z =-1.75) p = .08	r = -.15
M _group1	5.12	2.70	3.50
M _group2	4.13	4.05	4.00
M_diff	0.99	2.00	1.00	(z =-5.27) p < .001	r = -.47
SD _group1	1.22	1.00	1.25
SD _group2	1.41	8.00	10.00
Cohen’s d	0.78^b	0.60	0.30	(z =-3.88) p < .001	r = -.34

Note. ^aThe actual study reported a significant effect. Attempts to replicate the effect of temperature on social relations within the Many Labs Replication Project have failed, however [54].
^bCohen’s d = .78 is reported in the paper. Calculating effect size from means and standard deviations using the Campbell effect size calculator available at http://www.campbellcollaboration.org/resources/effect_size_input.php results in d = .75, 95% C.I = [0.05; 1.46].

ISSN: 1756-0500