Is it the end of the world if $\alpha=0.005$ is the new norm?

In this paper by Benjamin et al (2017) on redefining statistical significance, they proposed to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries. That is the proposed p-value is one tenth of the conventional one!! Suppose the world changed to p=0.005. Do we need 10X more sample? As a researcher without sufficient funding, we care about how much additional sample we need suppose our hypothesis is true.

First let’s review how sample size is calculated. It is really a good review of basic concepts in probability theory and statistics.

How Sample Size is Determined?

First we have to imagine that \(N\) participants will be randomly assigned into two groups: Treatment (T) and Control (C) group. Assume that there are equal numbers of participants in each group (\(N_T=N_C\)). The researcher is interested in testing whether the mean of the distribution \(\mu\) in two groups are different. More specifically, the researcher is comparing the two hypotheses:

\[H_0: \mu_T-\mu_C=\mu_0=0\] \[H_A: \mu_T-\mu_C =\mu_A \neq 0\]

The researcher is interested in the impact of the treatment on variable \(x\). Let \(X_T\) denote the sample average of the treatment group, and \(X_C\) the sample average of the control group. The significance level is defined as \(\alpha=Prob(Reject \quad H_0|H_0)\). It can be expressed in the following way: \(\alpha = Prob(X_T-X_C\geq v|H_0)\), where \(v\) is the critical value. For two-sided test \(\frac{\alpha}{2} = Prob(X_T-X_C\geq v|H_0)\). That is, if we want to reject the null hypothesis, the difference \(X_T-X_C\) should be large enough. Some simple algebra is needed to derive the critical value:

\[\frac{\alpha}{2}=Prob(\mu_T-\mu_C\geq v|H_0)\] \[=1-Prob(X_T-X_C\leq v|H_0)\] \[=1-Prob(\frac{X_T-X_C-\mu_0}{\sigma_N}\leq \frac{v-mu_0}{\sigma_N}|H_0)\] \[=1-Prob(\frac{X_T-X_C}{\sigma_N}\leq \frac{v}{\sigma_N}|H_0)\](under Null hypothesis \(\mu_0=0\)) \[=1-\phi(\frac{v}{\sigma_N})\](\(\phi\) is standardized normal distribution function)

Let \(z_{1-\frac{\alpha}{2}}=\frac{v}{\sigma_N}\), and then \(v=z_{1-\frac{\alpha}{2}} \cdot \sigma_N\).

Since power is defined as the probability of accepting the alternative hypothesis given that the alternative is true \(1-\beta=Prob(Accept \quad H_A|H_A)=Prob(X_T-X_C\geq v|H_A)\), which can be expressed as: \[1-\beta=1-Prob(X_T-X_C\leq z_{1-\frac{\alpha}{2}} \cdot \sigma_N |H_A)\] \[=1-Prob(\frac{X_T-X_C-\mu_A}{\sigma_N}\leq \frac{z_{1-\frac{\alpha}{2}} \cdot \sigma_N-\mu_A}{\sigma_N}|H_A)\] \[=1-\phi(\frac{z_{1-\frac{\alpha}{2}} \cdot \sigma_N -\mu_A}{\sigma_N})\] \[=\phi(\frac{\mu_A}{\sigma_N}-z_{1-\frac{\alpha}{2}})\] As a result, \(z_{1-\beta}=\frac{\mu_A}{\sigma_N}-z_{1-\frac{\alpha}{2}}\). Under the alternative hypothesis, \(X_T-X_C \sim N(\mu_A,\sigma^2/n)\). Now suppose the variable of interest is binomial distributed. Then \(\sigma_N=\sqrt{X_T(1-X_T)/N_T + X_C(1-X_C)/N_C}\) \[\frac{1}{\sqrt{X_T(1-X_T)/N_T + X_C(1-X_C)/N_C}}= \frac{z_{1-\beta}+z_{1-\frac{\alpha}{2}}}{X_T-X_C}\] $ $

\[N_T=N_C=(X_T(1-X_T)+X_C(1-X_C))\cdot (\frac{z_{1-\beta}+z_{1-\frac{\alpha}{2}}}{X_T-X_C})^2\]

Hence, if we want to calculate the sample size, we only need to specify the expected rate/proportion in treatment, the expected rate/proportion in control, power, and the significance level. The code to calculate sample size for binary outcome assuming \(\alpha= .05\) is:

## Calculate Sample Size Based on Binary Outcome
SampleSize = function(PropTreat, PropCont, Power){
  N = ((PropTreat*(1-PropTreat)+PropCont*(1-PropCont))*

Using the same logic, we can get the formula to calculate the sample size for normally distributed outcome:

## Calculate Sample Size Based on Continuous Outcome
## Two sided test at 0.05 significance level
## kappa is the ratio of the sample size in control and treatment
## SDTreat is the standard deviation 
SampleSizeM = function(MeanTreat, MeanCont, SDTreat, SDCont, Power){
  NTreat =(SDTreat^2+SDCont^2)*

Is it the end of the world if \(\alpha = 0.005\)?

The original paper gave an answer for this question. It only requires the sample size to increase by 70%. Consider the following senariors:

  • Suppose the rate of response in the control group is 50%, 40%, …,10%;
  • Suppose the treatment will reduce 2%, 4%, 6%, and 10% percentage points .

That is, if the rate of reponse in the control group is 50%, we look at the desired sample size if the treatment will reduce the rate to 48%, 46%, 44%, and 40%. Similar to other base rate. As a result, we will have a 5 X 5 matrix for respective sample size. We compare difference between the case where \(\alpha = 0.05\) and \(\alpha = 0.005\).

SampleSize05 = function(PropTreat, PropCont, Power){
  N = ((PropTreat*(1-PropTreat)+PropCont*(1-PropCont))*
SampleSize005 = function(PropTreat, PropCont, Power){
  N = ((PropTreat*(1-PropTreat)+PropCont*(1-PropCont))*

pCon = c(0.5,0.4,0.3,0.2,0.1)
Reduced = c(0.02,0.04,0.06,0.08,0.1)
mSampleSiz05 = matrix(NA,nrow=5,ncol=5)
mSampleSiz005 = matrix(NA,nrow=5,ncol=5)
for (i in 1:5){
  for(j in 1:5){
    mSampleSiz05[i,j] = SampleSize05((pCon[i]-Reduced[j]),pCon[i],0.8)
    mSampleSiz005[i,j] = SampleSize005((pCon[i]-Reduced[j]),pCon[i],0.8)

(Increased.Sample.Size = (mSampleSiz005-mSampleSiz05)/mSampleSiz05)
##           [,1]      [,2]      [,3]      [,4]      [,5]
## [1,] 0.6960424 0.6961145 0.6952909 0.6947195 0.6961039
## [2,] 0.6960249 0.6958406 0.6959526 0.6939502 0.6949153
## [3,] 0.6960505 0.6965552 0.6962617 0.6965812 0.6941581
## [4,] 0.6961564 0.6955017 0.6944444 0.6963190 0.6903553
## [5,] 0.6957334 0.6954103 0.6964286 0.6888889 0.6901408

As can be seen, all are very clost to 70% given different baste rate and different magnitude of effect. From 0.05 to 0.005, we do not need 10 times of the sample size.