Why is chance capitalized

A previous blog in the statistical significance series outlines what the term means, and mentions an issue that we will focus on in this part of the series: identifying when the principles of chance may be at play.

In the most commonly used statistical models, p-values indicate the likelihood that a set of observed data is compatible with the absence of whatever effect is specified in a given hypothesis for example, that mean differences exist between protected classes.

This is called null hypothesis testing. So, a p-value of. Things get trickier when running multiple statistical tests on your data. In other words, some results may be false alarms i. For example, assume a company has AAPs and 30 job groups. By the time analyses are also run on promotions and terminations, we are up to 15, analyses. The Bonferroni correction is one of the most widely used, though there are others as well.

The probability of rejecting at least one true null hypothesis of no difference is much higher than five per cent if we use a significance level of five per cent for each single t test.

We can correct in several ways for this type of capitalization on chance; one such way is the Bonferroni correction. This correction divides the significance level that we use for each test by the number of tests that we do.

In our example, we do three t tests on pairs of groups, so we divide the significance level of five per cent by three. The resulting significance level for each t test is. The Bonferroni correction is a rather coarse correction, which is not entirely accurate. However, it has a simple logic that directly links to the problem of capitalization on chance. Therefore, it is a good technique to help understand the problem, which is the main goal we want to attain, here. We will skip better, but more complicated alternatives to Bonferroni correction.

Note that we do not have to apply a correction if we specify a hypothesis beforehand about the two groups that we expect to differ.

In the example of celebrity endorsement, we would not have to apply the Bonferroni correction to the t test on the mean difference between participants confronted to Celebrity A and Celebrity C if we had hypothesized that the willingness to donate differs here. Of course, we could have skipped the analysis of variance and gone straight to the t test with such a hypothesis. Capitalization on chance occurs if we apply different tests to the same variables in the same sample.

This occurs in exploratory research in which we do not specify hypotheses beforehand but try out different independent variables or different dependent variables. It occurs more strongly if we first have a look at our sample data and then formulate the hypothesis.

Knowing the sample outcome, it is easy to specify a null hypothesis that will be rejected. This is plain cheating and it must be avoided at all times. Summary 1.

Summary 2. Summary 3. Summary 4. Summary 5.