P-value
Optional stopping

The difference between the two meanings of "extreme" appear when we consider a sequential hypothesis testing, or optional stopping, for the fairness of the coin. In general, optional stopping changes how p-value is calculated. Suppose we design the experiment as follows:

This experiment has 7 types of outcomes: 2 heads, 2 tails, 5 heads 1 tail,..., 1 head 5 tails. We now calculate the p -value of the "3 heads 3 tails" outcome.

If we use the test statistic heads / tails {\displaystyle {\text{heads}}/{\text{tails}}}, then under the null hypothesis is exactly 1 for two-sided p -value, and exactly 19 / 32 {\displaystyle 19/32} for one-sided left-tail p -value, and same for one-sided right-tail p -value.

If we consider every outcome that has equal or lower probability than "3 heads 3 tails" as "at least as extreme", then the p -value is exactly 1 / 2. {\displaystyle 1/2.}

However, suppose we have planned to simply flip the coin 6 times no matter what happens, then the second definition of p -value would mean that the p -value of "3 heads 3 tails" is exactly 1.

Thus, the "at least as extreme" definition of p -value is deeply contextual and depends on what the experimenter planned to do even in situations that did not occur.
detectable_content:number_placeholders[5]