The Future of Improvement – Learn More

Hypothesis Testing

The Hypothesis Testing document contains various hypothesis tests for various usages (for both attribute and variable data). Tests included:

Open the named file, and the Introduction tab will be presented. The document contains instructions as needed.

General Guidance

Each hypothesis test follows a standard structure, the left side contains instructions, setup, and data entry of the test. The right side will then display relevant statistical results and graphical analysis for deductions.

Figure 1 – Hypothesis Test Example showing breakdown of each tab.

The instructions on each test will vary however all will contain the following:

  1. When to use this test – explaining the purpose of the test and appropriate application.
  2. How to use this test – specific step by step instructions on completing the test.
  3. Data Considerations – Any considerations / assumptions that should be taken into account before applying the test.
  4. Additional Notes – any additional information that may be useful for those using the test but not essential reading.
  5. Results Report – Full statistical and graphical results of the test.
This article will explain briefly terminology on each tab to allow for correct application of each tool. Results/reports will be covered by your training (training material and trainer).

2 Sample t & Paired t

2 Sample t is used if you want to determine whether the population means of two independent samples differ or if you want to check the range of potential difference. For example, an improvement practitioner wants to compare satisfaction ratings between two hotels to see if there is a significant difference in satisfaction between the two.

Paired t is used if you want to determine whether the population means of two dependent samples differ or if you want to check the range of potential difference. For example, an improvement practitioner wants to compare the same product that has undergone coating treatment to see if the hardness of the product has changed.

Figure 2 – 2 Sample t guide

  • Brief title for 1st and 2nd samples – this is to correctly title tables and graphs on the report. This is purely for organisation purposes so they can be read and understood more easily.
  • Alpha level – this is the error margin you are willing to accept for the results of the test. For example, an alpha of 0.05 is equivalent to 5% error, which is equivalent to 95% confidence. An alpha of 0.05 is indicating one is willing to accept 5% risk of Type 1 error (rejecting the null when the null is true).
  • Hypothesised difference – this defines your null hypothesis, this is a target value you may be striving for. In most cases, this is left blank as there may not be a target difference the process is aiming for, simply trying to increase/decrease to any margin. For example setting hypothesised difference to 1 will make the null hypothesis x1-x2=1 instead of x1-x2=0 by default.
  • Alternative Hypothesis – the Alternative or Experimental Hypothesis reflects that there will be a statistically observed effect for the test. This is the hypothesis if the null is not true it is what we are attempting to demonstrate in an indirect way by the use of our hypothesis test. For example if a null hypothesis is x1-x2=0, the alternative will be x1-x2≠0 (not equal to).
  • (2 Sample t only) Variance assumption – By default, the hypothesis test must have an underlying understanding if variance in data are equal or unequal. Variance is a representation of spread/variation in a data set. Variance should commonly be the same across both samples, if this is not the case, it can increase risks of type 1 error and therefore lead to false positives. By default if you are unsure, leaving this to choose for me will perform a variance test and adapt according to the result.

2 Sample SD

Used if you want to determine whether the population variance / standard deviation of two samples differ or if you want to check the range of potential variance. You may choose this over 2 Sample or Paired t tests if the variation is of greater focus than the shift in means.

  • Brief title for 1st and 2nd samples – this is to correctly title tables and graphs on the report. This is purely for organisation purposes so they can be read and understood more easily.
  • Alpha level – this is the error margin you are willing to accept for the results of the test. For example, an alpha of 0.05 is equivalent to 5% error, which is equivalent to 95% confidence. An alpha of 0.05 is indicating one is willing to accept 5% risk of Type 1 error (rejecting the null when the null is true).

1 Sample T

Used if you want to determine whether the population mean of a sample differs from a target mean or to view the confidence interval range of a sample. Unlike a 2 sample t test, this only has 1 data set that is compared to a reference / target mean instead.

  • Brief title for sample – this is to correctly title tables and graphs on the report. This is purely for organisation purposes so they can be read and understood more easily.
  • Alpha level – this is the error margin you are willing to accept for the results of the test. For example, an alpha of 0.05 is equivalent to 5% error, which is equivalent to 95% confidence. An alpha of 0.05 is indicating one is willing to accept 5% risk of Type 1 error (rejecting the null when the null is true).
  • Target mean – this defines your null hypothesis, this is a target mean you are comparing your sample against.

1 Sample SD

Used if you want to determine whether the population standard deviation of a sample differs from a target standard deviation or to view the confidence interval range of a sample. Unlike a 2 sample SD test, this only has 1 data set that is compared to a reference / target standard deviation instead.

  • Brief title for sample – this is to correctly title tables and graphs on the report. This is purely for organisation purposes so they can be read and understood more easily.
  • Alpha level – this is the error margin you are willing to accept for the results of the test. For example, an alpha of 0.05 is equivalent to 5% error, which is equivalent to 95% confidence. An alpha of 0.05 is indicating one is willing to accept 5% risk of Type 1 error (rejecting the null when the null is true).
  • Target standard deviation – this defines your null hypothesis, this is a target standard deviation you are comparing your sample against.

1 Sample % Defective (1 Proportion)

Used if you want to determine whether the binomial population proportion is different to a target/reference value when your data contains 2 categories (such as Yes/No). You can use it to determine if a population % differs from a reference % or calculate a confidence interval of the population %.

   

  • Alpha level – this is the error margin you are willing to accept for the results of the test. For example, an alpha of 0.05 is equivalent to 5% error, which is equivalent to 95% confidence. An alpha of 0.05 is indicating one is willing to accept 5% risk of Type 1 error (rejecting the null when the null is true).
  • Reference proportion – this defines your null hypothesis, this is a target % defective rate you are comparing your sample against.
  • Number of events – this is the number of occurrences (defectives) encountered in your sample. For example, if you checked 1000 documents and 23 of them were defective, this would be 23.
  • Number of opportunities – this is the number of items sampled in total. For example, if you checked 1000 documents and 23 of them were defective, this would be 1000.
  • Alternative Hypothesis – the Alternative or Experimental Hypothesis reflects that there will be a statistically observed effect for the test. This is the hypothesis if the null is not true it is what we are attempting to demonstrate in an indirect way by the use of our hypothesis test. For example if a null hypothesis is x1-Reference=0.065 (6.5%), the alternative will be x1-Reference≠0.065 (not equal to).

 

2 Sample % Defective (2 Proportion)

Used if you want to determine whether 2 binomial population proportions are different when your data contains 2 categories (such as Yes/No). You can use it to determine the differences between samples or calculate confidence intervals of the population %. 

  • Alternative Hypothesis – the Alternative or Experimental Hypothesis reflects that there will be a statistically observed effect for the test. This is the hypothesis if the null is not true it is what we are attempting to demonstrate in an indirect way by the use of our hypothesis test. For example if a null hypothesis is x1-Reference=0.065 (6.5%), the alternative will be x1-Reference≠0.065 (not equal to).
  • Number of events – this is the number of occurrences (defectives) encountered in your sample. For example, if you checked 1000 documents and 23 of them were defective, this would be 23.
  • Number of opportunities – this is the number of items sampled in total. For example, if you checked 1000 documents and 23 of them were defective, this would be 1000.
  • Alpha level – this is the error margin you are willing to accept for the results of the test. For example, an alpha of 0.05 is equivalent to 5% error, which is equivalent to 95% confidence. An alpha of 0.05 is indicating one is willing to accept 5% risk of Type 1 error (rejecting the null when the null is true).
  • Hypothesised difference – this defines your null hypothesis, this is a target value you may be striving for. In most cases, this is left blank as there may not be a target difference the process is aiming for, simply trying to increase/decrease to any margin. For example setting hypothesised difference to 1 will make the null hypothesis x1-x2=1 instead of x1-x2=0 by default.
  • Alternative Hypothesis – the Alternative or Experimental Hypothesis reflects that there will be a statistically observed effect for the test. This is the hypothesis if the null is not true it is what we are attempting to demonstrate in an indirect way by the use of our hypothesis test. For example if a null hypothesis is x1-Reference=0.065 (6.5%), the alternative will be x1-Reference≠0.065 (not equal to).

Chi2 Goodness of Fit (Chi Fit)

Used if you want to determine whether proportions of items in different categories are significantly different from specified/expected proportions. You can apply CESP to up to 10 categories.

    

The above example is comparing trouser sales in a shop vs expected sales at a given period.

  • Category – the name of the categories that you are comparing, this is so you can specifically name categories for ease of reference/understanding.
  • Observed Count – this is the number of occurrences encountered in your sample. For example, if a shop sold 41 pairs of medium sized trousers during the given period, 41 would be entered next to the medium category. This is how many “events” being monitored were seen.
  • Expected % – the % (proportion) that one expected that particular category to account for of the entire samples. For example the sales manager expected large size trousers to account for 40% of all of their sales for the given period.
  • Alpha level – this is the error margin you are willing to accept for the results of the test. For example, an alpha of 0.05 is equivalent to 5% error, which is equivalent to 95% confidence. An alpha of 0.05 is indicating one is willing to accept 5% risk of Type 1 error (rejecting the null when the null is true).

Chi2 Test for Association (Chi Association)

If you want to determine whether proportions of items in different categories are significantly different or associated to one another depending on the category of the second variable.

    

The above example is comparing correctly delivered (Yes) to incorrectly delivered (No) products across a range of different products.

  • Items / Name – name of the categories or items that you are comparing. This is so you can specifically name categories for ease of reference/understanding.
  • Title – these are the names of the sub-categories being compared against each item.
  • Alpha level – this is the error margin you are willing to accept for the results of the test. For example, an alpha of 0.05 is equivalent to 5% error, which is equivalent to 95% confidence. An alpha of 0.05 is indicating one is willing to accept 5% risk of Type 1 error (rejecting the null when the null is true).
Related

Join our LinkedIn Network

Keep up to date with all the latest performance improvement resources to inform and inspire as you continue to grow.

Follow PMI on LinkedIn