Module Five
Reliability
 Home
Site Map
 Modules
Module 5 Notes
  WebBoard
Resources
 
STEPS FOR CALCULATING KR-20
    1. Create a matrix of wrong / right for each question and student. 
 
Student Name
1
2
3
4
5
6
Alice
0
0
0
0
0
0
Bob
0
0
0
0
1
0
Carol
1
0
1
1
1
0
David
1
1
1
1
1
1
Esther
1
1
1
1
1
1
Frank
0
0
1
0
0
0
Gloria
0
0
1
1
1
0
Hank
0
0
0
1
0
0
Irene
1
0
1
1
1
0
Jack
0
1
0
1
0
1
 
 
    2. Sum the one's and zero's across for each student 
 
Student Name
1
2
3
4
5
6
Total
Alice
0
0
0
0
0
0
0
Bob
0
0
0
0
1
0
1
Carol
1
0
1
1
1
0
4
David
1
1
1
1
1
1
6
Esther
1
1
1
1
1
1
6
Frank
0
0
1
0
0
0
1
Gloria
0
0
1
1
1
0
3
Hank
0
0
0
1
0
0
1
Irene
1
0
1
1
1
0
4
Jack
0
1
0
1
0
1
3
 
    3. Sum the total down 
 
Student Name
1
2
3
4
5
6
Total
Alice
0
0
0
0
0
0
0
Bob
0
0
0
0
1
0
1
Carol
1
0
1
1
1
0
4
David
1
1
1
1
1
1
6
Esther
1
1
1
1
1
1
6
Frank
0
0
1
0
0
0
1
Gloria
0
0
1
1
1
0
3
Hank
0
0
0
1
0
0
1
Irene
1
0
1
1
1
0
4
Jack
0
1
0
1
0
1
3
 
TOTAL
29
 
    4. Calculate the mean by dividing the sum of the total scores by the number of students 
 
Student Name
1
2
3
4
5
6
Total
Alice
0
0
0
0
0
0
0
Bob
0
0
0
0
1
0
1
Carol
1
0
1
1
1
0
4
David
1
1
1
1
1
1
6
Esther
1
1
1
1
1
1
6
Frank
0
0
1
0
0
0
1
Gloria
0
0
1
1
1
0
3
Hank
0
0
0
1
0
0
1
Irene
1
0
1
1
1
0
4
Jack
0
1
0
1
0
1
3
 
TOTAL
29
 
 
2.9 = 29 divided by 10
 
 
 
    5. Subtract the mean from each total score 

    Remember:  Watch your signs!  

    When the mean is larger  than the total for a particular student, then the difference will be a negative number.  

    Example: Alice has a total of zero. Her total is less than the mean of 2.9, so the difference is negative.  

    When the mean is smaller than the total for a particular student, then the difference will be a positive number. 

    Example: Carol has a total of 4. Her total is greater than the mean of 2.9 so the difference is positive. 
     

 
Student Name
1
2
3
4
5
6
Total
Total
Minus 
Mean
Alice
0
0
0
0
0
0
0
- 2.9
Bob
0
0
0
0
1
0
1
- 1.9
Carol
1
0
1
1
1
0
4
+ 1.1
David
1
1
1
1
1
1
6
+ 3.1
Esther
1
1
1
1
1
1
6
+ 3.1
Frank
0
0
1
0
0
0
1
- 1.9
Gloria
0
0
1
1
1
0
3
+ 0.1
Hank
0
0
0
1
0
0
1
- 1.9
Irene
1
0
1
1
1
0
4
+ 1.1
Jack
0
1
0
1
0
1
3
+ 0.1
 
TOTAL 29
0.0
 
    6. If you add this column,Total Minus Mean, the sum is ALWAYS zero. If this column does not add to zero, you have done your math incorrectly. 
 
Student Name
1
2
3
4
5
6
Total
Total
Minus
Mean
Alice
0
0
0
0
0
0
0
- 2.9
Bob
0
0
0
0
1
0
1
- 1.9
Carol
1
0
1
1
1
0
4
+ 1.1
David
1
1
1
1
1
1
6
+ 3.1
Esther
1
1
1
1
1
1
6
+ 3.1
Frank
0
0
1
0
0
0
1
- 1.9
Gloria
0
0
1
1
1
0
3
+ 0.1
Hank
0
0
0
1
0
0
1
- 1.9
Irene
1
0
1
1
1
0
4
+ 1.1
Jack
0
1
0
1
0
1
3
+ 0.1
 
TOTAL
 
 
 
 
 
 
 
 
 29
0.0
 
 
     7. Now square each of the Total-Mean's; remember, when you square a number, it always results in a positive number. 
 
Student Name
1
2
3
4
5
6
Total
Total
Minus
Mean
(Total
Minus
Mean)
Squared
Alice
0
0
0
0
0
0
0
- 2.9
8.41
Bob
0
0
0
0
1
0
1
-1.9
3.61
Carol
1
0
1
1
1
0
4
+1.1
1.21
David
1
1
1
1
1
1
6
+3.1
9.61
Esther
1
1
1
1
1
1
6
+3.1
9.61
Frank
0
0
1
0
0
0
1
-1.9
3.61
Gloria
0
0
1
1
1
0
3
+0.1
0.01
Hank
0
0
0
1
0
0
1
-1.9
3.61
Irene
1
0
1
1
1
0
4
+1.1
1.21
Jack
0
1
0
1
0
1
3
+0.1
0.01
 
TOTAL
 
 
 
 
 
 
 
29
0.0
40.9
 
 
     8. Sum the [(Total-Mean) Square] column
 
Student Name
1
2
3
4
5
6
Total
Total Minus 
Mean
(Total
Minus Mean)
Squared
Alice
0
0
0
0
0
0
0
-2.9
8.41
Bob
0
0
0
0
1
0
1
-1.9
3.61
Carol
1
0
1
1
1
0
4
+1.1
1.21
David
1
1
1
1
1
1
6
+3.1
9.61
Esther
1
1
1
1
1
1
6
+3.1
9.61
Frank
0
0
1
0
0
0
1
-1.9
3.61
Gloria
0
0
1
1
1
0
3
+0.1
0.01
Hank
0
0
0
1
0
0
1
-1.9
3.61
Irene
1
0
1
1
1
0
4
+1.1
1.21
Jack
0
1
0
1
0
1
3
+0.1
0.01
 
TOTAL
 
 
 
 
 
 
 
 29
0.0
40.9
 
 
You should get 40.9; this is called sum of squares.
 
 
    9. Divide this by the number of students minus 1 
 
40.9 divided by (10 minus 1) = 4.544; this is called the variance.
 
 
     

    10. Take the square root of the variance 

 
You should get 2.132; this is called the standard deviation.
 
 
    11. Now calculate what percent of the students got each test question correct. We do this by adding down each column and dividing by the number of students which in this example is 10. This is called "p". 
 
Student Name
1
2
3
4
5
6
Total
Total
Minus 
Mean
(Total
Minus
Mean)
Squared
Alice
0
0
0
0
0
0
0
-2.9
8.41
Bob
0
0
0
0
1
0
1
-1.9
3.61
Carol
1
0
1
1
1
0
4
+1.1
1.21
David
1
1
1
1
1
1
6
+3.1
9.61
Esther
1
1
1
1
1
1
6
+3.1
9.61
Frank
0
0
1
0
0
0
1
-1.9
3.61
Gloria
0
0
1
1
1
0
3
+0.1
0.01
Hank
0
0
0
1
0
0
1
-1.9
3.61
Irene
1
0
1
1
1
0
4
+1.1
1.21
Jack
0
1
0
1
0
1
3
+0.1
0.01
% Correct
0.4
0.3
0.6
0.7
0.6
0.3
 
    12. Calculate what percent of students got each test question incorrect. We do this by counting the number of zeros in each column and dividing by the number of students (10). This is called "g".
 
     
    Student Name
    1
    2
    3
    4
    5
    6
    Total
    Total
    Minus
    Mean
    (Total
    Minus
    Mean)
    Squared
    Alice
    0
    0
    0
    0
    0
    0
    0
    -2.9
    8.41
    Bob
    0
    0
    0
    0
    1
    0
    1
    -1.9
    3.61
    Carol
    1
    0
    1
    1
    1
    0
    4
    +1.1
    1.21
    David
    1
    1
    1
    1
    1
    1
    6
    +3.1
    9.61
    Esther
    1
    1
    1
    1
    1
    1
    6
    +3.1
    9.61
    Frank
    0
    0
    1
    0
    0
    0
    1
    -1.9
    3.61
    Gloria
    0
    0
    1
    1
    1
    0
    3
    +0.1
    0.01
    Hank
    0
    0
    0
    1
    0
    0
    1
    -1.9
    3.61
    Irene
    1
    0
    1
    1
    1
    0
    4
    +1.1
    1.21
    Jack
    0
    1
    0
    1
    0
    1
    3
    +0.1
    0.01
    %
    Correct
    .4
    .3
    .6
    .7
    .6
    .3
    % Incorrect
    .6
    .7
    .4
    .3
    .4
    .7
     
 
 
    13. Multiply each of the percents correct with its corresponding percent incorrect. This is called p x g.
 
% Correct
x
% Incorrect
=
?
0.4
x
0.6
=
0.24
0.3
x
0.7
=
0.21
0.6
x
0.4
=
0.24
0.7
x
0.3
=
0.21
0.6
x
0.4
=
0.24
0.3
x
0.7
=
0.21
 
             14. Calculate the sum of these products; this is called the sum of p x g. 
 
0.24 + 0.21 + 0.24 + 0.21 + 0.24 + 0.21 = 1.35
 
 
    15. Here is the formula for KR-20 
 
    Number of Items divided by the number of items minus one OR 6 divided by 5 = 1.2
     

    One minus the Sum of (% correct x % incorrect) divided by the variance  
    OR 1 - (1.35 / 4.544) = 1 - 0.297 = 0.703 
    Now multiply these two numbers 1.2 x 0.703 = .84
    The reliability is estimated to be .84 using KR-20.
 
       Remember: Reliability estimates range from 0 to 1. There are no negative reliability estimates. 
     
 
Readings

    Chapter 4  Reliability and other desired characteristics 

    from Linn R.L. & Gronlund, N.E. (1995). Measurement and assessment in teaching. Englewood Cliffs, NJ: Merrill. 
  
Home
Site Map
Modules
Module 5 Notes
WebBoard
Resources
 
Updated last February 1999 by CF&MD staff.  
Please direct comments to the website developer.  
Copyright 1999 FGCU, All rights reserved.
 		   
Florida Gulf Coast University  
College of Professional Studies  
School of Education