GOAL: To obtain valid evidence of student learning

Validity is built-in by matching behavioral objectives to test items / performance assessment rubrics.

Validity can be further increased or decreased by the assembling, administration, and scoring procedures.

To evaluate an item or task during assembly,

1. Set aside the items for a few days, then review them. As was stated before, the best sequence of events is first, identify your instructional goal and set of behavioral objectives. Second, create your assessment tool. Third, deliver instruction, and fourth, administer the assessment. If you have done these activities in this order, then you can review the test just prior to administering it and you will have, in fact, waited at least a few days between initially writing them and proofing them.
2. Ask a peer (fellow teacher) to review and critique the test items. The peer should be someone knowledgeable regarding the content you are assessing. It may be easier to arrange this if you offer to proof their assessment in return.

1. Is the format appropriate for the behavioral objective being measured?
2. Does the knowledge, understanding, or thinking skill called forth by the item or task match the specific behavioral objectives and subject matter being measured?
3. Is the point of the item or task clear?

5. Does the item have an answer that would be agreed upon by experts? How well would experts agree about the degree of excellence of task performance?

1) Like items should be grouped together

2) Ascending order by difficulty or complexity

3) Do not present both performance tasks and objective test items in the same administration (this is strictly an issue of

1) Should be written; do not rely on students' auditory memory

2) Sections to be included:

1. Do not increase anxiety
2. Allow plenty of time (Rule of Thumb: For objective test items, one minute per item. For performance assessments, add 5-10 minutes to however much time it took them to perform during practice)
3. Provide a quiet, well-lit, comfortable area with adequate space
4. Walk around the room. This discourages cheating and makes you more accessible to those who have questions.
5. For Performance Assessments: Provide students with the rubric you will use to score their performance. AND Encourage them to use the same instrument (ex: piano), tool (ex: hammer), or equipment (ex: tennis racket) that they used to practice with. You put them at an unfair disadvantage if you select the instrument, tool, or equipment for them and inadvertently hand them one they have never used.

1. Score the test according to pre-determined rules / rubrics
2. Check for reliability (consistency) of scoring
3. Correct for guessing when necessary

Did the item function as it was intended to?

1. Can help evaluate the effectiveness of each item and of the test as a whole
2. Will enable the instructor to build an item bank (test file)
3. Provides a basis for remedial work
4. Provides a basis for improved classroom instruction

How are responses analyzed?

1. Create a matrix with students' names in the first column and their response to each item in each of the following columns. If you have 50 questions, you will have one column for each of the 50 questions. FOR PERFORMANCE ASSESSMENT: The first column is the same, the following columns are for the different aspects or sections of your rubric. Each aspect or section would require enough columns for each of the raters. If you have used three raters, you will need three columns for each aspect or section of the rubric. You would put in these columns the ratings received for each aspect or section.
2. Mark all responses that are incorrect. A highlighter pen works well for this. FOR PERFORMANCE ASSESSMENT: Mark all responses that do not agree.
3. Look down each column. Count the number of correct responses and divide this by the number of students taking the exam. If this number equals or exceeds 80%, fine. If not, you need to review the item to determine if you feel you covered it adequately. If for some reason you suspect the error is with your instruction or the item itself, you may want to consider throwing this item out. You will need to make a change either in the item or in your instruction in the future (before administering the assessment again). FOR PERFORMANCE ASSESSMENTS: Count the number of agree's and divide by the total number of responses. The 80% rule still applies.
4. Again, looking down each column, count the number of times each option is selected. If the incorrect responses are equally distributed across wrong options, then the item is fine. If the incorrect responses are all the same wrong option, then your distractors are not working well and the item will need to be revised before administering the assessment again. FOR PERFORMANCE ASSESSMENTS: Areas (aspects or sections) that receive low agreement, less than 80%, indicate that the descriptor for that area is unclear and needs revision.
5. Calculate the reliability. If the estimated reliability is below .5, then work needs to be done to revise the instrument. If the estimated reliability is between .5 and .8, you still need to examine whether or not the reliability of the instrument can be improved. If the estimated reliability is above .8, there is cause to celebrate! FOR PERFORMANCE ASSESSMENTS: Your reliability is your rater agreement which is averaged across all aspects or sections. Do not tolerate less than .8.
6. You can calculate the discriminating power of your items, but this is of greater concern with standardized, norm-referenced tests than with classroom tests. I do not require this to be done for the project. FOR PERFORMANCE ASSESSMENTS: This can not be done.

NOTE: There is computer software available that will do this for you for the objective tests that are recorded on bubble sheets, and it is often available in the larger high schools.

Classroom assessments can be improved by using simple methods to analyze student responses and by building a file of effective items and tasks.