By Sandy DiLena, Education Officer, EQAO
Issue 3, 2008
This article describes the process that EQAO uses to create open-response questions (also called open-response assessment items), and shares lessons learned and information that could be useful to classroom teachers in the development of their own classroom assessments. EQAO measures the achievement of mathematics curriculum expectations of students across Ontario at the end of Grades 3, 6 and 9. EQAO results are reported at the provincial, school board, school and individual student levels. They are used by the Ministry of Education, school boards and schools to improve learning, teaching and student achievement.
EQAO’s assessments are summative. The reports provide summaries using achievement levels to indicate how well students are achieving the curriculum expectations. The following table describes the differences between EQAO’s large-scale assessments and classroom assessment. This table can be found in the framework for each of the assessments on EQAO’s Web site,
EQAO’s Large-Scale Assessments
Each fall, a committee of teachers from across the province creates questions that will be field tested in EQAO assessments a year and a half later. Using the field-test data generated during scoring, EQAO selects the best performing questions, which become the assessments for the following year. It takes approximately three years from the time it is written for a question to contribute to a student’s score.
Ontario educators with experience in the assessed grades are chosen to write questions. The question writing is done in two sessions of two days each during which the writers are trained and supported by EQAO education officers.
During the first session, the writers draft questions and any necessary graphics based on the following directions:
Between the first two sessions, the questions are edited and laid out produced by EQAO staff. Then they are sent back to the writers who try the questions out with students in cognitive labs. The purpose of the cognitive labs is to gather feedback from students about the success of the questions, not the success of the students (e.g., is a question clearly stated? do the students understand what they are being asked to do?) At this stage, the item writers are required to test their questions on students of the appropriate grade with a range of abilities. The writers sit with the students to discuss the clarity of the wording, graphics and presentation. Teachers are asked to reflect on student responses with questions such as:
During the second session, the writers use the information gathered during the cognitive labs to edit their questions. The writers also engage in peer editing with their colleagues to reflect critically on their questions. Once the editing is complete, using a generic rubric the writers develop question-specific rubrics to be used for scoring.
All EQAO assessments are configured according to an assessment blueprint. The blueprint clusters the curriculum expectations according to the number of open-response questions in each grade’s assessment. Some expectations cannot be assessed by paper and pencil tests and need to be assessed by alternative methods, such as performance tasks, journals and investigations. EQAO identifies the expectations not addressed by its assessments using italics in the blueprints.
Classroom teachers can consider many of these directions when putting together their own tests
The blueprint gives the number and types of questions to come from each cluster as well as the skill from the achievement chart each question is to address. Each open-response question is designed to assess one specific expectation from the mathematics curriculum and one skill from the Ministry’s achievement chart for mathematics, and to take students five minutes on average to complete.
As a classroom teacher, what are your opportunities to review the quality of your test questions with your colleagues?
It is of utmost importance that each question measure the specific expectation to which it is mapped. EQAO has numerous checks to ensure this mapping. All potential questions are presented to an Assessment Development Committee (ADC) composed of six Ontario educators per grade for review prior to field testing in addition to internal reviews by EQAO staff and peer reviews by other writers. The ADC brings “fresh eyes” and reviews the questions according to the following criteria:
Can the question be answered correctly by students who have acquired the skill being assessed?
Does the question assess the assigned
Is the question mathematically sound?
Are the following appropriate for the designated grade?
Is the question clear, concise and grammatically correct?Do the graphics assist or hinder? Are they clear and accurate?Does the question allow for scoring according to the four performance codes (see the generic rubric)?Does the question take about five minutes to complete?
What quality aassurance measures do you use when creating your classroom tests?
Following the ADC meeting, the same questions are presented to a Sensitivity Committee composed of Ontario educators for a review to ensure they are free of content that might cause a particular student to be advantaged or disadvantaged. This committee also uses the following questions to check for sensitive content:
Is the question
During training for item writing, the writers are given descriptions and samples of questions that address the skills outlined in the achievement chart for mathematics. For example, the skill “knowledge and understanding” is exhibited when students demonstrate subject-specific content (knowledge) and the comprehension of its meaning and significance (understanding). When Grade 9 academic students are asked to add or subtract polynomials, they are demonstrating knowledge and understanding.
The skill of “thinking” is exhibited when students engage in a critical-thinking process to select and sequence a variety of tools to solve a problem. To answer a question that addresses this skill, students need to make a plan. The plan is demonstrated through their solution process.
The skill of “application” is exhibited when students select the appropriate mathematical tool or get the necessary information and “fit” it to the problem. For example, on the primary division assessment, a question that states the number of stickers that each of two children has and asks for the total number of stickers would be considered a question addressing application. To answer the question, a student needs to choose the tool of addition and apply it correctly.
To illustrate further, the following examples have been chosen from EQAO’s released materials.
Example 1 (Junior Division, Spring 2007, Question 9)
This sample addresses the skill application. To be successful, students need to select from their knowledge information about a trapezoid and accurately apply it to draw, measure and label the required construction.
Example 2 (Primary Division, Spring 2007, Question 30)
This question addresses the skill thinking. In order to be successful, a student’s plan would include selecting the tool of addition to add $5 amounts until a total close to $18 was reached. Next, the student would need to deal with the fact that $18 is not a multiple of $5 and, finally, determine how many bundles of newspapers needed to be delivered.
Perhaps the most challenging part of writing open-response questions is ensuring they are rich enough to allow students to demonstrate four distinct performance codes. Sometimes this challenge becomes evident only after students have attempted the question. As mentioned above, question writers create question-specific rubrics with four codes to be used to score their open-response questions. These rubrics include a description of a sample response corresponding to each of the four codes of performance. These descriptions force the writers to think about and be accountable for whether their questions allow four distinguishable performance codes.
EQAO does not use open-response questions to address the skill knowledge and understanding, because such questions do not provide an opportunity for students to demonstrate four performance codes. These questions tend to elicit an answer that is either correct or wrong. Because of this, EQAO measures knowledge and understanding using only multiple-choice questions.
As stated above, one of the key differences between large-scale assessment and classroom assessment is the method according to which the open-response questions are scored. In the classroom, the teacher is usually the one who decides how well a student has performed. At the EQAO scoring site, a large number of educators are trained to score student responses by matching them to well-chosen anchor papers. Another committee of Ontario educators, the Range-Finding Committee, meets to select these papers. The anchor papers exemplify each of the codes on the scoring rubric.
Sometimes questions present scoring problems after field testing and cannot be used on an operational test.
Consider the example below.
This question was not chosen for the EQAO assessment, because there are so many items in this question for students to choose from. Every student response must be scored validly and reliably, and maintaining scoring consistency here would be very difficult. This question would be better in a classroom, where discussion could take place.
Consider the following question.
It was written to address the expectation from the Data Management and Probability strand that states that “students will express theoretical probability as a ratio of the number of favourable outcomes to the total number of possible outcomes, where all outcomes are equally likely.” Because all outcomes on the spinner are not equally likely, this question does not assess the curriculum expectation. Further, if the spinner were like the one to the left, on which all outcomes are equally likely, the question would not elicit the four performance codes.
Correctly answering the question now involves counting the number of red, green and yellow sections and expressing this as a fraction with a denominator of 16. This question was not used on an EQAO assessment.
We hope that this article has provided insight into the process EQAO uses to ensure that its data are valid and reliable. We also hope that you will be able to adapt this process to benefit your classroom.
Good-quality assessment questions are the cornerstone of good-quality assessment. You are invited to
apply to participate on one of the committees mentioned in this article: Item Writing, Assessment Development, Sensitivity or Range-Finding.