By Michael Kozlow, Director of Data and Support Services, EQAO
Issue 3, 2008
It is useful to consider the process of interpreting EQAO assessment data as a journey through several layers of information. Assuming the role of detective can help you use the clues contained in the EQAO assessment reports to answer the following fundamental question: What do the assessment results tell me about how well my students are meeting the expectations in
The Ontario Curriculum?
Dealing with the various layers of data from the EQAO assessments may appear to be a complex task, but it is quite manageable if approached in a systematic manner. Working with the data can be simplified if you ask a series of key questions about each level of data that help answer the fundamental question above. Asking the right questions can help you demystify the data and reveal clues that can be useful in guiding decisions about instruction. As the detective, you are looking for meaning, patterns and relationships.
A prerequisite for demystifying the data is recognizing that you do not need to be a statistician to understand the numbers. While the statistical procedures for equating assessments from one year to the next and for generating the results are complex, the data actually presented in EQAO reports are not. Most of the results are expressed as percentages, which educators deal with on a daily basis and readily understand.
A key point to keep in mind is that the percentages in the EQAO reports are different from the percentages that are most commonly used in classrooms. Teachers frequently convert a student’s scores on a test to a percentage of the total points on the test or calculate an average percentage of all student scores in a class or in the school. The percentages in the EQAO reports are percentages of students demonstrating specified achievements. When EQAO reports that 78% of the Grade 3 students in the school met the provincial standard, this does not mean that the students in the school have an average of 78% on the test.
While the discussion in this article applies to all EQAO assessments, performance levels and the provincial standard apply most directly to the primary, junior and Grade 9 assessments.
Before you delve into the data with your questions, consider the following guiding principles to put the results into context:
Become familiar with the assessment frameworks as they relate to the curriculum, the assessment items and the provincial results prior to your detailed examination of school results.
EQAO publishes a variety of reports that present different perspectives on the assessment results. Using the full range of reports will enable you to get a more complete understanding of your students’ achievement and will enable you to take full advantage of all aspects of the results as you explore the data for your school. To some extent this is like peeling an onion: you encounter many layers and there may be some discomfort as you peel, but the flavourful rewards are obvious. The key questions in the rest of this article will lead you through the following layers of data:
The dialogue provoked by solid data makes a significant contribution to improving student learning. In this sense, data rests at the heart of good teaching. Of course, numbers alone can never tell the whole story. But they can prompt you to ask the questions that will lead you to a better understanding of student achievement.
The following questions will assist you in interpreting the EQAO assessment data and converting the numbers in the various reports into useful information for making decisions:
Start your exploration of the EQAO assessment data with your school report. This report gives a good indication of how the students in your school performed relative to those in previous years, and relative to those throughout the board and the province. Consider the following sub-questions:
It is important to examine both sets of results in the reports: those of all students and those of participating students. Educators sometimes ask, “Which of these sets of results is correct or more valid?” The answer is that they are both equally correct and valid, but that they provide different information. It is important to know the results of all students in the school, since educators are accountable for all students. It is also important to consider the results of those students who participated in the assessment to evaluate the impact of instructional strategies on them. School staff members will know which students were exempted and what the school is doing to meet their needs. These efforts must be evaluated in terms of expectations for these students and considerations of when they might be ready to write the EQAO assessments.
The Cohort Tracking Report provides an additional perspective on change in student performance over time, since it tracks individual students as they move through the school system. The report provides data on how a given cohort’s achievement has changed from Grade 3 to Grade 6. This will help you develop an understanding of the impact that your school is having on students as they progress from Grade 3 to Grade 6.
It is important to determine the characteristics of the students who did not achieve the provincial standard. Which subgroups are over-represented among this group? The school report includes information on achievement among a number of subgroups:
For example, of the 48 291 Grade 3 students in the province who did not achieve the provincial standard in reading in 2008,
Instructional strategies must be differentiated to meet the specific needs of each subgroup and those of the individual students within these subgroups.
There are two sources of data that will help you identify areas in which students demonstrated strength and areas in need of improvement: the “Profile of Strengths and Areas for Improvement” and the “Item Information Report: Student Roster.” The Profile report gives an indication of strengths and areas in need of improvement in relation to the skills and strands covered by an assessment. The results for your school are given relative to the results for the province as a whole. For example, if the percentage of students in your school showing strength in Strand A is larger than the percentage showing strength in Strand B, the performance in Strand A among students in your school is stronger relative to the performance of students in the province than their performance in Strand B is. This report provides a quick indication of the achievement of students across the strands and skills.
It is also useful to review the summary data about individual items for the school, board and province in the Student Roster reports. It is particularly useful to look at the results for groups of items. For example, how does your school’s performance on multiple-choice items compare with that on open-response items? How did your students perform on the long-writing task? In making these comparisons, always consider the results for your students in relation to those for the province.
You should not draw conclusions about performance on specific expectations based on the results of one or two items. However, it can be instructive to examine the results for individual items that students in your school did not answer well. Just keep in mind that any conclusions about instructional issues must be limited to the specific content of that item and manner in which the item is constructed. For multiple-choice items, look at which wrong options the students in your school selected.
In all of your discussions, always refer to
The Ontario Curriculum?
The final layer of information is the achievement of individual students, presented in the “Individual Student Report” and in the “Item Information Report: Student Roster.” Some questions to ask: Who are the students achieving each level? Which students were exempted? Which students are reported as having no data? Why do these students have no data? What is being done for these students?
The “Individual Student Report” for Grade 6 students also includes the students’ achievement results when they wrote the assessment in Grade 3. Some questions to ask: Which students moved to a higher or lower achievement level from Grade 3 to Grade 6? Who are the students who performed below the standard again? Did these students show improvement (e.g., from Level 1 to Level 2)? If students were exempted in Grade 3, were they able to participate in the assessment in Grade 6? This type of review will assist teachers in identifying students requiring additional attention.
A number of the reports contain suggestions to help you understand and interpret the data they present. Additional data interpretation guides are available on the EQAO Web site. The assessment frameworks and the provincial reports contain useful information to assist in the interpretation of your school results.
We wish you a fruitful journey as you explore your EQAO data.