Scoring Activities

The following terms are most commonly used:

• Scoring Window: a specific period (with predefined start and end dates) during which the e-assessment scoring system is available for training and scoring.
• Item: a question or task that appears on an assessment or test.
• Response: a student’s response to an item on an assessment or test.
• Pooling: the process of adding student responses available to score into the e-assessment scoring system. This generally occurs on weekday evenings throughout the administration window.
• Training Module: material for the training session, which includes an introductory video, a scoring guide, training responses, a practice test and a qualifying test.
  • Practice Test: a test that scorers complete as part of the training module, used to prepare scorers for the qualifying test.
  • Qualifying Test: a test used to assess a scorer’s ability to score student responses accurately.
• Batch: a predefined number of student responses to a test question. (Each batch is expected to take an average of one hour to score.)
• Partial Batch: a batch that is partially scored, or a batch with a partial number of responses to be scored, which was automatically assigned by the system.
• Rolling Batch: a set number of scored batches used to calculate the scorer’s average validity on those batches. (This average is used to compare against the minimum validity threshold.)
• Expert Scorer: an experienced scorer with high validity.
• Validity: the measure of a scorer’s accuracy assessed by examining the agreement between the scores assigned by the scorer and those assigned by expert scorers. The following indices are computed: percentage of exact agreement, percentage of exact-plus-adjacent agreement, percentage of adjacent-low agreement and percentage of adjacent-high agreement.
• Adjacent: a difference of one score code between the score assigned to a certain response by the scorer and that assigned by the expert scorers.
• Adjacent-low: the score assigned to a certain response by a scorer is one score code below the score assigned by the expert scorers.
• Adjacent-high: the score assigned to a certain response by a scorer is one score code above the score given by the expert scorers.
• Non-adjacent: the difference between the score assigned to a certain response by a scorer and that assigned by the expert scorers is greater than one score code.
• Exact Agreement: the score assigned to a certain response by the scorer and the expert scorers is the same.
• Cumulative Exact Validity Percentage: the total percentage of validity responses scored that match the score preassigned.
• Cumulative Exact + Adjacent Validity Percentage: the percentage of validity responses scored that are either a match with the score given by the expert scorers or are one score code different (higher or lower).
• Validity Response: a response that matches the anchors in the corresponding scoring guide.
• Average Rolling Validity: the average validity achieved on a rolling batch. Average rolling validity is calculated and made available only when the pre-determined number of validity items are scored. (This average is used for comparison against the minimum validity threshold, which is required for scorers to be able to score).
• Minimum Validity Threshold: the minimum average rolling validity required to continue to score without retraining.
• Cumulative Validity Incentive Threshold: the minimum cumulative exact and cumulative exact + adjacent validity percentage required to receive incentive pay.

If you are interested in scoring the assessments, please create or update your EQAO profile.

Training and scoring will be self-directed through the e-assessment scoring system, which will be available 24 hours a day, seven days a week, except statutory holidays, during the scoring window. This means that if scorers meet the technical requirements and qualifications, they can score on their own time and from any location. All the resources required for training and scoring will be available in the e-assessment scoring system.

Once scorers have completed their training and have passed the qualifying test on the e-assessment scoring system, they will be able to select batches, which they will have 24 hours to complete. A batch takes one hour on average to score.

Scorers are not to select any batches unless they are able to score them in the 24-hour period. If a scorer completes the selected batches before the 24-hour period expires, the scorer can select one or more additional batches, for which a new 24 hours is allotted for completion.

Online scoring is self-directed. Scorers do not need to score daily as long as they score a minimum of 10 batches per week during the entire scoring window and meet the minimum validity threshold. These requirements are outlined in the Confidentiality, Non-Disclosure and Employment Agreement for an EQAO Scorer (Online) or Committee Member (Online).

If a scorer falls below the minimum validity threshold, the scorer will have the opportunity to retrain and pass another qualifying test. If successful, the scorer will be able to continue scoring. Otherwise, the scorer will be withdrawn from their current scoring assignment.

Scorers can score as many batches as are available but should not exceed 48 hours in a work week (Sunday to Saturday).

Scoring generally takes place during the latter stages of the administration windows of the Ontario Secondary School Literacy Test and the primary- and junior-division Assessments of Reading, Writing and Mathematics. Scoring invitations are generally sent to qualified candidates close to the start of the administration window.

You will receive an e-mail from [email protected] on the first day of the scoring window. The e-mail will contain a link that will allow you to register for an account on the e-assessment scoring system. It will also include important dates by which you must register for your account, complete your training and start scoring. Please refer to your EQAO confirmation letter for more information.

Your name in your EQAO profile must match the name in your profile on the OCT website. If you change your name, you must ensure that you update your name with the OCT. Once the name change is reflected on the OCT website, the name will automatically be updated in your EQAO profile. As EQAO’s database is linked with OCT’s database, a different name will create communication errors between the two sites. If you have a middle name, you can try adding your middle name to the first-name field or the last-name field one at a time, which may rectify the error. Otherwise, please ensure that your name is the same in both profiles.

If you cannot remember your password, click Forgot Password, and an e-mail will be sent to you with instructions on how to reset your password. If you no longer have access to the e-mail account on file with EQAO, please contact us at [email protected] or 1-888-327-7377.

Some scorers, upon making changes to their EQAO profile, may find that a scoring opportunity is no longer available to them. This is a normal occurrence, as the automated system verifies that the details, qualifications and current employment status in an applicant’s EQAO profile match the requirements for the scoring position. It is therefore important that each scorer’s profile be maintained with accurate information regarding address, qualifications, current role in education, experience, etc.

There are a limited number of scoring positions. As applicants accept offers to score, the scoring positions are filled quickly on a first-come, first-served basis. Sometimes further scoring positions are added or scoring positions can become available if scorers withdraw. Applicants are encouraged to check their EQAO profiles regularly, as scoring positions can become available at any time.

Qualified candidates will receive e-mails with offers to score. As e-mails may be stopped by firewalls or sent automatically to junk or spam folders, it is recommended that applicants check their EQAO profiles regularly for scoring offers. Applicants can add the e-mail addresses below to their “safe senders” list to avoid having the e-mail scoring invitations end up in a spam or junk folder. Applicants are also encouraged to check these folders periodically. EQAO e-mail addresses to add to your “safe senders” list:

• [email protected]
• EQAO/[email protected]
• EQAO/[email protected]

Please note that EQAO is unable to accept job applications or employment inquiries by e-mail or by telephone.

This means that a scoring opportunity was made available, but the position was filled as multiple applicants received an invitation for the same position. Applicants are encouraged to check their EQAO profiles regularly, as additional scoring positions can become available at any time.

Log in to your EQAO profile. All of the scoring opportunities that you have accepted are listed under “Status” on your profile dashboard page. For more information about each scoring opportunity, click Details beside each opportunity.

This means you have qualified for more than one scoring position. You may accept as many of the scoring opportunities as you would like, provided the dates are not conflicting. If the dates conflict, please select and accept only one offer. To accept an offer to score, click Accept/Decline Position beside the scoring position on your profile dashboard page; then click Accept and continue the process until you click Confirm.

Unfortunately, the task you have been offered cannot be changed.

You must withdraw from your accepted scoring position immediately. Scoring with EQAO is considered employment, and EQAO relies on every scorer in order to provide timely results to schools, parents and the public. If you have accepted an offer to score for EQAO, you are expected to participate in the scoring activity. A scorer who accepts a scoring position and does not withdraw from it or does not participate will be considered a “no show” and may not be invited to participate in future EQAO events.

Do not accept an invitation to score until you can commit to the scoring opportunity.

Log in to your EQAO profile, and locate the offer you wish to decline on your profile dashboard page. Next, click Details, and then click Withdraw. If you are not able to withdraw through your EQAO profile, please contact EQAO at [email protected] or 1-888-327-7377.

You need to update your proof of enrolment in each year of your program. Once you receive your Ontario College of Teachers (OCT) certificate and your information is on the OCT website, you will need to update your current educator role and enter the OCT number in your EQAO profile. Ensure that the first name and last name in your EQAO profile matches those on the OCT website.

Your EQAO profile may not be up to date, or your qualifications and experience may not have met the requirements of the scoring position—or, simply, there may not be any more scoring opportunities available at the moment.

The system updates in real time. This means that while schools are still administering the test, student responses are being captured and uploaded into the system. If you are not able to claim batches for your current item(s), please press the Refresh button at the top of the item card to see if more responses have been made available. Batches can become available at any time during the administration window as students complete their test. You may check at your own convenience for additional available batches. Once all responses have been scored, EQAO will notify you that all the responses have been scored.

A partially scored batch or a fully scored batch with a partial number of responses to be scored is pro-rated according to the percentage completed. For example, if you score a batch that contains only half the number of responses that are in a full batch, you will see 0.5 on your report.

Please refer to the Payment Guidelines for details on remuneration.

You must go to drakehire.com to complete or verify your payroll information. For more information, please contact Drake at [email protected].