Scoring Activities

• EQAO selects potential scorers to score the open-response questions completed by students. Potential scorers must
  • create a profile and keep it up to date, and
  • accept a scoring-specific invitation (if you receive one) by the specified deadline to express your interest in participating in that activity.

Note that EQAO is unable to accept inquiries related to scoring activities by e-mail or by telephone. EQAO does not directly offer an employment position; individuals are hired by a third-party employment vendor of record (Quantum) and then assigned back to EQAO.

Training and scoring are self-directed through the e-assessment scoring system, which is available 24 hours a day, seven days a week (except on statutory holidays), during the scoring window. Scorers who meet the technical requirements and qualifications can score on their own time and from any location. All the training and scoring resources are available in the e-assessment scoring system.

Once scorers have completed their training and have passed the qualifying test (available in the e-assessment scoring system), they will be able to select batches, which they will have a set time frame to complete. A batch takes approximately one hour to score.

Scorers are not to select any batches unless they are able to score them in the set time frame. If a scorer completes the selected batches before the time frame ends, the scorer can select one or more additional batches, for which a new time frame for completion will be set. On the last day of scoring, EQAO may shorten the time frame to complete batches to two hours or less, and the number of batches that can be claimed at a time to one (to enable the distribution and availability of batches for everyone). EQAO can also revoke claimed batches at any time to facilitate the completion of scoring.

Online scoring is self-directed. Although scorers do not need to score daily, they must score a minimum of 14 batches per week, if available, during the entire scoring window and meet the minimum validity threshold.

If a scorer falls below the minimum validity threshold, the scorer will have the opportunity to retrain and pass another qualifying test. If successful, the scorer will be able to continue scoring. If unsuccessful, the scorer may be withdrawn from their current scoring assignment. Scorers can score as many batches as are available but must not score for more than 48 hours in a work week (from Sunday to Saturday).

Scoring generally takes place two to three weeks before the end of the administration windows of the Ontario Secondary School Literacy Test and the Assessments of Reading, Writing and Mathematics, Primary and Junior Divisions. Scoring invitations are generally sent to qualified candidates six weeks before the start of the administration window.

• You will be required to register with Quantum, the employment vendor of record, and sign an employment agreement.
• After the employment agreement is signed:
  • You will receive an e-mail from [email protected] on the first day of your assigned scoring task.
  • The e-mail will contain a link allowing you to register for an account in the e-assessment scoring system.
  • The e-mail will also include important dates by which you must register for your account, complete your training and start scoring.
• Please refer to your EQAO confirmation letter for more information.

The name in your EQAO profile must match the name in your profile on the OCT website. If you change your name, you must ensure that you update your name with the OCT. Once the name change is reflected on the OCT website, the name will automatically be updated in your EQAO profile. As EQAO’s database is linked with the OCT’s database, a different name will create communication errors between the two sites. If you have a middle name, you can try adding your middle name to the first-name field or the last-name field one at a time, which may rectify the error. Otherwise, please ensure that your name is the same in both profiles.

If you cannot remember your password, click Forgot Password, and an e-mail will be sent to you with instructions on how to reset your password. If you no longer have access to the e-mail account on file with EQAO, please contact us at [email protected] or 1-888-327-7377.

Some scorers, upon making changes to their EQAO profile, may find that a scoring opportunity is no longer available to them. This is a normal occurrence, as the automated system verifies that the details, qualifications and current employment status in an applicant’s EQAO profile match the requirements for the scoring activity. It is therefore important that each scorer’s profile be maintained with accurate information regarding address, qualifications, current role in education, experience, etc.

Openings for scoring activities are limited, and fill up quickly on a first-come, first-served basis, as applicants accept their invitations to score. Openings become available if scorers withdraw. Applicants are encouraged to check their EQAO profiles regularly, as openings can become available at any time.

Qualified candidates will receive e-mails with offers to score. As e-mails may be stopped by firewalls or sent automatically to junk or spam folders, it is recommended that applicants check their EQAO profiles regularly for scoring offers. Applicants can add the e-mail addresses below to their “safe senders” list to avoid having the e-mail scoring invitations end up in a spam or junk folder. Applicants can also add the following e-mail addresses to their “safe senders” list to ensure they receive EQAO e-mails:

• [email protected]
• EQAO/[email protected]
• EQAO/[email protected]

Please note that EQAO is unable to accept inquiries related to scoring activities by e-mail or by telephone.

This message means that a scoring opportunity was made available but has already been filled, as multiple applicants received an invitation for the same opening. Applicants are encouraged to check their EQAO profiles regularly, as additional opportunities can become available at any time.

When you log into your EQAO profile, all the scoring opportunities that you have accepted are listed under “Status” on your profile dashboard page. For more information about each scoring opportunity, click Details beside each opportunity.

You are seeing more than one opportunity because you have qualified for more than one opening. You may accept as many scoring opportunities as you like, provided the scoring dates are not conflicting. If that is the case, please select and accept only one offer.

To accept an opening, select Accept/Decline beside the opening and continue to follow the instructions until you click Confirm.

Unfortunately, the task you have been offered cannot be changed. However, you may be offered a different task to score based on your validity measure during the scoring activity.

You must immediately withdraw from your accepted invitation to score. To do so, please follow the instructions below. Note that if you do not withdraw, you will be considered a “no show” and may not be invited to participate in future EQAO events.

Log into your EQAO profile and locate the offer you wish to decline or withdraw from on your profile dashboard page. Next, select Details and then click Withdraw. If you are not able to withdraw through your EQAO profile, please contact EQAO at [email protected] or 1-888-327-7377.

Do not accept an invitation to score until you can commit to the scoring activity.

You must update your proof of enrolment in each year of your program. Once you receive your Ontario College of Teachers (OCT) certificate and your information is on the OCT website, you will need to update your current educator role and enter the OCT number in your EQAO profile. Ensure that the first name and last name in your EQAO profile match those on the OCT website.

Your EQAO profile may not be up to date or your qualifications and experience may not have met the requirements to perform scoring activities. Additionally, there may not be any scoring opportunities available at the moment.

Please refer to the Payment Guidelines for details on remuneration.

No. EQAO is not directly offering an employment position by offering involvement in EQAO activities. Individuals involved in EQAO activities will be hired by a third-party employment vendor of record and then assigned to perform EQAO activities including scoring activities, committee-related activities or any other activities designated by EQAO.

EQAO will not cover any additional expenses, including supply teacher cost, Internet charges, equipment rentals, or other related expenses. Employees who are currently employed by another employer (e.g., school, school board, etc.) shall not perform duties assigned by EQAO during their regular work hours. EQAO will not reimburse the expenses incurred by an employer for “supply workers” to cover for absent or missing employees.

The following terms are most commonly used:

• Scoring Window: a specific period (with predefined start and end dates) during which the e-assessment scoring system is available for training and scoring
• Item: a question or task that appears on an assessment or test
• Response: a student’s response to an item on an assessment or test
• Training Module: material for the training session, which includes an introductory video, a scoring guide, training responses, a practice test and a qualifying test
  • Practice Test: a test that scorers complete as part of the training module, used to prepare scorers for the qualifying test
  • Qualifying Test: a test used to assess a scorer’s ability to accurately score student responses
• Batch: a predefined number of student responses to a test question (each batch is expected to take an average of one hour to score.)
  • Partial Batch: a batch that is partially scored, or a batch with a partial number of responses to be scored, which was automatically assigned by the system
• Expert Scorer: an experienced scorer with high validity
• Validity: the measure of a scorer’s accuracy assessed by examining the agreement between the scores assigned by the scorer and those assigned by expert scorers. The following indices are computed: percentage of exact agreement, percentage of exact-plus-adjacent agreement, percentage of adjacent-low agreement and percentage of adjacent-high agreement.
• Reliability: a measure of a scorer’s consistency, determined by evaluating the agreement between scores assigned by two independent scorers to each student response
• Backreading: the process of scoring leaders reviewing previously scored student responses to verify alignment with established scoring guides and rubrics and ensure consistency
• Adjacent: a difference of one score code between the score assigned to a certain response by the scorer and that assigned by the expert scorers
  • Adjacent-Low: the score assigned to a certain response by a scorer is one score code below the score assigned by the expert scorers.
  • Adjacent-High: the score assigned to a certain response by a scorer is one score code above the score given by the expert scorers.
• Non-Adjacent: the difference between the score assigned to a certain response by a scorer and that assigned by the expert scorers is greater than one score code
• Exact Agreement: the score assigned to a certain response by the scorer and the expert scorers is the same
  • Cumulative Exact Validity Percentage: the total percentage of validity responses scored that match the score preassigned
  • Cumulative Exact + Adjacent Validity Percentage: the percentage of validity responses scored that are either a match with the score given by the expert scorers or are one score code different (higher or lower)
• Validity Response: a response that matches the anchors in the corresponding scoring guide
• Minimum Validity Threshold: the minimum average rolling validity required to continue to score without retraining
• Cumulative Validity Incentive Threshold: the minimum cumulative exact and cumulative exact + adjacent validity percentage required to receive incentive pay