If you’re considering a Physical Ability Testing program for your organization, choosing an evidence-based, research-validated and objective protocol is essential. But even the most rigorous screening program can let you down if it’s objective in theory but subjective in practice. What’s at stake here is something called "rater error." It’s often unconscious, seriously dangerous, and all too common.

Simply put, rater errors are the result of an evaluator considering subjective factors that cause them to rate applicants incorrectly. When rater errors are considered in conjunction with pre-hire Physical Abilities Testing, the result could be rating an applicant’s ability to perform the test either too high, too low, or too neutral to provide an accurate rating. When this occurs, the potential for serious injury, or even death, significantly increases.

There are several types of errors that can transpire. By exploring each type and addressing the root causes of each, we can uncover the best ways to avoid triggering rater errors that can come with such high consequences.

• Leniency Error: Leniency errors occur when the evaluator has a tendency to rate an applicant’s ability to perform tasks too high. When leniency errors occur, it creates an employer expectation that is more than what can be performed by the employee, amplifying the hire’s risk of injury. This error can stem from various sources:
  • The raters simply like the individual, thus providing more favorable results.
  • The raters don’t like giving low scores because they don’t feel justified doing so.
  • The raters don’t have a solid understanding of the areas/factors they are rating.
• Strictness Error: As opposed to the leniency error, strictness errors occur when the evaluator has a tendency to rate an applicant’s ability to perform tasks too low, which can impact the applicant’s ability to be employed. Persistent strictness errors can result in questions of discrimination and occur when:
  • The raters have higher personal standards to which they are comparing the candidates.
  • The raters dislike the candidate for a reason outside of the interview.
  • The raters lack a clear understanding of the areas/factors they are rating.
• Central Tendency Error: Central tendency errors occur when the rater has a tendency to score the majority of candidates as "average" across the board. This error causes evaluators to fail to correctly identify candidates who are legitimately qualified, or under-qualified, for the position. This error can occur when:
  • The raters lack an understanding of the areas/factors they’re rating.
  • The raters are not able to justify what they would consider a more extreme rating.

Each of these three rating errors – leniency, strictness, and central tendency – represent some form of bias and taint the true, objective performance of the candidate. Through thorough test development research, ErgoScience has determined that implementing a consistent standardized scoring system for rating performance on Physical Abilities Testing will decrease the frequency of these errors occurring.

Other errors that can occur during Physical Abilities Tests are based not on intentional bias, but rather on the tendency for subjectivity. These include:

• Similar-to-Me Error: The similar-to-me error is triggered by evaluators’ tendencies to rate applicants that they perceive as similar to themselves more highly. This is caused by the natural psychological human tendency to prefer those who are in our 'in-groups,' and to fear and/or dislike for those in our 'out-groups.'
• Contrast Effect Error: The tendency of humans to compare themselves, and each other, to other individuals can result in the contrast effect error. Most often, this error occurs either when the current candidate was preceded by someone who performed very poorly on the PAT, thus the current candidate is scored artificially high simply by subconscious comparison to the previous extremely impaired individual. Or, alternately, the current candidate was preceded by an incredibly strong applicant who performed very well on the PAT, causing the current candidate to seem much worse in comparison.

Evaluators must rate candidates only against the employer’s standards and requirements for the position, not against each other. The similar-to-me error and the contrast effect error can be remedied by ensuring that evaluators are focused only on the candidate in front of them, and that they are only using the objective standardized PAT scoring system. Although the "clinical judgement" component can never be fully taken out of PAT, if evaluators allow his or her brain to function strictly "intuitively" without a scoring system, these errors create a great deal of subjectivity.

One final error that can occur is the halo error.

• Halo Error: This error stems from the human tendency to make illogical generalizations. For example, if one of the candidates being tested has terrific lifting ability, that singular aspect of the candidate’s test performance will naturally stand out. The halo error occurs when the ratings of the performances on the following tasks of the test are influenced, either higher or lower, by the outstanding performance on one of the previous tasks. In this example, the halo error could occur if the candidate was rated high on balance, despite marginal or poor performance, because of their impressive lifting.

To avoid the halo error, the evaluator must be aware of the fact that there may be little or no connection between performances on individual test items. Just because an individual scores highly on one task does not mean they will necessarily score highly on all of the other test items. Each test item should be scored independently, using the specific scoring system developed for it, not allowing performance on other items to influence the score.

In order to protect employees and prevent potentially serious injuries and even death, it is essential that employees are appropriately and accurately placed in positions that suit their individual abilities and strengths. When it comes to Physical Abilities Testing, it is crucial that evaluators use a standardized, objective, reliable and valid testing protocol and are effectively trained to avoid allowing these six errors to impact the quality of your PAT results.