Eliminating the Guesswork in Assessment
By Tim Adams
Organizations use workplace assessments for many strategic reasons. Performance measurement, skills gap analysis, business improvement reviews and return on investment charts all help determine how an individual is doing within the organization and how closely the organization is aligned with its strategic goals. Unfortunately, according to the U.S. Department of Labor’s recently released “Human Resource Guide,” most of our human performance assessments fail to measure individual ability with any accuracy. The study points out that selection errors, or invisible errors that result when an individual guesses correctly on a true-or-false or multiple-choice test, are “both especially common and potentially damaging.”
How damaging? According to the “Human Resource Guide,” employee errors can have serious financial, health or safety consequences, and can also put organizations in legal jeopardy if assessment procedures fail to meet existing legal and professional standards.
Why One-Dimensional Assessments Don’t Work
Even organizations that recognize the ineffectiveness of standard assessment processes for measuring individual knowledge have had a difficult time identifying better solutions. Typically, they have improved the technical aspects of assessment processes, but they have not been able to overcome the “guesswork” factor involved in the testing outcome. In fact, this cannot be avoided in a single-dimensional assessment approach.
Guesswork comes into play with multiple-choice, true-or-false and yes-or-no answers that force individuals to make a decision about what they consider to be the correct answer. This approach naturally leads people to guess at answers even if they don’t really know the right one. We have all been in that spot (remember that high school mathematics test?), but the stakes are much higher now. In the workplace, an individual who has correctly answered a question and really does know that it is right is not distinguishable from the person who has guessed correctly and arrived at the right answer by sheer luck.
As a result, we have no way of avoiding situations where an assessment score wrongly predicts someone to be a good (and knowledgeable) worker when, in fact, he may be clueless or, worse, a critical risk to the enterprise. Equally problematic is a worker who selected answers with less luck than his colleague and received a lower score, yet may be a better (and more knowledgeable) performer. Clearly, more effective testing approaches need to be implemented if organizations are going to rely on their assessment outcomes.
I Guess I Know This…
One of the most important questions in training these days is how to ensure that employees really understand what they need to know to perform quickly, confidently and reliably. Current multiple-choice tests fail to measure the degree of confidence that individuals have in their knowledge or the amount of information they retain that can be applied in the performance of their duties. Research confirms that confidence is one of the most significant predictors of performance, yet traditional testing methods have no way of addressing or measuring confidence. This means that companies may find themselves in critical situations where outcomes depend on individuals who guessed their way through a test—or learned just enough to answer the questions, but not enough to retain the information.
In current right-or-wrong assessments, a wrong answer is interpreted simply to mean that the person is uninformed about the material and does not have the correct information. However, there is another outcome that can be equally damaging to both the individual and the organization as a whole.
I Know I’m Right—Even If I’m Wrong!
In addition to individuals who guess correctly (those fortunate few “lucky guessers” who knew they didn’t have a clue), there are also individuals who may be wrong about an answer but strongly believe that their wrong answer is correct. This high level of confidence in incorrect information is commonly referred to as “confidently held misinformation.” Such misinformation not only leads to poor—sometimes even dangerous—decisions and errors in performance, but can also become counterproductive to learning new material effectively.
Unfortunately, the distinction between being uninformed and being misinformed cannot be made in traditional one-dimensional assessments. Instead, this approach allows misinformation to remain hidden. When such individuals are assigned to a task, assessment results do not alert managers to the possibility that specific misinformation can, and very likely will, produce errors in a person’s performance. These errors can lead to costly mistakes and legal or financial liabilities.
Misinformation can spread like a virus in an organization when undetected and uncorrected. Incorrect information that is confidently believed with any authority at all inevitably and gradually finds its way into the “culture” of an organization, manifesting itself in the form of bad or faulty business practices that can negatively impact internal morale, productivity, customer satisfaction and numerous other critical business functions.
Confidence-Based Assessments
Recognizing some of the issues around potential errors in the assessment process, UCLA professor James Bruno developed an assessment methodology that remedies these problems by using a two-dimensional assessment model that:
- Identifies a person’s certainty of information as an essential element in defining that person’s knowledge.
- Employs a method of testing, scoring and interpreting the test results based on a model that underlines the confidence a person has in the information as well as identifying the correctness of their answer.
Bruno’s research noted that traditional multiple-choice testing techniques used to assess the extent of a person’s knowledge in a subject-matter area typically include a number of possible choices that are selectable by right or wrong answer. For example, a typical multiple-choice test will include questions with a number of possible answers, one of which can usually be eliminated by an individual upon first impression. This gives rise to a significant probability that a guess on the remaining answers could result in a correct response. In this situation, a successful guess would mask how much an individual actually knows—or doesn’t know—about a topic. As the evaluator, we would not be able to tell whether he or she was:
- Informed: Confident with a correct response.
- Misinformed: Confident in the response, yet not correct.
- Lacking information: Had no idea about the subject, but guessed correctly.
Bruno’s research resulted in the creation of the confidence-based assessment model (originally referred to as Information Reference Testing, or IRT), an assessment approach that accomplishes two critical goals. First, it extracts an individual’s response to a query. Second, and most critically, it identifies the confidence level associated with that response to generate a true knowledge profile. The true knowledge profile helps individuals identify areas of misinformation, helping them to better understand the quality of their information, as well as whether they overvalue or undervalue their understanding of the material.
How Does This Affect Learning and Application?
According to Darwin Hunt, New Mexico State University professor emeritus, research shows that the retention of newly learned material is systematically related to “how sure” people are of their answers when they learn them. Specifically, if people are “not sure at all” of a correct answer, then a week later they can only remember (and apply) 25 percent of the material. According to Hunt, if individuals are “extremely sure” of a correct answer, then they retain 91 percent of the information they have learned.
Developing a True Knowledge Profile
The confidence-based assessment model uses a two-dimensional technique to “extract” the knowledge and confidence levels of individuals, allowing them to state their confidence level in a substantive multiple-choice answer format. The confidence categories are predefined confidence levels: “100 percent confident or sure,” “50 percent confident or partially sure” or “I don’t know.” Individuals receive a maximum reward for a correct answer and maximum penalty for an incorrect answer, thus discouraging them from guessing when answering a question.
The questions consist of three answer choices in a two-dimensional answering pattern that includes the individual’s response and his or her confidence in that choice. As noted, the confidence categories are: “I am sure,” “I am partially sure,” and “I don’t know.” The uniqueness of the confidence-based assessment is that it utilizes these categories to generate two metrics simultaneously, both confidence and correctness. The confidence-based assessment focuses on identifying what an individual really knows and thus encourages the individual to answer each question honestly. To reinforce this, there is no penalty for an answer of “I don’t know.”
The confidence-based assessment is then scored and a knowledge profile generated and grouped according to distinct information quality regions. These regions (or knowledge quadrants) include:
- Misinformed: Answered 50 percent to 100 percent confident and incorrect, also known as “confidently held misinformation.”
- Uninformed: Answered “I don’t know.”
- Partially informed: Answered 50 percent confident and correct.
- Fully informed: Answered 100 percent confident and correct.
The assessment approach links the knowledge profile to specific learning materials, which are organized and prioritized based on the identified informational needs of the individual. The learning materials can be presented to the individual for review and re-education, thereby ensuring the individual’s acquisition of the true knowledge and accurate information in a cost-effective manner. The learning materials include a correct answer with detailed explanation as to why it is correct, as well as direct links to other sources of learning materials to reinforce understanding.
This assessment approach provides a measure of how well the material will be retained—the more confident the person is of correct answers, the better the material will be remembered.
According to Hunt, often the newly acquired knowledge is applied immediately as part of learning to perform a task or as a foundation for more advanced learning. The repeated practice-with-confirmation or randomization of material serves to maintain the knowledge over time.
Additionally, research shows that if learned knowledge is part of an infrequently or rarely performed task, such as what to do in a specific emergency situation, then the necessary—but unused—knowledge tends to begin to deteriorate. Incorporating information regarding “how sure” a person is regarding specific knowledge provides helpful supplementary data for deciding when remediation is necessary.
Covering Both Bases
Using confidence-based assessment methods allows employers to determine not only whether or not their employees are able to identify the best answer, but also how confident they are in their answers. Have they truly mastered the appropriate knowledge? Will they retain it? Will they be able to apply it? Test scoring based on confidence-based assessment evaluates not only the employees’ degrees of correctness, but also their level of confidence in gauging what they know and don’t know.
Equally important, however, is that this type of assessment identifies misinformation that might cause employees to make serious errors. Nothing is more dangerous than an employee who is highly confident and is prepared to act quickly and decisively on knowledge that is totally wrong. Because of its two-dimensional approach, this type of assessment allows employers to quickly identify—and address—such high-risk situations (and employees).
Psychologists report that most human decisions are driven by a desire for gain or a fear of loss. The confidence-based assessment methodology covers both bases by providing a platform where the workforce can learn quickly, effectively and reliably, while simultaneously enabling organizations to avoid the damages and liabilities of misinformed employees whose poor decisions compromise their—and their organizations’—effectiveness.
Tim Adams is CLO of Knowledge Factor, a Denver-based leader in competency assessment and remediation. Tim has been a pioneer in the distance education, training and development world for more than 30 years. He can be reached at tadams@wpsmag.com.
| 
Event
Calendar
