Assessment Validation: Beyond Checklists and SMEs

Let’s explore some familiar scenarios, shall we?

– A standard checklist ensures all topics are thoroughly covered. But does it guarantee that students have a deep understanding of the material? Probably NO.

– An SME validates the accuracy of content, but does that ensure relevancy?

– A test looks great on paper. But, can it predict real-world success?

Sure, you can think of plenty more scenarios. In short, the standard assessment checklists are falling short. For example, if creativity is one aspect in the checklist, each person’s perspective on creativity differs. One might perceive creativity as uniqueness and originality, while the other might consider how visually appealing it is. In turn, this leads to a difference in scoring.

Assessment Validation is not as effective as we think!

• Rigid checklists: Rigid checklists fail to capture the nuanced understanding that students might possess. They often focus on surface-level skills and neglect deeper cognitive abilities and critical thinking. For example, generally, the checklist for a high school-level experiment has the following criteria: hypothesis, material, procedure, results, and conclusion. The depth of analysis and quality of the research are not taken into consideration, which leads to underappreciation of the student’s skills.
• Expert bias: SMEs have preconceived notions about a specific topic being more important than others, which limits the scope of test design. For example, in medicine, SMEs focus on critical topics like pharmacology but not palliative care, leading to failure in real-world performance.
• Technological challenges: Advanced technologies like simulation-based tests make it hard to score. Only <20% of evaluators are able to justify the scores provided. It is important to justify scores to ensure transparency and fairness. So, automating these simulation-based tests is essential.
• Accessibility issues: About 7.5 million students in the U.S. face accessibility issues, as standardized assessments do not always accommodate their diverse needs and learning styles. This hinders an equitable education system that ensures all students reach their full potential. Implementing questions based on students’ learning styles will help avoid these issues.

• Around 43.2% and 37.8% of students were not in support of computer-based tests for Physics and Chemistry subjects, respectively. This was due to the limitation of the interface design.
• It led to developing designs that support complex problem-solving and stepwise calculations for a thorough assessment.
• This percentage was greater than other subjects with simpler answers and multiple-choice questions.
• About 89.3% of students emphasized the need for training before taking these assessments, to manage time effectively.

Keeping these limitations in mind, let’s explore ways to ensure fairness, inclusivity, and real-world performance in our assessment validation process.

Validation—A continuous Cycle of Testing, Reviewing, and Improving

1. Define clear, measurable learning objectives and align them with the content under review. These objectives should be continuously checked to ensure relevance, fairness, and accuracy.
2. Allow cross-functional collaboration to create valid and reliable content to align with the needs of the students and educational institutions. Avoiding collaboration can lead to bias.

3. Align assessments with evolving learning needs to adapt to measure students’ competencies better than just focusing on basic checklists.

a. Accessibility: Students from various cultural backgrounds have different learning styles. As studied by Gunduz and Ozcan,

• Cypriot students – reflective learning style; Arabic and Turkish – active style
• Turkish students – intuitive, sequential; Cypriot and Arabic – sensing, global

Providing different question-answer formats caters to diverse learners. Additionally, audio and text-to-speech formats aid students with visual or motor impairments. Following one format will lead to unfair scoring and improper assessment of their skills.

A tiered approach is an effective way to help students overcome their learning difficulties. For example, in the GRE, the difficulty level is adjusted based on the previous answers, which challenges the students accordingly.

• Use of simulations, online quizzes, and gamified assessments allow students to perform tasks that will better reflect real-world challenges.
• They provide immediate feedback and ensure accurate measurement of the students’ abilities.

c. Shift from summative to formative assessments:

• Assessments like quizzes and polls provide ongoing feedback.
• They help students track their progress and work on areas where they are weak.
• They help in achieving better outcomes than summative assessments.

d. Systems Thinking:
• Assessments need to move beyond linear, cause-and-effect questions toward those that evaluate a student’s ability to analyze complex, interdependent systems.
• For example, a student might be asked to assess the potential environmental, social, and economic impacts of a policy decision or business strategy, requiring them to identify feedback loops, potential unintended consequences, and leverage points.
e. Critical thinking:
• Assessments must be designed to not only test a student’s ability to recall facts but also their capacity to engage in higher-order thinking.
• Critical analysis involves evaluating evidence, considering multiple perspectives, identifying biases, and making informed, reasoned conclusions.
• Assessments aligned with this need should include open-ended questions, essays, and case studies.
• This helps the students to evaluate complex issues, justify their decisions with evidence, and articulate well-supported arguments.

Despite having several approaches and strategies, ensuring fairness is crucial.

How can Fairness be ensured?

Yes, avoiding bias is one way. But can you think of other ways? Here are a few:

Inclusive Review Process: Identify potential cultural assumptions and biases by allowing subject matter experts from diverse groups to review the assessment.

Differential Item Functioning Analysis: Analyze test items to assess differences in performance of various learners based on race, gender, and disability to name a few. If significant differences are found, the questions will have to be revised or deleted.

Clear instructions: Ensure questions and instructions are clear and written in simple language to avoid misunderstandings.

Accessibility Considerations: Provide accessible assessments to accommodate students with disabilities.

To maintain fairness in assessments, we must identify and mitigate biases within assessment items.

A prime example involves comparing the scores of two students: one with a disability who scored 40% yet demonstrated extraordinary problem-solving skills and empathy, and another academically strong student with a 90% who struggles with effective patient care and ethical judgment. This instance illustrates the importance of assessing skills beyond mere numerical scores.

How to Optimize Assessments?

1. Cognitive walkthrough: Through cognitive walkthrough, we can ensure that assessments are intuitive and not overly stressful. Assessments should undergo this testing, which involves a cross-functional team of reviewers, to uncover the potential causes of failure of these assessments.

2. Positive user experience: Conduct usability testing to address issues that cause eye strain, confusing interfaces, and excessive time pressures. For example, by using legible fonts, contrast between text and background.

Assessment Validation – EduQual’s focused approach

Moving beyond rigid checklists and SME biases to adopt a more comprehensive approach to assessment validity can lead to fairer and more relevant evaluations. At EduQual, we understand that assessments are not just about ensuring accuracy. It is about creating fair, meaningful, and real-world applications, providing an accurate reflection of learners’ capabilities.

To review your assessments comprehensively, contact us at .