Katherine Kelley, Ph.D.
Katherine Kelley, Professor of Clinical Pharmacy and Associate Dean for Assessment and Strategic Initiatives in the College of Pharmacy has more than 20 years of experience in a variety of roles from student services to assessment and accreditation.
Kelley saw a connection between a College of Pharmacy curriculum redesign effort and Instructional Redesign, the third and final component of the Teaching Support Program. She chose to create and submit an Instructional Redesign portfolio to further explore this connection and to provide faculty engaged with colleges in curriculum work a model for translating that work to Instructional Redesign.
From 2016 to 2018, Kelley and her colleagues noted a decline in standardized test scores achieved by Doctor of Pharmacy students on a written assessment administered with no associated individual student stakes. Kelley described the college’s efforts to gather more data about the decline.
“In spring of 2018, student focus groups were conducted to determine student persepctives on the declining performance. The results of the focus groups were informative in terms of revealing student attitudes surrounding their approach to the exam," she said. "Students reported that they were uncertain about how to prepare for a comprehensive knowledge exam. Additionally, students adopted strategies such as skipping questions that required higher levels of mental effort.”
Three resources were consulted to help understand the decline:
- Small Teaching: Everyday Lessons from the Science of Learning by James Lang provided summaries of educational theories to help understand what might be underlying declining student performance on the Pharmacy Curriculum Outcomes Assessment (PCOA). This work served as a foundation to support the notion that students needed opportunities to practice taking comprehensive knowledge exams prior to a long (three hour) exam.
- Examine pharmacy literature on no stakes versus incentivized assessments. Given that the PCOA is used by specialized accreditation to determine program effectiveness, creating conditions under which students are incentivized to perform their best seems to be a sound strategy.
- The Big Ten Academic Alliance Pharmacy Assessment Collaborative (comprised of Big Ten pharmacy schools who have PharmD programs) worked collectively from 2016-2019 to compare processes of exam delivery, individual student stakes and outcomes.
Kelly's Instructional Redesign project involved a series of program-level assessment courses integrated into three didactic years of the curriculum to replace a capstone practical assessment course to give students opportunity for “interleaving.” Other aspects of the new curriculum included the addition of a laboratory course sequence and the instituion of a minimum PCOA for passing the course.
As a result of instituted instructional and curricular changes, overall PCOA scores improved significantly, in addition to the college's ranking within the Big 10.
“If students do not take the test seriously to perform their best, an inaccurate assessment of the effectiveness of a program may be made. In addition, the importance of addressing student motivation on no stakes exams is critical to being able to use the program-level data for evidence-based decision making," Kelley reflected in her Instructional Redesign porfolio. "Otherwise we run the risk of falling victim to changing curricula to fix problems that may or may not exist.
"Another realization from this process is that I have solidified my understanding of the importance of interleaving and refined how I think that can be operationalized in our PharmD program," she added. "Didactic instruction combined with practice in our lab sequence that is reinforced by assessment activities in the end-of-year Program-Level Assessment courses is a design that is supported by the literature.”
Read Kelley's portfolio to learn more about connecting curriculum work to Instructional Redesign.
Kelley and her colleagues also recently published resources about employment trends of PharmD graduates and identifying high impact assessment practices and managing low impact ones.