The need to transform undergraduate science, technology, engineering and mathematics education is not new, but the sense of urgency has accelerated in recent years as economic forecasts point to the need for greatly increased numbers of graduates in these fields.
For example, a 2012 report from the President’s Council of Advisors on Science and Technology projected that upwards of one million more college graduates in STEM field will be needed in the next decade.
In 2013, Michigan State University was awarded a grant from the Association of American Universities as part of its STEM Education Initiative—a coordinated institutional effort to transform MSU’s undergraduate STEM gateway courses in general chemistry, physics and biology. The MSU project is guided by “A Framework for K-12 Science Education,” an approach focused on scientific practices, cross-cutting concepts and core ideas, together called three-dimensional learning.
But how effective has this 3DL approach been in course transformation?
There are many ongoing efforts to transform STEM education, but few tools to measure change, particularly in curricular materials such as assessments. To address this issue, a team of MSU-led researchers are developing and using tools to measure 3DL instructional change and student learning.
In a study published in the current issue of Science Advances, researchers used an assessment tool—the Three-dimensional Learning Assessment Protocol, or 3D LAP,—to evaluate the extent of change over four years of chemistry, physics and biology course exams. Their goal was to demonstrate that the 3D-LAP can be used to measure change in assessments across multiple disciplines and across multiple instructors within a given discipline over time. In previous work, 3D-LAP was demonstrated to be sensitive enough to distinguish between a complete curricular transformation from an ongoing, more slowly progressing effort.
“This project provides a novel and meaningful way to assess the extent of change in course transformation projects,” said Melanie Cooper, Lappan-Phillips Professor in Science Education in the MSU Department of Chemistry and the project’s principal investigator. “The goal of this project was to use theories of learning and evidence to support student learning by emphasizing the use of knowledge rather than recall of facts and exercises. Changing this emphasis requires changing course assessments, so we monitored the extent of change by analyzing the nature of course assessments.”
To test their hypothesis, three types of data were collected from these courses—exams, video recordings of class meetings and course grades.
“This work is based on coding more than 4,000 questions from the high-stakes mid-term and final exams that students (and instructors) tend to highly value,” said Becky Matz, academic specialist at MSU’s Hub for Innovation in Learning and Technology and the study’s lead author. “The nature of those exams in the two general chemistry courses and the introductory cell and molecular biology course is fundamentally different now, compared to four years ago. Further, in most courses, we find that more 3DL questions on exams correlates significantly with a higher mean final course grade and a lower D-grade, F-grade, and withdrawal rate.”
The results also show that, although the transformation efforts in each discipline were guided by the 3DL approach, the measurable changes appear to be highly dependent on other factors including course organizational structure, perceived ownership of the course, departmental culture, available resources, faculty expertise, and the power dynamics between faculty and those calling for change.
Moving forward, the team recommends that a combination of tools, including the 3D-LAP, be used to further evaluate and understand STEM transformation efforts, particularly those in science disciplines.
In addition to Cooper and Matz, other MSU contributors are Marcos Caballero, Diane Ebert-May, Cori Fata-Hartley, Lynmarie Posey and Ryan Stowe. Additional researchers include James Laverty, Kansas State University; Justin Carmel and Sonia Underwood, Florida International University; and Deborah Herrington, Grand Valley State University.