Curriculum assessment methods

Independent assessment of the effectiveness of each Macrosystems EDDIE ecological forecasting module was administered by the Science Education Resource Center (SERC) at Carleton College. Through SERC’s secure, online portal, we delivered pre- and post-assessments of students who completed one or more modules, and also collected instructor feedback after module completion. Each module’s student assessment included Likert scale ranking and multiple choice questions that were consistent across all modules as well as multiple choice and short answer questions that were specific to the individual module. Students who completed multiple modules, regardless of which modules they completed, were given the pre- and post-assessments for the Introduction to Ecological Forecasting module to avoid survey fatigue. Instructor feedback surveys included Likert scale ranking questions about module ease of use and efficacy in meeting learning objectives, as well as multiple choice and short answer questions about the delivery of the module, how likely the instructor was to use high-frequency/long-term data and data from sensor networks after teaching a module, and any other feedback the instructor wished to provide.
Modules were assessed from January 2021 through May 2023 in 32 courses across 22 institutions (Table S1), with a total of 596 students completing one or more questions for both the pre- and post-assessment and 26 instructors completing a feedback survey. We compared pre- and post-module responses with paired Wilcoxon signed rank tests to evaluate student growth. Due to widely varying numbers of students across experience levels and courses (Table S1), we aggregated all students’ responses together for statistical analysis. All module assessment and instructor survey questions, as well as details regarding the analysis of assessment responses, can be found in the supplemental material (Text S1, Tables S2-S7). All assessment was conducted following approved Institutional Review Board (IRB) protocols (Virginia Tech IRB 19-669 and Carleton College IRB 19-20 065). Anonymized, aggregated student assessment and instructor feedback data and code to reproduce figures and statistics are published in Lofton et al. (2024d).

Curriculum assessment results