Curriculum assessment
methods
Independent assessment of the effectiveness of each Macrosystems EDDIE
ecological forecasting module was administered by the Science Education
Resource Center (SERC) at Carleton College. Through SERC’s secure,
online portal, we delivered pre- and post-assessments of students who
completed one or more modules, and also collected instructor feedback
after module completion. Each module’s student assessment included
Likert scale ranking and multiple choice questions that were consistent
across all modules as well as multiple choice and short answer questions
that were specific to the individual module. Students who completed
multiple modules, regardless of which modules they completed, were given
the pre- and post-assessments for the Introduction to Ecological
Forecasting module to avoid survey fatigue. Instructor feedback surveys
included Likert scale ranking questions about module ease of use and
efficacy in meeting learning objectives, as well as multiple choice and
short answer questions about the delivery of the module, how likely the
instructor was to use high-frequency/long-term data and data from sensor
networks after teaching a module, and any other feedback the instructor
wished to provide.
Modules were assessed from January 2021 through May 2023 in 32 courses
across 22 institutions (Table S1), with a total of 596 students
completing one or more questions for both the pre- and post-assessment
and 26 instructors completing a feedback survey. We compared pre- and
post-module responses with paired Wilcoxon signed rank tests to evaluate
student growth. Due to widely varying numbers of students across
experience levels and courses (Table S1), we aggregated all students’
responses together for statistical analysis. All module assessment and
instructor survey questions, as well as details regarding the analysis
of assessment responses, can be found in the supplemental material (Text
S1, Tables S2-S7). All assessment was conducted following approved
Institutional Review Board (IRB) protocols (Virginia Tech IRB 19-669 and
Carleton College IRB 19-20 065). Anonymized, aggregated student
assessment and instructor feedback data and code to reproduce figures
and statistics are published in Lofton et al. (2024d).
Curriculum assessment
results