Carleton’s Moodle Evaluation & Needs Assessment

By Carly Born, Academic Technologist

moodle-eval-logoCarleton College has been using Moodle for nearly 10 years without having ever done a real evaluation of our use.  Recent data from our participation in the MISO survey suggest some dissatisfaction with Moodle on our campus.  And many current conversations (this Educause white paper is a great place to start) in ed-tech are centered around the changing role of the LMS or even whether an LMS is really needed anymore.  But how do we know? And how do we know what we need?

That is why we are taking a step back and trying to engage the campus on attempting to understand our use of Moodle on a deeper level. The Moodle Evaluation & Needs Assessment project is designed to answer this question:

Does Carleton College need to investigate new technologies to support the functions currently provided by Moodle? Before we can answer this question, we are carrying out an assessment to gain a  deeper understanding of the how we are using Moodle now, how we feel about it, and what it is that we actually need.

I’ve designed this evaluation project to try to assess Moodle from a number of different angles in order to get a more complete picture. There will be database diving, surveys and oh, so many focus groups!

Survey to assess the satisfaction with Moodle

Carly Born, Academic Technologist at Carleton College
Carly Born, Academic Technologist at Carleton College

A survey was distributed in March 2016 to all faculty at Carleton and also to staff who have logged into Moodle more than 5 times in the last year at Carleton.  A separate survey was also distributed to a representative sample of Carleton students.  I have some anecdotal information on general feelings towards Moodle, but it will be so much better to get some quantifiable feedback to specific features of the system.

Survey to assess the importance placed on features typically served by tools like Moodle

This is arguably more important than the questions about satisfaction, and is the heart of the needs assessment portion of this study. I’m hoping to get responses from everyone, even those who don’t use Moodle so that we can understand more about what it is that we need.

Database Diving

I am spending a lot of time up close and personal with the Moodle database. Moodle tracks a lot of information, and so I’m hoping to take advantage of that to understand our current usage patterns. Some questions I will be looking at include:

  • How many courses use Moodle per term?
  • What features of Moodle are used more?
  • Are there disciplinary trends in the usage patterns?
  • How often do students access a Moodle course page? What do they do there?

I’m sure there will be other questions that arise as I get further into this, but this is where I’m starting for now.

Focus Group Discussions

Inevitably, some trends will come out of the data gathered above that will generate even more questions. And I also want to give people an opportunity to talk in more depth about their opinions on these topics. I am especially looking forward to hearing from students at these focus groups, they are the largest audience that uses these tools and their opinions will be valuable in understanding what we need.

Comparative Data

Our friends at St. Olaf College are also participating in this project as part of our continuing efforts to collaborate through a Mellon-funded initiative called Broadening the Bridge. They will be delivering the same survey and possibly having similar focus group discussions on their own campus. This kind of comparative data can lend us even more information when attempting to answer questions for our campus, and I’m really looking forward to working with St. Olaf on the comparative analysis.

Then what?

Once all of the data are gathered and analyzed, it will be time to start to answer the original question: Do we need to investigate new technologies to support the functions currently provided by Moodle? To arrive at the answer to this question, I will host more discussion opportunities on Carleton’s campus in Fall 2016. Throughout that term, I will curate the conversation and hopefully come to an answer that is satisfactory to the campus. Once we know if we should be entertaining other tools, we can design a product evaluation process to make sure our identified needs are met with any possible new tools.

Contact me if you have questions, or would like to run a parallel process for comparative analysis on your own campus!

(This post also appears on Carly’s blog, The Space Between)