Wednesday, January 6, 2016

Social engagement in Dino 101

Happy new year folks! There are no pictures in this post, SORRY NOT SORRY.

A group of researchers at the University of Alberta recently published a study on learner engagement in Dino 101, and I thought I'd summarize it briefly here and share a few thoughts about it. You can read the original article online for free via Google Books: "Emotional and social engagement in a massive open online course: an examination of Dino 101". You might also want to check out another summary of their data at the University of Alberta's site.

Daniels et al. lay out four components for describing the otherwise somewhat nebulous term 'engagement':
  • Cognitive engagement (investment into thinking about tasks and mastering content; could also be thought of as motivation for learning)
  • Behavioural engagement (things like attendance, paying attention, and participation)
  • Emotional engagement (things like anxiety, boredom, interest, etc.)
  • Social engagement (willingness to socialize with others, to make connections)

They mainly investigated emotional and social engagement, since cognitive and behavioural engagement are largely measured by things like course completion rates and grade averages. Daniels et al. sent a survey to people enrolled in Dino101 towards the end of its initial offering, as well as the students enrolled for credit in Paleo200/201 at the UofA, and also interviewed a total of 30 students representing all three versions of the course. Because all of the lessons and study material for Dino101 was released at the beginning of the semester (in contrast to many MOOCs which release new lessons on a weekly basis, probably in line with the in-person lectures at the university), I wonder if conducting the survey at the end of the semester, in December, influenced the results. We heard from many students that they completed Dino101 very very quickly – some in as little as a day or two, and many within 2-6 weeks. From managing the forums, I can also attest that discussion board participation dropped pretty dramatically after about six weeks. Therefore, students that completed the course are not necessarily students that stuck around until December, and I wonder how well this slice of the long statistical tail might represent the majority of the people who engaged with Dino101.

Overall, the results showed that we did a pretty good job of emotionally engaging Dino101 students: many said they were inspired by the material, didn't get bored, and felt attentive during the lectures. We fared less well in terms of social engagement: many students were neutral about whether or not they had a sense of belonging in Dino101, but only 16% had expected student-student interactions to contribute to their own learning. (In other words, most students expected to learn primarily from the videos and course notes, and not from discussions with other students.) I'm not sure if this is typical for a MOOC or unique to Dino 101, so I'd be curious to see if there are similar studies for other courses that show similar results.

In the data from the student interviews, Daniels et al. highlighted both positive and negative aspects to the discussion forums – some students were really interested in the forums even if they didn't participate, some didn't look at them at all, and some found them overwhelming because of too many email notices. I think there are a couple of takeaways just from this one portion of their paper:
  •  If you're doing a MOOC, make sure the button for the discussion forums is prominently displayed towards the top of the course page near the lecture videos. I think ours was located kind of far down the menu and some students might just overlook it.
  • Give students some guidelines for using the forums, both technically (i.e. how to turn off email notifications for new posts, because that would have driven me COMPLETELY MAD if I hadn't known how to turn it off for most posts), and by giving suggestions for how to participate in the forums. Do this right at the beginning of the course.

Another thing that popped up in this paper is that while social engagement online wasn't as high as it could be, social engagement in meatspace was unexpectedly common. There were lots of people taking the course with their kids or other family members at home, and they would sit and do the course together. So, here is another takeaway suggestion: find ways to increase social engagement by giving people things to do at home together, in the physical world. Maybe we could provide suggestions for local museum trips or fossil sites based on where people in the course are based, or give some suggestions for hands-on activities people can do with stuff you find in your house.

Finally, there were mixed feelings about engagement with the instructors of the course (here limited to Phil Currie and Betsy Kruk, who presented the material, but excluding myself and the other teaching assistants who did most of the online interactions in the forums). Some students felt like they had a good connection with Phil and Betsy despite the scripted presentations, and others found they didn't like the scripted format at all. Strangely, the paper doesn't include much discussion about engagement with the instructors (and here I'm including the TAs) in the discussion forums, the only part of the course where that was really possible given the pre-recorded nature of the videos.

My final comment here is that if engagement with an instructor is important for social engagement in MOOCs (and I think engagement with the instructor is important in education generally, so it should probably be important in MOOCs), I don't know what we're going to do. One of the comments from the interviews that's highlighted in the paper is that one student didn't have his question answered in the forums – and with my reading between the lines, that probably means we failed to socially engage this student, which sucks. I feel badly that a student felt ignored. But the reality is also that we had 23 000 people enrolled in Dino101, and had to manage hundreds of forum posts on very limited time budgets, in addition to managing the for-credit version of the course at the university. There is no way to scale up personal interactions between students and instructors in a learning environment without scaling up your teaching staff – either you need more instructors (in the form of profs, TAs, whatever), or you need fewer students, or you're not going to be able to interact with every student that wants interaction. And we shouldn't be asking instructors in a university to educate for free, so somebody needs to be paying for those extra people. So that's one more important takeaway here: social engagement requires a lot of time investment from the instructors to encourage discussion and set up an environment that encourages social interaction, so if you want to run a MOOC with high social engagement, you need to budget money for lots of instructor/TA time.

Anyway, that's probably one of the last posts I'm going to have about Dino101/MOOCs for a while, but I thought I would mention one other piece of news: before I moved down to Raleigh in 2014, I had started work on some new palaeontology mini-MOOCs in my role as the Science Digital Learning Manager at the UofA. After much hard work from the palaeo crew at the UofA, these courses are now just about finished and will be launching on Coursera over the next few weeks – if you liked Dino101, you might want to take a look at Theropods and the Origin of Birds (starting later in January), Ancient Marine Reptiles (starting in February), and Early Vertebrate Evolution (starting in March, not yet available for registration at Coursera but keep an eye on the UofA's page there). 

7 comments:

  1. As someone who has completed a lot of MOOCs (including the initial offering of Dino 101! :-) ), I am baffled at the importance that is given to participation in the fora.

    I can see how participation in meatspace (love this new-to-me term.... lol.........) might help people get engaged with a class.

    But in a MOOC? Where you are one of thousands? Or tens of thousands? Sometimes hundreds of thousands?

    Even if you find a person or two you enjoy engaging with, the chance that you will ever "see" them again, in another class, is very small. In my experience.

    I have basically stopped trying to read the discussion in any classes I take. Even when there are "only" 80 or 100 responses in a given thread .... that's a LOT to read. Especially when the signal-to-noise ratio is as low as I have essentially always found it to be.

    When a particularly interesting topic comes up, there are always multiple threads (thread creation needs to be moderated to prevent dupes), and an interesting thread is likely to have hundreds or thousands of posts (scattered across multiple threads). (And the VAST majority of all these words will not be of any interest, in my experience.)

    The current model of a deluge of verbiage with only occasional bright spots of interest doesn't work for me (and I suspect doesn't work for a lot of others).

    There's no way the staff for a class can have enough time to do the sort of work that would be necessary to make the fora really useful. Until Coursera and the other platforms figure out how to make use of the crowd to raise the signal-to-noise ratio in the fora, I predict the number of people who find the fora really useful will be (relatively) small.

    Which doesn't stop us from loving taking classes, loving individual classes (I really enjoyed Dino 101 and recommend it every time I realize it's being offered again), or from finishing classes. :-)

    ReplyDelete
    Replies
    1. I know there are a few things that Coursera has implemented to help with signal-to-noise, but it may not be enough - there's an option to give positive or negative points to a post, and then you can sort the discussion forum by 'top posts', but that may not be obvious to most students. It would be interesting to see if there's a way to merge threads with similar topics, although I don't know how you'd preserve the conversations within disparate threads.

      But yes, I basically agree with your general assessment that it may not be possible to easily make real connections to other students in one semester with thousands of people enrolled - it's hard enough in an in-person class where you might have a lab partner, say, but then never see that person again. Making meaningful connections takes real time and effort. This is partly why I find the idea of 'meatspace' (glad you liked that) connections for MOOCs really encouraging, because that might be the best way to fulfill the social aspect of engagement.

      Delete
    2. Yeah, the "top post" thing definitely not enough.

      My suggestion to Coursera for moderation of thread creation (to avoid duplication of thread topic -- someone would actually look to see if that topic already existed, and then send the student to the existing thread rather than creating a new one) has been roundly ignored.

      I know the class staff doesn't have time to moderate in this way -- my proposal was that student up-votes basically up-vote the person who made up-voted posts, over time, and that those up-voted people be (eventually) regarded as "trustworthy enough to be allowed to moderate thread creation."

      But even if the platforms were willing to do something like that, I think "unique thread topics" would be a drop in the bucket, compared to the crushing need to raise the signal to noise ratio to a reasonable level.

      As you say -- even in meatspace ;-) the possibility of actually making a real friend in one term is very low. Friendship takes longer than that.

      And in moocspace, where there are millions of students, whose interests may overlap but are not identical.......... The chance of making a lasting connection isn't very great. In my opinion. :-)

      But then I also think that it isn't very important. Talking about what we are learning may be important, but we can tell our friends and families................ :-)

      Delete
  2. I saw that theropods/birds is coming right up, and went to see the teaser video ... but didn't find a video? I make heavy use of teaser videos, when deciding which class to take next......

    Also ... wondering if class staff have indefinite access to emailing previous students? I hope so. And if so, I would love to see email about upcoming classes from staff teaching classes I've taken in the past..... Both repeats and new classes. I do recommend classes I enjoyed, whenever I see they are coming around again.

    ReplyDelete
    Replies
    1. If you have things you liked/disliked about how the course is being advertised or put together, I'd recommend contacting the production company (Onlea - website here: http://www.onlea.org/contact/) to give them some feedback! I'm not sure many people contact them directly, and I'm sure information related to how people choose to enroll would be valuable to them. (The preview videos are also something outside the scope of what the instructor does, so the production company is the best place to ask about this.)

      In terms of whether staff can email previous students: At least for the sessions I taught, it looks like it is possible to send an email, but I'm no longer at the UofA and the new instructor might not be able to access old sessions. This is another good suggestion to send to Onlea!

      Delete
  3. I followed your suggestion (thank you!) and talked to Onlea -- who are very responsive. They have made very nice teaser films for both the theropods/birds class and the marine mammals class but... Apparently the current Coursera tools are much harder to use than the previous ones, and it can take a week to get the teaser video up on the course intro page...........

    Odd, to make the tools harder to use.................

    Who knows.

    I'm signed up for theropods/birds........ :-)

    I think the staff ability to email previous students is likely to be a Coursera thing rather than something Onlea could impact. I hope staff will make use of it to let us know about new MOOC offerings from their departments.

    ReplyDelete
    Replies
    1. I noticed that new courses being offered by the UofA have a different format compared to Dino101 (different colours, the sidebar is arranged differently, etc.), and it looks like the way quizzes are incorporated into the videos has changed as well. Will be interesting to see what the new palaeo MOOCs look like!

      Delete