Split from Team based learning part 2: educational research issues

[physics girl phd]

I just won my grant application (just for a mini-internal grant offered through our university's "Innovative Technology Center", just enough to give myself and my undergraduate student travel funds) to support our study of our implementation of team-based learning. Yeah!

Our findings thus far are based on expectations/attitudes surveys: Our students in a TBL environment see more "real world connection" at the end of our term (by ~10%). Usually, there is a DECLINE, see some University of Maryland College Park publications on their results from the MPEX, the Likart-scale Maryland Physics EXpectations survey). Our students overall attitudes still decline over a term, but our decline is low (about 5% shift to "novice" views), and mostly the shifts in views come about from large shifts in students' perceptions about the connection of math to physics... for a non-math-based course, these questions probably aren't appropriate. It could be that overall, students also just are sick of a course by the end of the term. I think however, that for a science-credit for non-science students or for an introductory course to possibly interest students in science careers, attitudes are important.

But we are looking at learning outcomes as well: Next term, we will be using some standard surveys for this, but at the end of this term I will compare the students results on the final exam to the results from a previous lecture-based version (using the same final exam). I am, however, pleased at the results from my midterm exams (which are, by the way, fairly difficult... not definitions-based as many 101 classes might be!). I'm not seeing grades suffer, and their performance is slightly improved (but the tests are not identical so analysis perhaps isn't appropriate). Note, I do online "pre-class" quizzes over the reading (some applications-based questions are for-completion only and I use extensive feedback to all their responses as one alternative form of lecture). I do have a summary session at the end of every 1-2 chapters (typically every 3-5 class sessions... I've had a bit more recently just because activities are difficult with thermo- and fluids in a lecture hall with 100+ students, and there aren't great simulations out there yet for these topics). I'll post more about this when the term ends and when my undergrad analyzes all the data... and of course next term when I start getting results from standardized concepts surveys.

Other anecdotal things: I know my students by name (probably also very important in a class with lower-level students), they ask me for interviews that they need for other classes and they ask me for recommendations. In a 101 course where the text states one of the main goals is to "reduce student fears" this is, in my opinion, good. Of course, I do want my students to demonstrate a foundation of basic science literacy by the end of the term also.

 

[Moonbear]

An obstacle I've run into that I hadn't considered fully before is the hurdle of IRB approval to collect useful data from the course. Unfortunately, my original plan to give out pre-tests, teach the class, and then give out post-tests is viewed as an "intervention" (the intervention being teaching the class in between ), plus, if I don't let students opt out, then it's coercion to participate in my research project. I might still be able to get the appropriate permissions to do this, but it would require full board review and approval, and that would have taken too long to get it going this semester (since I hadn't fully formulated my design in my mind until the term was halfway through). Now I see why people rarely include good outcomes measures in educational research...it's hard to do if the only subjects available to you are your own students.

My work-around is going to be waiting until the term is over when I'm no longer their instructor and sending the survey then, along with some quiz type questions to measure retention. On the plus side, I can eliminate the "lucky guess" factor by giving an option to every question of "I don't remember," which, being an anonymous survey and not any part of their grade, I think they might actually use if they don't remember this stuff after the course ends. The the hugely negative side, I'm worried that once the class is over, I'm going to have a hard time getting any reasonable return of surveys. The best I can do is offer an incentive that if they return the survey with an optional entry form enclosed with it, I'll enter them in a raffle for a gift card to the bookstore and hope enough of them could use the extra money for books to enter. It adds a few extra hoops to collecting the data to keep the surveys anonymous while getting entry forms that identify participants, but hopefully I've addressed that sufficiently for rapid IRB approval rather than having to wait for a full board review (at worst, it might need to do expedited review, which means going to a smaller panel of reviewers that can meet earlier than the full board).

The most annoying part though is how vaguely written all the regulations are for human subjects research. I mean, it gets more clear-cut if you want to do biomedical research, but there are so many categories and so much of the definition of those categories is based on intent rather than objective criteria that it's hard to navigate this approval process. It's much more straightforward to do animal research...I think the animals are better protected...it doesn't matter what you want to do with them, it always goes to a full board and all the rules are really clear-cut on what you can and cannot do. There's no second-guessing intent, the only intent they care about with animals is the intent to use them in any way, whether it's research or not.

 

[physics girl phd]

An obstacle I've run into that I hadn't considered fully before is the hurdle of IRB approval to collect useful data from the course.

Prior to receiving this grant, I was operating under the assumption I was exempt-status based on the fact that I was studying my own class... eliminating identifiers from data files, etc. and generally working under best-intent kind of procedures, but this grant does require proof of IRB exemption or status. So, while I was off-line just now... I was at a meeting with our institutions IRB compliance officer. What a coincidence!

My info:
It turns out with my survey tools and collection procedures, I will meet our criteria for "Expedited Form B" IRB approval because collecting transcribed audio-recorded interviews and written or online survey data that does not fully eliminate identifiers (aka. "blackboard" survey software). My present set of online data will be admissible under exempt-status if I have a colleague take my unopened- downloaded- file and replace identifiers with numbers, or if I have my undergraduate do this with a signed form about confidentiality (which she has to do to conduct and transcribe the interviews anyways). My new subjects next term will have to sign off on use via a consent form (but I plan to motivate them on this based on some "preliminary data from other UT classes").

Note: Giving the class credit/no credit for survey tools towards the class grade is OK too (in fact the institution that produced one of my survey tools recommended this to get high # of surveys returned). Why? it's not coercion to make them DO the tool, just coercion make them BE in the data set. In addition, just to cover my own rear, when I include the requirement on my syllabus, I also tell them I do not look at survey responses until the term is completed and that they have the option to discuss alternatives with me if they have concerns about the survey tools.. My option: basically I'd rescale the grade and take away the "free points." Right now they really LOVE those free points and keep asking me WHEN that exit survey will be there so they can get those free points to see its effect on their grade (like adding 2.5% is so hard! )

Conclusion: It doesn't seem like the "Expedited Form B" is too bad (although I have to enclose copies of all my survey instruments, confidentiality statements and eventually submit the signed consent forms, retaining copies in the department). The officer said approval for the Expedited Form B is typically a quick procedure, and she seemed comfortable with the precautions I'd been trying to take while operating without knowing about IRB procedures and gave me hints about how to treat the data sets make those meet exempt-status (even though I'd been operating with the actual "Exempt Status Form A" certification. And I think students will be ok with the consent form. I already tell them in my syllabus to discuss with me if they have any concerns about how the survey instruments will be used and give them an option to opt out in some way... they just have to sign off on this now. A hurdle, but really truly not that bad. I after all, have a bum-knee from a ski-accident and like to avoid hurdles.

 

[Moonbear]

Note: Giving the class credit/no credit for survey tools towards the class grade is OK too (in fact the institution that produced one of my survey tools recommended this to get high # of surveys returned). Why? it's not coercion to make them DO the tool, just coercion make them BE in the data set.

Yes, if they can opt out of the study, it's okay as long as they can opt out in a way that is not apparent to others in the class (apparently our IRB considers peer pressure to be coercion too...or maybe that was under the heading of not being confidential). The key is that if it's in the syllabus, then it's just standard educational practices and you're obtaining consent for the inclusion in the research study, but the data are collected for other purposes anyway. But, in my case, this isn't in the syllabus, at least not this year, so I can't do it that way. Next year, if I build this all into part of their grade, and the "research" part is just the preference survey and consent to let me use their grades, I also could do it with expedited review.

I wonder how many people do these types of studies and only after-the-fact realize they should have had IRB approval? I actually had someone from our research compliance office give a talk to a group of faculty in our department all interested in educational research last week, because as I went around asking other teaching faculty for help about how to fill out the forms, what category it was in, etc., I was greeted with an awful lot of blank stares. Those who have no research background and are only lecturers had a lot of projects planned but didn't even know they needed IRB approval (it's only a recent push by administration for them to be doing educational research in those positions). Fortunately, because I was bugging them with questions, we found this out BEFORE they started collecting any data, so they won't have collected anything that will have to be thrown out unused. The few where data are already collected, they were collected for other purposes initially (i.e., to help instructors evaluate the class) and only after-the-fact did they realize there was something interesting in those evaluations, so those all will fall under the exempt category as retrospective analyses of existing data sets.

The hassle here is that they've computerized the forms, and there's no way to skip fields for exempt protocols that really aren't necessary if it's in the exempt category. I WISH I had different forms for different categories. I've been filling out a lot of fields with "not applicable." I have to fill out the exact same set of forms as if I was doing a drug trial, which is frustrating.

 

[physics girl phd]

I've been filling out a lot of fields with "not applicable." I have to fill out the exact same set of forms as if I was doing a drug trial, which is frustrating.

You COULD change your topic to: The effect of drug X on student performance in course Y.

My main pro-point I mentioned RIGHT away at my meeting today... I'm not shooting them up with morphine before their tests or anything like that! (Looking through guidelines, I've now learned to append to that: I'm also not collecting hair or fingernail samples... other things that start to veer one towards full-review form B status!)

Also: count your joys that you aren't doing research on students that are in K-12... there you need parental consent. Guess I was lucky to have researcher instinct and know about needing approval for human subjects (I even looked at IRB "exemption status" before I started my project last year, decided I likely met that status, and then made notes on my syllabus, gave students option to opt out, etc.).

I wonder how many people do these types of studies and only after-the-fact realize they should have had IRB approval?

Not uncommon. If you don't have external funding (i.e. if you're mostly doing it to improve your own class or departmental classes without solid intent on publishing), and if your department chair or collegues aren't up-to date on this kind of research, you can probably still publish any real interesting findings, etc. without being "caught." My own husband confesses that once (~10 years ago) he probably should have had IRB approval because his sociology class collected interviews that he later used to some small degree in a publication. It's all soft ground here... slippery slope/ best-intent things ask forgiveness rather than approval. Although I certainly agree with the idea of IRB approval as important, as I'm sure you'll agree based on your more serious drug-study IRB projects. It's best to have regulation so you don't go down the wrong pathways at all.

Of course, I'm also glad that I'm finally getting some valid funding/ recognition and therefore do finally NEED to IRB it. That alone sets my project a bit apart and establishes my project as research (other faculty say they are interested in education research but aren't EVER collecting DATA)! Isn't data needed to even have any possibility of research results? But as I'm sure you're feeling: do we have to always train EVERYBODY else?

 

[Moonbear]

That alone sets my project a bit apart and establishes my project as research (other faculty say they are interested in education research but aren't EVER collecting DATA)! Isn't data needed to even have any possibility of research results? But as I'm sure you're feeling: do we have to always train EVERYBODY else?

Tell me about it! I'm quickly realizing that those who hired me and expect me to do educational research all think they are doing educational research too, when all they're doing is creating educational resources, not doing research. Mostly flash programs...I'd love to see what sort of evaluations they get on those, because I walked up to a group of students laughing really hard during lab one day and asked what was up, and they were pointing to one of these flash files and asking if it was supposed to be a picture of a dinosaur, because the graphics were so horrible, that's what it looked like (it was supposed to be a human skull!) Yes, it all counts as scholarly activity, but it's not educational research, but it seems that's what they had in mind when they were hiring. I'm not taking any chances. My promotion depends on convincing more than the people in my department that my activities are educational research. I don't think it's research until there are controls!

Hmm...maybe I should split off this discussion from the TBL part. We've really hijacked this thread.

 

[Moonbear]

Woo hoo! I got my IRB approval! What a relief! Now I just need to navigate the treacherous path of purchasing gift cards off state funds. (They're my incentive to get them to return the surveys...a legitimate expense, but I'm sure some bean counter is going to make me jump through hoops for them.)