|
|
Education Section
A randomised controlled trial comparing two methods of teaching medical students trauma and orthopaedics: traditional lectures versus the “donut round”
C. Bulstrode
F.A. Gallagher
E.L. Pilling
D. Furniss
R.D. Proctor
University of Oxford Medical School, John
Radcliffe Hospital, Headington, Oxford OX3 9DU
Correspondence to: C Bulstrode, University of Oxford Medical School, John Radcliffe Hospital, Headington, Oxford OX3 9DU Email:christopher.bulstrode@ndos.ox.ac.uk
|
|
|
Keywords: Randomised control, teaching, trauma, orthopaedics
Surg J R Coll Surg Edinb Irel., 1 April 2003, 76-80
Objective: To assess whether a new form of teaching, the ‘donut round’, is as good at imparting factual knowledge as interactive lectures in both the short-term and the long-term. Design: Randomised controlled trial. Setting: University of Oxford Medical School. Participants: 106 fifth year clinical medical students taught half of their A&E/trauma course by donut round and half by lecture. Main outcome measures: The results of multiple choice questions (MCQs) divided according to how the material was taught. Three MCQ papers were set: one at the end of a four-week course, one approximately 10 weeks later and a final exam approximately 17 months after the first. Results: At the first MCQ, the average result for questions taught by donut round was 41.0 (out of 50) and for those taught by conventional lecture was 40.1. At 10 weeks these averages fell to 36.3 and 37.3 and at 17 months they were 38.7 and 38.1, respectively. None of these pairs were significantly different. Ratios were calculated for each candidate by dividing their donut round score by their lecture score. The average ratios for the first, second and third MCQ papers were: 1.029, 1.007 and 1.027, respectively, and were not significantly different. The individual ratios of all candidates in all three MCQs were plotted against their equivalent total mark. The calculated linear regression showed a statistically significant advantage of donut rounds over lectures in those candidates who scored a total mark less than 89 (n=260, p=0.02). Conclusions: Donut rounds are at least as good as lectures in imparting factual knowledge and may provide a selective advantage to weaker students
INTRODUCTION
Many teaching methods are introduced into
the undergraduate medical curriculum without
objective assessment of their efficacy. Recently
in Oxford, the “donut round” has gained
popularity as a method of undergraduate
teaching in subjects as diverse as surgery, paediatrics, and accident and emergency
(A&E) medicine. Although they have been
shown to be popular with students, there are
no studies comparing the efficacy of this new
method with more conventional forms of teaching.1 The primary aim of the study was
to determine whether donut rounds or lectures
are better at transferring factual information in
both the short-term and the long-term.
Donut Rounds
Donut (Doughnut) rounds were first described
by Fleiszer et al in 1997.1 They consist of a
group of students and an expert facilitator
who meet on a regular basis (e.g. weekly).
The students are given the reading material to
be covered in advance. They are expected to
read this and to construct 10 questions from
the material in advance to test their fellow
students at the next one-hour session. Students
then take it in turns to ask their questions
to fellow students in the whole group. If an
individual cannot answer the question posed
to them, it is then opened out to the group.
Any student who fails to understand or
agree with the answer is permitted to begin
a discussion around the topic. This discussion
is then chaired by the facilitator who may
correct any misconceptions. In this study the
expert facilitators were all specialist registrars
(residents) in orthopaedics. Students sit in a
circle and donuts are frequently consumed
during the sessions (hence the name!).
LECTURES
Lectures were given by one of the authors of
the study (CB) using a traditional interactive
format where questions were encouraged.
Slides and overheads were used. The lectures
were based around the same material as used
for the donut rounds. The lecturer was not
involved in the writing of the multiple-choice
questions and was not shown them until the
end of the study.
SETTING
The clinical students in each year at Oxford
University are divided into six firms in year
five which rotate through six specialties. The
trial was conducted on all the 106 students
passing through the eight-week accident
and emergency (A&E), orthopaedics and
trauma attachments during the academic year
1998-9. Each half of the firm spent four weeks
studying A&E and trauma, where the trial was
conducted.
MATERIAL
The written material used in the donut rounds
was prepared by one of the authors (CB) and
had been edited by critical evaluation over two
years. Each section covered a different aspect
of orthopaedics and trauma and was around
5000 words long. The notes were divided into
four sections for the purposes of teaching:
Figure 1: Average total marks for each of the three MCQs with 95% confidence limits plotted alongside the average donut and lecture results
INFORMATION GIVEN TO
STUDENTS
At the start of the A&E
attachment each student was
asked for his or her verbal
consent to enter the trial. They
were told that it was a trial of
teaching methods and involving
a multiple-choice examination
(MCQ) at the end of the course,
the results of which did not
count towards their course
assessment. They were also
told that they would be asked to sit a second
MCQ in two months with a further one at the
end of their clinical course in year six. They
were free to opt out of the second and third
MCQs. Participants were given a brief outline
of the nature of the donut round and were
informed that their attendance at both lectures
and donuts rounds would be recorded for the
purposes of the study. Each student was also
given a copy of the notes for all four sections
of the course irrespective of their allocation
to be taught by lecture or donut round. Ethics
permission was not sought as patient care was
not involved directly or indirectly.
RANDOMISATION
The four topics were taught either by lecture
or by donut round. Each group received
two lectures and two donut rounds but the
decision as to how each topic was delivered
was determined by random allocation. Twelve
envelopes (one for each half of each firm in
the year) were prepared before the start of
the study. There were two envelopes for each
of the six possible permutations of lecture or
donut round (Table 1). At the beginning of
each attachment, an envelope was randomly
selected by one of the authors to determine
how each of the four sections was to be
taught.
ASSESSMENT
At the end of the four-week attachment, the
factual knowledge learnt in these four sessions
was tested using one hundred true or false
questions, 25 from each of the four sections.
The questions were randomly ordered and not
grouped according to which section they were
from. The answers were positively marked,
with one point being awarded for a correct
answer, and no points awarded for a wrong
answer or where no answer was offered. The questions were prepared by those organisers of
the trial who were not involved in the teaching
of the sections. The questions were validated
by both a specialist registrar in orthopaedics
and a small number of students in the year
above those in the trial to eliminate incorrect,
ambiguous and unreasonable questions. It was
not feasible to perform any more sophisticated
validation on the MCQs given that they were
only being used for this trial.
The same test was administered as near to 10 weeks as possible. The logistical problems of getting all the students together meant that the second test took place between six and thirteen weeks after completion of the first MCQ. Those who agreed to take the test received a £5 book token.
The third sitting of the MCQ took place as near to 17 months as possible (12 to 22 months) after the completion of the first exam. The variation in timing is because all the candidates took the exam on the same day during the revision course for their medical finals. Once again, all those who completed this exam received a £5 book token.
The candidates were not given their results for any of the MCQs, nor were they allowed to keep the MCQ papers or given the answers until after they had completed the third and final MCQ. At that meeting, the correct answers were discussed with them to give the exercise educational value on the part of the candidates.
STATISTICAL ANALYSIS
On marking the questions, they were divided
into those that had been taught by donut round
and those that had been taught by lecture
for each candidate. The average result for
questions taught by the two methods was
calculated for each of the three MCQs. These
means were compared by using the standard
error of the difference between two means
and the significance of this standard error was
calculated from the appropriate table using a
level of 0.05 as indicative of likely statistical significance.2
Secondly, the ratios of questions taught by donut and answered correctly to those taught by lecture and answered correctly for each individual student were calculated. These were averaged for each of the three MCQs. Again, the standard error of the difference between two means was used to calculate the significance between averages.
Linear regression lines with corresponding correlation coefficients and statistical significance were calculated by the plotting programme Microcal® Origin.
Numbers and exclusion (Figure 1)
Out of 106 candidates in the 1997
intake to the Oxford University clinical
school, eight were excluded for the
following reasons: one candidate was ill
throughout the course, one had already
attended the course the previous year,
one dropped out of the year and two
studied for the course at another centre.
Three candidates failed to attend the
first MCQ within the allotted period.
Therefore, 98 candidates (92%) of the
year participated in the study.
Of those candidates included in the study, 67 (68%) attended for the second MCQ at 6-13 weeks and 95 (97%) attended for the final MCQ.
MCQ results
The results were analysed in two
separate ways. Firstly, the average result
for each MCQ was calculated according
to the method of teaching. Of the 98
candidates sitting the first MCQ at the
end of the course, the average total mark
was 81.1 out of the 100 questions (95%
confidence intervals CI: 68.3-94.0). The
candidates did slightly better at the half
of the exam taught by donut (41.1, 95%
CI: 34.4-47.6), compared with the half
taught by lecture (40.1, 95% CI: 32.0-48.2), although this difference was not
significant. The second MCQ showed
a reduction in the total average mark
(73.6, 95% CI: 47.4-99.9). The average
donut result was 36.3 (20.1-52.5) and
the average lecture result was 37.3
(27.0-47.7), the difference being
nonsignificant. The total average result for
the final MCQ was 76.8 (64.1-89.5).
The average donut result was 38.7
(32.0-45.5) and the average lecture
result was 38.1 (29.7-46.4) and, again,
the difference was not significant. A
summary of these findings can be found
in Figure 1.
The second way in which the results were analysed was on an individual candidate basis. A ratio of the questions taught by donut and answered correctly to those taught by lecture and answered correctly was calculated for each candidate for each MCQ that they sat. Thus, a ratio of greater than one would reflect a candidate who scored higher on the questions taught by donut than on those taught by lecture. This compensated for the different abilities between candidates and allowed a direct comparison between the two methods of teaching. These ratios were averaged for all candidates sitting each of the MCQs. In the first MCQ, the average ratio was 1.029 (95% CI: 0.828-1.231), in the second the average was 1.007 (0.780-1.235) and in the third the average was 1.027 (0.794-1.261). There was no significant difference between these. These results are summarised in Figure 2.
To assess whether the donut had a selective advantage for academically weaker students, a figure was plotted of each candidate’s donut/lecture ratio against their corresponding total mark assuming that the total mark was a reflection of overall academic ability. This was done for each individual MCQ and for all three MCQs combined. Linear regression was calculated for all of these figures and a line of negative gradient indicated a relative advantage of donut rounds over lectures for weaker students. The combined linear regression for all three MCQs (n=260) showed a statistically significant negative gradient (p=0.02) and was fitted by the line: y=1.18-0.002x (Figure 3). Therefore, below a total mark of 89, the average donut/lecture ratio was greater than one, that is, students who scored less than 89 did better on questions taught by donut round rather than by lecture. On analysing the individual MCQs, the donut round showed a statistically significant advantage to weaker candidates in both the first and third MCQs (p=0.002 and p=0.01, respectively). There was no selective advantage to either form of teaching in the second MCQ (p=0.95).
DISCUSSION
This trial was started because the
authors believed that donut rounds are
an important advance in teaching factual
knowledge. The results show that
donut rounds are as good as interactive
lectures in imparting factual knowledge.
Furthermore, the donut round appears to
have a selective advantage for weaker
students. One explanation for this is
that academically successful students
are more likely to be motivated and,
therefore, will learn regardless of the
method by which they are taught. Less
academic students may benefit more
from the stimulation provided by novel forms of teaching than from didactic
lectures.
Figure 2: Average donut/lecture ratios for the three MCQs with 95% confidence limits
Figure 3: Individual donut/lecture ratios plotted according to total mark for all three MCQs. The line represents the calculated linear regression
The study has several limitations. Firstly, the numbers in the study were small and the attendance at the second MCQ was low due to the voluntary nature of the examination. The second and third MCQs were not at a fixed point because of the problems in organising follow-up among volunteers during their subsequent attachments in different specialities. Therefore, there is a great deal of variation in the times between the different MCQs. Also, the overall marks in the examination were consistently very high, which may reflect the high quality of both the lectures and the donut rounds but as a result this reduces the sensitivity of the MCQ to discriminate between the two teaching methods.
Clearly the students will have learnt some of the material tested in the MCQ examination by means other than the donut round or lectures, for example, informally from other doctors.
Also, as we did not assess the precourse knowledge or ability of the students, we cannot be sure that (for each section) those taught by donut round or lecture were equal. We felt that to add a fourth exam to do this would have reduced compliance with subsequent follow-up.
However, every attempt was made to try and ensure that these potentially confounding factors were balanced by the randomisation process such that we would expect equality. Even if there were to be an unexpected disparity in ability, there should be no impact on outcomes as each student acted as their own control. Comparisons were made between sections for each student rather than between students for each section.
Furthermore, the nature of the MCQs designed means that only factual knowledge was tested. Although this has relevance in the practice of medicine, many other aspects of learning are equally important such as understanding, problem solving, practical skills, and communication. We have no evidence that the donut round is of any benefit in these forms of learning. It is possible that because of the discussion involved during the donut session - which may not occur during a lecture - an understanding of a subject may be created which is deeper than the simple factual knowledge measured by the MCQ. However, this trial did not test this.
One further limitation of the donut round is the need for good source material, small groups and a facilitator. This has resource implications in already overstretched medical schools, and also limits their use in situations where the ratio of students to teachers is high. One lecturer can teach large groups of students in the conventional lecture setting. However, the facilitator requires less preparation for the donut round than for a traditional lecture and although the source material in the case of the study was written specifically for the examination, this may be substituted with chapters from an appropriate textbook providing that it is clear, relevant and set at the correct level. Despite the increased preparation time on the part of the students, informal feedback from the participants has revealed that in the main they found donut rounds at least as enjoyable as traditional lectures. The study was not designed to formally test this.
In conclusion, this randomised controlled trial has shown that a new form of medical teaching is at least as good as traditional lectures in imparting factual knowledge both in the short, and long-term and that it may be especially beneficial in less academically able students. This article also shows that it is possible, albeit difficult, to study teaching interventions by well-designed randomised controlled trials. The authors do not suggest that the donut round should replace didactic forms of teaching but rather that it should be used in conjunction with them to provide a range of teaching methods aimed at different abilities.
ACKNOWLEDGEMENTS
The authors would like to thank the Skills
Unit (Oxford) for providing the book
tokens and logistical support. Thanks
are also due to Mr Harry Brownlow for
reviewing the examination questions
and Miss Cass Kellett and Mr Mark
Emmerton for assistance in running the
donut rounds.
REFERENCES
1. Fleiszer D, Fleiszer T, Russel R.
Doughnut rounds: a self-directed
learning approach to teaching critical
care in surgery. Medical Teacher
1997;19: 190-93.
2. Swinson TDV. Statistics at square
one. 1997; Available at: http www.bmj.com/collections/statsbk.
Copyright: 19 December 2002