AUDIT SECTION

A comparison between different formats of educational feedback to junior doctors: a prospective pilot intervention study

V. G. HADJIANASTASSIOU, D. KARADAGLIS and M. GAVALAS
Accident and Emergency Directorate, University College Hospital, London, U.K.

Introduction Methods

Results Discussion References

Objectives: Audit feedback is conventionally given in the format of an average performance of the department relative to established guidelines. The impersonality of this feedback format may compromise audit outcome. The aim of the study was to compare personal, peer-comparison feedback, an effective method of changing physician practice, to the ‘conventional’ departmental one. Methods: Eleven SHOs working at a central London A&E department were audited in terms of the adequacy of their documentation process. The study comprised patients presenting with closed head injury, acute asthma and chest pain of possible cardiac origin. Personal peer-comparison feedback and average departmental feedback was given for the first two groups, respectively. No feedback was given for the chest pain group, acting as a temporal control. The outcome measure was documentation of the minimum variables, as specified by departmental guidelines. Results: The peer-comparison feedback group showed a significant improvement (p-value<0.0001) in two out of three target variables audited, in contrast to the departmental feedback group and the control group which did not show a significant improvement in any of the variables. Conclusions: Personal, confidential peer-comparison feedback was more effective than departmental feedback, in an audit of process of junior doctors. This pilot study should form the basis of a larger study (more SHOs, wider range of conditions) to establish unequivocally the best format for educational feedback.

Keywords: medical audit, emergency service, education, continuing medical education, practice guidelines

J.R.Coll.Surg.Edinb., 46, December 2001, 354-357 

INTRODUCTION

Departmental audit feedback presentations in the United Kingdom are conventionally given in the format of an average performance of the department relative to established guidelines. The impersonality of this feedback format ignores interpersonal variations among doctors’ performances and hence might be partly responsible for any failures in audit outcome.

Devlin (see Frostick et al, (1993)) made the following state-ment: “direct feedback to each individual ... is clearly the ideal, ...probably logistically impossible to every (one) in each and every detail of practice.”¹ It was decided, therefore, to test this hypothesis by reviewing the process of documentation (“audit of process”) by Senior House Officers (SHOs) at a central London Accident and Emergency (A&E) department.

Two different formats of feedback to SHOs (closing the audit loop) were compared: a personal but confidential peer-comparison feedback to each individual SHO and a ‘conventional’ departmental one (no peer comparison).

METHODS

The study, extending over two consecutive 2.5-month periods, included all consecutive patients presenting with one of three common and potentially dangerous presentations: acute asthma, closed head injury or chest pain (of probable cardiac origin).

The patient sample was obtained from the hospital database, which contained information entered from the A&E case-notes of each patient. The audited case-notes, written by a group of eleven A&E SHOs, were traced, reviewed and inappropriate cases were excluded: (1) all patients seen by the authors; (2) all patients admitted; (3) asthma patients who had documented chronic obstructive airways disease, children who had never used a peak-flow meter device before, patients asking for medication; (4) closed head injury patients with a presenting Glasgow coma scale (GCS) of less than 14, all children brought to the department by their parents, those coded wrongly in this category when, after reviewing the case notes, there was no clinical suspicion of an actual “closed head injury” (such as facial lacerations or minor injuries to the head) and (5) all cases of chest pain with pleuritic pain or chest pain clearly of non-cardiac origin as assessed from the history.

The department had clear established guidelines of minimum standards of case-note documentation for a multitude of conditions. These guidelines were particularly detailed for patients presenting with potentially serious symptoms, assessed by SHOs and then discharged. All SHOs were expected to have read those guidelines, at the beginning of their posting. The guidelines served both the role of being a risk management mechanism as well as an educational tool. No specific induction training covered any of the three presenting conditions investigated in this study. The audit compared the variation between the actual practice of case-note documentation and the established guidelines.

The information collected from the case-notes was that specified by the local guidelines as the minimum essentials to be documented, in the assessment of these patients prior to their discharge. This comprised the following set out below.

For ‘asthma’, documentation of the peak expiratory flow rate (PEFR) on admission, the expected (or best) PEFR and the PEFR before discharge (or post-treatment). The rationale for this was derived from the national consensus guidelines for the management of asthma in A&E Departments.²

For closed head injury, documentation of an adequate assessment of symptoms and signs (half or more out of these four aspects: loss of consciousness, headache, vomiting, focal neurology), advice of the need for a person to escort the patient home and advice to have a reliable overnight companion after discharge. The need for documenting the latter two was incorporated in the departmental guidelines following the suggestions of the ATLS^® protocols for the management of “mild head injury”.³

 For chest pain, documentation of an adequate assessment of the cardiovascular risk factors (half or more out of the six: hypertension, smoking, cholesterol, diabetes, past cardiac history, relevant family history), the chest-pain symptom itself (half or more out of the four: time factor, quality of pain, relation to exertion, normal exercise tolerance), and investigations (Electrocardiogram (ECG), cardiac enzymes (CK)) requested.

The study period covered a 6-month post of the SHOs, with retrospective case-note audits covering the time period before and after a single feedback presentation. The latter was given to the SHOs halfway through their post, in the presence of their consultants. It has been reported that an educational intervention is more efficient when senior staff, in addition to juniors, are involved in the audit cycle^.4 The feedback was focussed at the SHOs, the “performers”, who were able to deliver directly the desired outcome of the audit. This was shown to be more effective than directing the feedback to their seniors alone.⁵ The feedback was given in two formats:

(a) on a confidential personal basis (peer-comparison) for the closed head injury group (b) on a ‘conventional’ departmental basis (no peer-comparison) for the asthma group.

No feedback was given for the chest-pain group in order to act as a temporal control to detect any differences in the standards of note keeping achieved by experience alone, without any influence of feedback, or any “Hawthorne” effect.⁶ The latter refers to the non-specific improvement in workers’ behaviour when they are made aware that their performance is monitored, irrespective of which individuals are singled out.

Each SHO was given confidentially a personal code number. As in most information feedback studies, a reminder of the relevant departmental guidelines immediately preceded the feedback session.4 This served the purpose of helping the doctors recognise the need for improvement in their practice; for feedback to have any effect, doctors have to recognise the need for change first and believe in it.⁷

Feedback on head injury documentation was presented in a table detailing the statistics for each SHO code. Each SHO could identify only their own personal performance and compare it anonymously with other colleagues’ performances and with the departmental average performance. In contrast, feedback for asthma was only given as a departmental average performance. The SHOs were blind to the purpose of the study, being a comparison of feedback methods, but were only informed at the feedback session that the results presented formed part of the usual audit project that each of the SHOs in the department was expected to perform.

RESULTS

Table 1 shows the documentation rates achieved by the same group of SHOs, for variables extracted in three common patient presentations, before and after the feedback session, with the chest pain group being the temporal control.

A two-tailed, Fisher’s exact test (95 % Confidence Intervals) was performed to compare the documentation rates before and after the feedback session, and also to compare the results of the corresponding time periods for the control group. Although the SHO group before and after the feedback session was the same, the distribution of patients seen per SHO was very different and, thus, the two groups could not be considered as ‘paired’ and, therefore, Fisher’s test was applied.

The table shows that the closed head injury group (personal feedback) had a highly significant improvement in the documentation performance of at least two of the variables extracted. The asthma and chest-pain groups did not show any statistically significant changes.

Table 1: The effect of feedback format in an audit of process

 DISCUSSION

On comparing the two feedback mechanisms, a statistically significant difference was found in the documentation standards achieved by SHOs. In addition, this difference was found to be independent of experience gained during the post, as shown by the results of the control group. This suggests that giving a personal peer-comparison feedback is a more effective way of promoting adherence to established guidelines of documentation and closing the audit loop in an audit of process.

The term audit is used in this article in both its limited meaning of a review of an activity as well as the broad sense of a three-part cycle.⁸ The cycle begins by defining a set of desirable standards, comparing actual practice with the standards and finally implementing a change, if required.

In the current ethos of practising medicine, the necessity for quality improvement and accountability is ever more real. A few effective methods of improving medical practice have been described.⁹ Feedback is one of these methods and involves giving physicians information about how they perform relative to expected standards or relative to their peers. Feedback has been described as “...more than simply education, it plays on the physician’s sense of achievement and desire to excel.”10 It is well documented that peer comparison feedback is successful in effecting a change in a doctors practice such as laboratory test use and antibiotic prescribing. ^11,12

Physicians can resist a change, which they perceive as threatening to their autonomy, value of clinical judgement and sense of competence.¹³ However, the SHOs gave a positive response to this type of personal feedback when presented with the results of the completed study. The general feeling was that the personalised audit gave them an incentive to improve, kept their concentration during the feedback presentation more than the impersonal feedback and was not intrusive, as the confidentiality of the participants was kept by the coding system.

It was felt that this personal approach to feedback was effective in disseminating good medical practice and identifying and changing bad practice. In this particular audit of process it was also a risk management tool by promoting better documentation. All the above features are consistent with the principles of clinical governance. As the authors were fellow colleagues of the SHOs the personal feedback has also been a form of peer review. The latter is a way to give a role to physicians in setting their own practice standards, which can mean that change appears less threatening.⁷

The feedback in this pilot intervention study was appropriately directed towards the people who were able to act on its recommendations.⁷ In a previous study where feedback was directed at senior doctors only, it failed to produce an improvement in the biochemical test requesting patterns.¹⁴ The cause was attributed to the lack of focus on the real target population - the junior doctors who requested the tests. The time period of the study was also specific to the target population, as all SHO posts are of 6 months’ duration. The single intervention halfway through their post, of personal peer-comparison feedback, was simple, easy to implement and was shown to be more effective than departmental (non-peer) comparison format.

A Medline^® search of the literature was performed, using the exploded index terms “Medical audit”, “Nursing Audit” and “Feedback” but could not identify any publications specifically comparing personal (peer comparison) versus departmental feedback format. A study by Hanson et al (1994), of personalised feedback to A&E SHOs on data recording, reported improvements consistent with our findings.¹⁵

One of the limitations of the current study is that different feedback was given for two different conditions and a third condition was used as a concurrent control. However, all three conditions share the same characteristics of being common presentations and the important decision-making variables, which had to be documented, were expected to be well known to SHOs at that stage of their career. In this way, the base-knowledge was not expected to be different for the three conditions. Another limitation was the low sample number, which was inevitable for a 6 month SHO post and the strict eligibility criteria set. Despite the low numbers, this pilot study has documented some significant statistical results showing that better audit outcomes can be achieved with a simple personalised peer-comparison feedback.

The study may be repeated in the near future, for a new cohort of SHOs at the department by reversing the feedback methods to each medical presentation. Such a “cross-over” design would give an indication whether the particular medical condition was relevant (confounding) to the feedback outcome. We advocate the institution of personalised peer-comparison feedback as the routine method of departmental feedback, when closing the audit loop. We recognise, however, that a larger study incorporating more SHOs and documenting a wider range of conditions is required. It is hoped that our results will encourage a multi-centre trial to establish unequivocally this educational issue, which encompasses most of the principles of clinical governance.

REFERENCES

1. Frostick SP, Radford PJ, Wallace WA. Rationale and practicalities. In: Medical audit. Cambridge University Press; 1993: 37-54
2. Neville RG, Hoskins G, Smith B, Clark RA. Asthma in accident and emergency departments. Thorax 1997; 52 (Suppl 1). Chart 5 S15: 153-6
3. American College of Surgeons Committee on Trauma.Head Trauma. In: Advanced Trauma Life Support^® for Doctors student course manual. 6th Edition.1997: 181-206
4. Mugford M, Banfield P. O’Hanlon M. Effects of feedback of information on clinical practice: a review. BMJ 1991; 303: 398-402
5. Restuccia JD. The effect of concurrent feedback in reducing inappropriate hospital utilisation. Med Care 1982; 20: 46-62
6. Roethlisberger FJ., Dickson WJ. In: Management and the worker. Cambridge, Massachusetts: Harvard University Press; 1956.
7. Greco PJ., Eisenberg JM. Changing physicians’ practices. New Engl J Med 1993; 329: 1271-74
8. Kogan M., Redfern S. What is audit? In: Making use of clinical audit. A guide to practice in the health professions. Open University Press; 1995: 1-16
9. Parrino TA. The non-value of retrospective peer comparison feedback in containing hospital antibiotic costs. Am J Med 1989; 86: 442-8
10. Eisenberg JM., Williams SV. Cost containment and changing physicians’ practice behaviour: can the fox learn to guard the chicken coop? JAMA 1981; 246: 2195-201
11. Berwick DM, Coltin KL. Feedback reduces test use in health maintenance organisation. JAMA 1986; 255: 1450-4
12. Schwinghammer TL, Romano MJ, Wing EJ. Cost savings associated with use of gentamicin vs. tobramycin. Am J Hosp Pharm 1985; 42: 1745-49
13. Klein D. Some notes on the dynamics of resistance to change: the defender role. In: Bennis WG., Benne KD., Chin R., Corey KE. Eds. The planning of change. 3rd Edition. 1976: 117-24
14. Grivell AR., Forgie HJ., Fraser CG., Berry MN. Effect of feedback to clinical staff of information on clinical bio-chemistry requesting patterns. Clinical Chemistry 1981; 27 (10):1717-20
15. Hanson J.M., Johnson G., Clancy M.J. The effect of audit and feedback on data recording in the accident and emergency department. J Accident Emerg Med 1994; 11: 45-7

Copyright date: 30th June 2001

Correspondence: V.G. Hadjianastassiou, Department of General Surgery, John Radcliffe Hospital, Headley Way, Headington, Oxford OX3 9DU, UK  
E-mail: vassilis@doctors.org.uk