December issue.indd

Selecting the best from the rest

C. Bulstrode and V. Hunt
University of Oxford Medical School, Headington, Oxford, U.K.

Correspondence to: C. Bulstrode, Stream House, Mill Street, Stanton, St John, Oxon, OX33 1HQ

Introduction

The current method for selecting surgeons of the future is neither objective nor relevant to present day needs. An assessment is needed which ranks potential trainees according to aptitude, motivation and diligence, all of which is validated. This task should be a core duty for the Royal Colleges of Surgeons working in close co-operation with surgical specialty associations

INTRODUCTION
Entry to surgery remains highly competitive.There are still large numbers of applicants for every place on a training programme. The creation of seamless training programmes, amalgamating the registrar and senior registrar posts into the single specialist registrar grade (SpR), means that the selection of the best potential trainee from all the other applicants is even more crucial than before. Once chosen, there are now no further significant barriers to that individual proceeding to become a consultant surgeon.

Any assessment at entry to training should be primarily aimed at ranking applicants so that the ‘best can be chosen from the rest’. It should, therefore, aim to rank candidates (norm-based assessment) rather than simply measure whether they have reached a given standard (competence-based).

Selection exams should also focus on measuring potential rather than current ability, because, you are looking to what that trainee will be able to learn in the future, not what they can do now.

The current method of selection of surgical trainees relies on a set of unreliable, and/or irrelevant measures, which may or may not be surrogate measures of how the trainee will perform in the future. Certainly, there has been no research to show whether the appropriate attributes are being measured, or indeed how reliable these measurements are. The selection is divided into three main areas. The first is that the candidates must have spent a minimum period of time in a set of approved SHO posts. There is currently no attempt to determine how much or what they have learnt in these posts. A logbook must also be filled-in but this merely records what operations they attended and in what capacity. It is not discriminatory because there is no lower threshold of cases that must have been attended, nor does the logbook attempt to record what has actually been learnt when taking part in these cases.

The second hurdle is the MRCS examination itself, which is divided into two halves. Part one is a test of knowledge of basic science in relation to surgery. The second part of the examination tries to test problem solving and some basic technical skills. Both parts are supposedly designed as a competence exam with a minimum threshold for passing. There are some fundamental problems with this particular approach. The first is that adult learning theory suggests that facts are most easily learnt when related to relevant and practical tasks being performed as learning is taking place. The current requirement to learn large numbers of facts in isolation is at best a surrogate test of ‘motivation to become a surgeon’ or even a ‘rite of passage’ and is not based on any sound educational principle. Secondly, testing competence at the start of a training programme is illogical, because it is the training programme itself, which should lead to competence. Clearly, the candidate must be safe to start service work at the appropriate level of responsibility on day one of a training post, but this cannot be used for selecting the best, only for excluding the very few who, for whatever reasons, have not progressed in their SHO posts. Finally, the marking schemes used in the MRCS are not simply pass/fail, but rank candidates into as many as five grades. These grades are then used to cross-compensate, so that a candidate who is marked as ‘just not competent’ in one area can still pass the whole exam if they can make up marks by being found ‘more than just competent’ in other areas. This invalidates the whole concept of testing competence across the whole curriculum, but also fails to rank the candidates in any meaningful way.

The final and most difficult hurdle for an aspiring trainee is the application for a training post. Once again this is divided into two parts.

Short listing is done on the basis of a curriculum vitae or on a points system (sometimes it is a combination of the two).

Selection by curriculum vitae is both subjective and open to prejudice and should be quite unacceptable in this day and age.¹

The problem with a points system for short-listing is that only those attributes can be scored that are measurable (such as the number of research papers published). This means that important attributes such as ability to lead a team get little or no weighting in the selection process. It may be argued that the amount of research papers published is a way of measuring diligence, but once again it is a surrogate measure and, therefore, open to major confounding factors such as whether the applicant has worked in a research unit, or only in service-based posts.²

The next hurdle is the interview. This is usually unstructured or at best only minimally structured. There is good research evidence to show that the attributes most consistently selected at interview are based on a perception by the interviewers of likeness to themselves, the so-called ‘search for the son/daughter of yourself’.¹ Selection in the likeness of the last generation is comfortable but is hardly likely to be conducive to a profession moving forward in a rapidly changing environment. The final hurdle is the reference. This document should be powerful in giving a broad view of an applicant’s aptitude from observation over a period of time. Unfortunately, the use of the unstructured reference has made the document of little value in discrimination.

An assessment designed to provide fair, relevant and transparent ranking of surgical applicants would provide considerable advantages to candidates, selectors and to the profession.

As far as candidates are concerned, a ranking would give them a measure of how good they are and how happy they are likely to be in surgery (aptitude). Those who did not rank highly might then reflect on whether they should leave surgery at an early stage and pursue a career better suited to their attributes.

For the selectors a ranking score would give them good reliable evidence on which to base their decision on who to select for training in what is a very competitive environment.

The specialty should also benefit in the longterm by being able to consistently select those who will most enjoy and best serve the profession in the future.

WHAT ATTRIBUTES SHOULD WE SEEK?
If the profession worked out what attributes they were seeking in trainees and in future surgeons, without worrying how these might be measured, the debate might prove heuristic, drawing out what it is that we really should be seeking in trainees and surgeons in the future. This should be the first task of the Surgical Royal Colleges and the specialty associations, the Guilds who represent the interest of present and future surgeons.³

Some of the attributes being sought might be regarded as good things in any profession (e.g. honesty); others might be regarded as useful by many professions not just surgery (such as hand/eye coordination).⁴ Some may be unique to surgery. Whatever the attribute, there will undoubtedly be measures for some which have already been developed and validated elsewhere. The second task of the Surgical Guilds should, therefore, be to seek out these measures to determine if they have any value in the selection of surgeons.⁵

If any of the chosen attributes are found to be unique to surgery or do not already have a validated measure, it should be the third task of the Surgical Guilds to commission research into developing valid tools.⁶

Each attribute being measured will have a different level of importance in determining who would make a good surgeon. This weighting needs to be decided and validated before the whole tool is usable. Once again the Surgical Guilds are clearly the place where these problems should be considered and decisions reached.⁷

Tasks for the Royal Colleges and the Surgical Associations in developing fair, open and reliable methods for selecting surgical trainees:

• Search for validated tools already developed that measure these⁹

• Develop and validate tools for those that are not currently measured

The outcome of this work would be a relevant and reliable tool for ranking trainees that could be used by training programmes to ensure that they recruited the best applicants. Each training programme might choose to give a different weighting to each of the measures but the selection would be fair, relevant and open.

Intrinsic ability
These attributes are those which the trainee possesses as part of their personality. They might also be called aptitude and are not necessarily dependant on previous training or experience.⁴ They can probably be subdivided into technical, social and personal.

Technical
Surgery requires a significant amount of technical ability, although the amount and type may vary between different types of surgery. The most obvious that might be appropriate for surgeons are:

• Manual dexterity and hand/eye co-ordination (especially when working down a microscope or scope)4

Social
Social skills are not normally tested in selection (unless they are covered in the reference) because it is difficult to design relevant and reliable methods of measurement. This is no excuse. The challenge should be to define the most important domains and then devise validated tests for them. Three of these in surgery might be:

• Ability to communicate with patients and other members of staff10

Personal
There is a degree of overlap here with other domains, but the headings for selecting surgeons might include:

There is also a degree of interaction within these attributes. A highly motivated trainee will probably appear more diligent and indeed learn better than an unmotivated one, but nevertheless these variables all independently affect the quality of the trainee and the result achieved at the end of training.

ACQUIRED ATTRIBUTES
It may also be that it is felt that doctors entering surgical training will require a high level of knowledge of anatomy and physiology, to take but two examples. It would be perfectly reasonable to set tests of these, which must be passed to be even considered for selection. We know that learning facts in isolation, however, is not nearly so easy as learning in relation to skills which are being practiced.¹¹ It might, therefore, be argued that setting a high level test of basic factual knowledge at this stage is actually acting as a surrogate measure of motivation to become a surgeon. It is also putting in place a hurdle which, if hard enough, begins to look like a ‘rite of passage’ rather than serving any real function in selecting the best. Some would argue that the acquired attributes should only include basic care of the acutely sick patient and that a test of factual knowledge of such things as anatomy should come later when they are actually being used. There are, however, other acquired skills such as the ability to handle a complaint, resolve conflict, take consent, negotiate with a patient and organise a team that can be learnt and which ought to be both taught and assessed. If these characteristics were defined as basic requirements by the guilds then aspiring surgeons would seek out teaching in these areas and the calibre of young doctors applying for training posts would improve.⁸

SUMMARY
It is now time that the Surgical Guilds should define what it is they are seeking in surgical trainees, design, validate and weight the relevant tools to measure these, and set up systems whereby the best young doctors who wish to be considered for training as surgeons can be reliably ranked and so selected on fair, open and relevant grounds. The process would benefit trainees, trainers and the profession itself. It is a task for which the Surgical Guilds are uniquely suited.

REFERENCES
1. Bulstrode C, Pearson C, Hunt V. Appointing doctors. Skills Unit Oxford 1998.
2. Lettin A. Pers comm. BOA Instructional Course 1996.
3. Paisley A, Baldwin PJ, Paterson-Brown S. Validity of surgical simulation for the assessment of operative skill. Br J Surg 2001; 88(11): 1523-32.
4. MacMillan A, Cuschieri A. Assessment of innate ability and skills for endoscopic manipulation by the adavanced Dundee Endoscopic Psychomotor Tester: predictive and current validity. Am J Surg 1999; 177(3): 274-77.
5. Treasure T, Valencia O, Sherlaw-Johnson C et al. Surgical performance measurement. Health Care Manag Sci 2002; 5(4): 243-48.
6. Darzi A, Mackay S. Assessment of surgical competence. Qual Health Care 2001;10 (suppl 2): 64-69.
7. Ritchie W. The measurement of competence. Bull Am Coll Surg 2001; 86(4): 10-15.
8. Ilott I, Bunch G. Competencies of basic surgical trainees. Ann Roy Coll Surg Engl 1998; 80 (suppl 1): 14-16.
9. Darzi A, Smith S, Taffinder N. Assessing operative skill. Needs to become more objective. BMJ 1999; 318(7188): 887-88.
10. Perkins J, Sanson-Fisheer R, Anseline P et al. A preliminary exploration of the interactional skills of trainee surgeons. Aust N Z J Surg 1998; 68(9): 670-74.
11. Bulstrode C, Hunt V. Training in practice. Skills Unit Oxford 1998.

Under the patronage of
His Highness Sheik Makhtoum Al Makhtoum in association with the Ministry of Health, UAE

For further information please contact: Ms. Louise Loughran, Conference and Functions Officer, Royal College of Surgeons in Ireland, 123 St. Stephen’s Green, Dublin 2, Ireland, Tel: + 353 1 40 22 437, Fax: + 353 1 40 22 458, Email: conferences@rcsi.ie

TABLE 1. SECTIONS USED FOR CURRENT SELECTION OF TRAINEES FOR SURGERY
Task	Assessment method	Criteria	Assessment type
Work in appropriate posts	Signed up	Arbitrary minimum time	Criterion
Logbook	Operations attended/performed	None defined	N/A
Written test. Part 1 MRCS	Factual knowledge	Arbitrary pass mark	Criterion
Vivas MRCS Part 2	Facts and problem solving	Arbitrary pass mark with compensation	Notional competence
Short listing	Review of CV	Points system only on those things that can be measured	Normative subjective
Interview	Question and answer	Likeability	Subjective
Reference	Unstructured prose	Likeability	Subjective