The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of twelve UK medical schools BMC Medicine 2013, 11:244 IC McManus, Chris Dewberry, Sandra Nicholson and Jonathan S Dowell
This study looked at 4,811 students in 12 UK medical schools who took the UKCAT from 2006 to 2008.
Researchers concluded that UKCAT scores and educational attainment measures were significant predictors of outcome. The incremental validity of UKCAT taking educational attainment into account was significant, but small. Medical school performance was also affected by a contextual measure of secondary schooling, students from secondary schools with greater average attainment at A-level performing less well.
This collaborative study in 12 medical schools shows the power of large-scale studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training.
Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: Meta-regression of six UK longitudinal studies BMC Medicine 2013, 11:243 I C McManus, Chris Dewberry, Sandra Nicholson, Jonathan S Dowell, Katherine Woolf and Henry WW Potts
Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register.
Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance. The researchers suggest that some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills.
The UKCAT-12 Study Technical Report Educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in twelve UK medical schools. IC McManus, C Dewberry, S Nicholson, J Dowell
The following Technical Report was previously submitted to the UKCAT Consortium to inform the Consortium of progress in analyzing the UKCAT-12 datasets. The full report is now being released, unchanged, to coincide with and to supplement the publication in BMC Medicine of a group of three papers, two of which analyse the UKCAT-12 data in more detail. The Report provides more details on some of the analyses and may be of use to those interested in UKCAT. It should be emphasized that the Report was a working document, that none of the conclusions were final, and indeed some of the conclusions and the methods on which they were based have been updated, in some cases as a result of minor errors having been found. The reader should therefore interpret the findings in that context and in conjunction with the three BMC Medicine papers.
Download the Technical Report
Socioeconomic status of applicants to UKCAT Consortium Medical Schools 2009 – 2012. Published on the Medical Schools Council website May 2013 Lynda Cochrane, David Ridley, Bruce Guthrie, Jon Dowell
This analysis examines the socioeconomic status of applicants to the 26 medical schools in the UKCAT consortium. It assesses the impact of increased fees on the socioeconomic profile of applicants to medicine. It was produced by the Health Informatics Centre, University of Dundee, on behalf of the Medical Schools Council.
The UK clinical aptitude test and clinical course performance at Nottingham: a prospective cohort study BMC Medical Education 2013, 13:32 Janet Yates and David James
The study built on earlier work, investigating whether 2006 UKCAT results predicted perfomance in the clinical components of the medical programme at Nottingham. The authors report that verbal reasoning and the UKCAT total score show modest correlation with clinical course marks. The authors go on to conclude that these are significantly less strong than the predictive ability of performance earlier in the programme.
Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study BMC Medical Education 2012, 12:69 Jane Adam, Miles Bore, Jean McKendree, Don Munro and David Powis
This study shows numerous significant relationships between both cognitive and non-cognitive test scores, academic examination scores and indicators of professional behaviours in medical students. This suggests that measurement of non-cognitive personal qualities in applicants to medical school could make a useful contribution to selection and admission decisions. Further research is required in larger representative groups, and with more refined predictor measures and behavioural assessment methods, to establish beyond doubt the incremental validity of such measures over conventional cognitive assessments.
The UK Clinical Aptitude Test: Is it a fair test for selecting medical students? Medical Teacher posted on-line May 2012 Paul Lambe, Catherine Waters, David Bristow http://informahealthcare.com/doi/full/10.3109/0142159X.2012.687482
The aim of this study was to determine if differences in: access to support and advice, in modes of preparation, type of school/college attended, level of achievement in mathematics, gender and age influence candidate performance in the UKCAT and unfairly advantage some candidates over others. The study was carried out using questionnaires of applicants to study on an undergraduate medical degree course. Researchers found that differentials in access to support and advice, in modes of preparation, type of school/college attended, in level of achievement in mathematics, gender and age were found to be associated with candidate performance in the UKCAT. The findings imply that the UKCAT may disadvantage some candidate groups and that this inequity would likely be improved if tutors and career advisors in schools and colleges were more informed about the UKCAT and able to offer appropriate advice on preparation for the test.
Widening access to UK medical education for under-represented socioeconomic groups: modelling the impact of the UKCAT in the 2009 cohort BMJ 2012;344:e1805 Paul A Tiffin, Jonathan S Dowell, John C McLachlan
The authors set out to determine whether the use of the UK clinical aptitude test (UKCAT) in the medical schools admissions process reduces the relative disadvantage encountered by certain sociodemographic groups. They were able to analyse detailed candidate and admissions data and look at how and how strongly different medical schools used the test. They concluded that the use of the UKCAT may lead to more equitable provision of offers to those applying to medical school from under-represented sociodemographic groups. This may translate into higher numbers of some, but not all, relatively disadvantaged students entering the UK medical profession.
Can the UKCAT select suitable candidates for interview? Medical Education 2011: 45: 1041–1047 Rebecca Turner, Sandra Nicholson
This paper examines current selection practices and questions the role that the UKCAT may take, focussing in particular on whether UKCAT can select suitable candidates for interview. The rejection rate before interview of candidates with low UKCAT scores was 2.7 times that of candidates with high UKCAT scores. However, no relationship between overall UKCAT score and overall interview score existed within a pre-selected cohort of applicants with high UKCAT scores. The paper concluded that the UKCAT can facilitate the independent selection of appropriate candidates for interview but that it is not predictive of success at interview.
A mixed-methods study identifying and exploring medical students’ views of the UKCAT Medical Teacher 2011 Vol. 33, No. 3 , Pages 244-249 J.A. Cleland, F.H. French, P.W. Johnston
This was a mixed-methods study using a paper-based survey and focus groups with first year medical students in Scotland in 2009–2010. Questionnaire data were analysed using SPSS, focus group data using framework analysis. The UKCAT was viewed unfavourably by first year medical students completing it pre-admission. These negative views seem due to concern as to the use of UKCAT data, and the fairness of the test. The authors conclude that more evidence as to validity and fairness of the UKCAT and how it is used in practice is required.
Predictive validity of the personal qualities assessment for selection of medical students in Scotland. Medical Teacher 2011, Volume 33 Jon Dowell, Mary Ann Lumsden, David Powis, Don Munro, Miles Bore, Biokanyo Makubate, Ben Kumwenda
The Personal Qualities Assessment (PQA) has been used within the non-cognitive sub-test of the UKCAT, having been developed to enhance medical student selection by measuring a range of non-cognitive attributes in applicants to medical school. This study pre-dates the introduction of the UKCAT with applicants to the five Scottish medical schools piloting the PQA in 2001 and 2002. The study aimed to evaluate the predictive validity of PQA through a longitudinal cohort study in which PQA scores were compared with senior year medical school performance. The most significant finding was that students identified by PQA as ‘not extreme’ on the two personal characteristics scales performed better in an OSCE measure of clinical performance.
Use of UKCAT scores in student selection by UK medical schools, 2006-2010 BMC Medical Education 2011, 11:98 Jane Adam, Jon Dowell, Rachel Greatrix
This paper summarises the way medical schools have used the test since its inception in 2006, categorises that use and demonstrates trends in use over time. Annual telephone interviews were conducted with UKCAT Consortium medical schools. The paper reports that four ways of using the test results have emerged (Borderline, Factor, Threshold and Rescue methods) with many schools using more than one method. The considerable variation in how medical schools use UKCAT supports the wish of the Consortium that applicants are clearly informed about how the test will be used in order that they can make best use of their limited number of applications.
Comparison of A-level and UKCAT performance in students applying to UK medical and dental schools in 2006: cohort study BMJ 2010;340:bmj.c478 David James, Janet Yates, Sandra Nicholson
The object of this study was to determine whether the UK Clinical Aptitude Test (UKCAT) adds value to medical and dental school, and in particular whether UKCAT can reduce the socioeconomic bias known to affect A levels. The study suggests that the UKCAT has an inherent favourable bias to men and students from a higher socioeconomic class or independent or grammar schools. At the same time however, it does provide a reasonable proxy for A levels in the selection process.
The value of the UKCAT in predicting pre-clinical performance: a prospective cohort study at Nottingham medical school. BMC Medical Education 2010 10;55 Janet Yates, David James
The study aimed to determine whether UKCAT scores predict performance during the first two years of the 5-year undergraduate medical course at Nottingham. This limited study suggested that the predictive value of the UKCAT, particularly the total score, is low although section scores may predict success in specific types of course assessment.
Has the UK Clinical Aptitude Test improved medical student selection? Medical Education 2010: 44: 1069 – 1076 Sarah R Wright, Philip M Bradley
The study aimed to determine whether the UKCAT had made any improvements to the way medical students are selected. The analysis studied the ability of previous school type and gender to predict UKCAT, personal statement or interview scores in two cohorts of accepted students. The results demonstrated that previous school type was not a significant predictor of either interview or UKCAT scores amongst students who had been accepted onto the programme. UKCAT scores were significant predictors of knowledge examination performance for all but one examination administered in the first two years of Newcastle Medical School whereas interview scores are not.
A comparison of the UKCAT with a traditional admission selection process. Informahealthcare 2009, Vol. 31, No. 11 , Pages 1018-1023 Nishan Fernando, Jennifer Clelland, Kathryn Greaves, Hamish Mckenzie
The aim of this study was to compare candidate UKCAT performance with medical student selection outcomes at the University of Aberdeen. Results from this study indicate that UKCAT scores show weak correlation with success in Aberdeen’s medical admissions process indicating that the UKCAT examines different traits.
Does the UKCAT predict Year 1 performance in medical school? Medical Education 2009: 43: 1203 – 1209 Bonnie Lynch, Rhoda MacKenzie, Jon Dowell, Jennifer Clelland, Gordon Prescott
This study set out to identify whether UKCAT total score and subtest scores predict Year 1 outcomes at the Universities of Aberdeen and Dundee. UKCAT scores did not predict Year 1 performance at the two medical schools suggesting that studies of UKCAT’s ability to predict outcomes in later years of medical school and in subsequent qualification and practice as a doctor are indispensable in determining whether the use of the test as a selection tool is justified.