A summary of relevant published work since 2009.
The Predictive Validity of a Text-Based Situational Judgment Test in Undergraduate Medical and Dental School Admissions
Academic Medicine (2017), Fiona Patterson, MSc, PhD, CPsychol, Fran Cousans, MSc, Helena Edwards, MSc, CPsychol, Anna Rosselli, MSc, Sandra Nicholson, MRCGP, MSc, PhD, and Barry Wright, MRCPsych, MD.
Situational judgment tests (SJTs) can be used to assess the nonacademic attributes necessary for medical and dental trainees to become successful practitioners. Evidence for SJTs’ predictive validity, however, relates predominantly to selection in postgraduate settings or using video-based SJTs at the undergraduate level; it may not be directly transferable to text-based SJTs in undergraduate medical and dental school selection. This preliminary study aimed to address these gaps by assessing the validity of the UK Clinical Aptitude Test (UKCAT) text-based SJT.
Report to the UKCAT board: Enhancing the scoring of the situation judgement test component of the UKCAT; reanalysis of the validity pilot study data
Paul Tiffin & Lewis Paton, University of York, October 2016 (Summary report produced March 2017)
Previous work analysing item level data from the situation judgement tests (SJTs) suggest that reliability could be potentially increased by using an unweighted scoring system, selecting a subset of items that relate strongly to the main dimensional being measured and exploring scoring patterns using item response theory (IRT) approaches. However, it was unclear whether increasing the reliability of the test would automatically increase validity, as it may have been possible that important test content could be lost taking this approach. The original validity pilot study data provided an opportunity to evaluate competing scoring approaches in respects of both reliability and validity.
Exploring the validity of the 2013 UKCAT SJT- prediction of undergraduate performance in the first year of medical school: Summary Version of Report
Paul A. Tiffin & Lewis W. Paton, Health Sciences, University of York, March 2017
Previously it has been shown that the scores from some forms of the UKCAT 2013 SJT predict tutor ratings in medical students. In addition, the cognitive scale scores of the UKCAT tend to predict undergraduate performance on both theory and skills-based assessments in medical school. In these analyses we model the relationship between such undergraduate academic performance measures and the 2013 SJT scores.
Do personality traits assessed on medical school admission predict exit performance? A UK-wide longitudinal cohort study
Adv in Health Sci Educ (2016), MacKenzie, R.K., Dowell, J., Ayansina, D. et al.
Traditional methods of assessing personality traits in medical school selection have been heavily criticised. To address this at the point of selection, “non-cognitive” tests were included in the UKCAT, the most widely-used aptitude test in UK medical education. This study examined the predictive validity of these non-cognitive traits with performance during and on exit from medical school by sampling all students graduating in 2013 from the 30 UKCAT consortium medical schools.
Predictive validity of the UKCAT for medical school undergraduate performance: a national prospective cohort study
BMC Medicine 2016 14:140, Paul A. Tiffin, Lazaro M. Mwandigha, Lewis W. Paton, H. Hesselgreaves, John C. McLachlan, Gabrielle M. Finn and Adetayo S. Kasim
The UKCAT has been shown to have a modest but statistically significant ability to predict aspects of academic performance throughout medical school. Previously, this ability has been shown to be incremental to conventional measures of educational performance for the first year of medical school. This study evaluates whether this predictive ability extends throughout the whole of undergraduate medical study and explores the potential impact of using the test as a selection screening tool.
Does the UKCAT predict performance on exit from medical school? A national cohort study
BMJ Open 2016 ; 6 R K MacKenzie, J A Cleland, D Ayansina, S Nicholson
Most UK medical programmes use aptitude tests during student selection, but large-scale studies of predictive validity are rare. This study assesses the UKCAT, and 4 of its subscales, along with individual and contextual socioeconomic background factors, as predictors of performance during, and on exit from, medical school.
Longitudinal assessment of the impact of the use of the UK clinical aptitude test for medical student selection
Medical Education 2016 50:10, Jonathan Mathers, Alice Sitch, Jayne Parry
This paper aims to provide the first longitudinal analyses assessing the impact of the different uses of UKCAT on making offers to applicants with different socio-demographic characteristics. Multilevel regression was used to model the outcome of applications to UK medical schools during the period 2004–2011. The authors’ findings demonstrate attenuation of the advantage of being female but no changes in admission rates based on White ethnicity, higher social class and selective schooling. In view of this, the utility of the UKCAT as a means to widen access to medical schools among non-White and less advantaged applicants remains unproven.
Do differentials in access to advice and support at UK schools on preparation for the UK Clinical Aptitude Test disadvantage some candidate groups?
Paul Lambe, Rachel Greatrix, Keith Milburn, Jon Dowell, David Bristow
The UKCAT is part of the selection process of the majority of UK medical and dental schools. However, a survey of applicants to a single UK medical school indicated that differentials in access to advice and support on test preparation may disadvantage some candidate groups. To assess the generalizability of this finding and extend understanding, a survey of the 2012 UKCAT candidate cohort was conducted.
Candidates’ views on the UK Clinical Aptitude Test: Results from the Post 2012 UKCAT Survey
Paul Lambe, Rachel Greatrix, Keith Milburn, Jon Dowell, David Bristow
Aptitude test are used internationally with the aims of complementing existing selection tools, improving the fairness of selection and widening access to the study of medicine. The UKCAT is employed by the majority of UK medical and dental schools. Recent studies have raised concern about the lack of transparency about test usage in the selection process and the impact this may have on widening access. Along with examining candidates’ views about the test, this study examines the question whether candidates’ understanding of how universities use the UKCAT in the selection process influences application decisions, and, what impact this may have on widening access, given UKCAT score achieved and candidate background.
Widening access in selection using situational judgement tests: evidence from the UKCAT
Medical Education 2016 50:6, Filip Lievens, Fiona Patterson, Jan Corstjens, Stuart Martin and Sandra Nicholson
Widening access promotes student diversity and the appropriate representation of all demographic groups. This study aims to examine diversity-related benefits of the use of situational judgement tests (SJTs) in the UK Clinical Aptitude Test (UKCAT) in terms of three demographic variables: (i) socio-economic status (SES); (ii) ethnicity, and (iii) gender.
Does a selection interview predict year 1 performance in dental school?
European Journal of Dental Education 2016, R. McAndrew, J. Ellis and R. A. Valentine
This study was undertaken in order to determine whether different interview methods (Cardiff with a multiple mini interview and Newcastle with a more traditional interview process) along with other components used in selection predicted academic performance in students.
A Cross-sectional Study of Current Doctors’ Performance in a Modified Version of a Medical School Admission Aptitude Test
Medicine (Baltimore) 2016 95:18, James P. Blackmur, Nazir I. Lone, Oliver D. Stone, David J. Webb, and Neeraj Dhaun
This study aimed to compare the performance of senior doctors in primary and secondary care and across a range of specialties, in a modified version of the medical school entrance examination—the mUKCAT. Lay people were also included in the study. Despite its widespread use, this is the first study that examines the performance of senior clinicians in the UKCAT.
UKCAT SJT: a study to explore validation methodology and early findings
Fiona Patterson, Stuart Martin – download report
The paper summarises the initial evaluation of the SJT undertaken by the Work Psychology Group in 2014. The evaluation of the methodology and initial outcomes informed the development of a further study which took place the following year. Results from the second study are being prepared for publication.
Fair access to medicine? Retrospective analysis of UK medical schools application data 2009-2012 using three measures of socioeconomic status
BMC Medical Education 2016 16:11 Kathryn Steven, Jon Dowell, Cathy Jackson and Bruce Guthrie
This study used UKCAT data to examine variation in socioeconomic status (SES) of applicants to study medicine and applicants with an accepted offer from a medical school. The authors conclude that regardless of the measure, those from less affluent backgrounds are less likely to apply and less likely to gain an accepted offer to study medicine. Previously unreported variation between UK countries and between medical schools warrants further investigation.
The UKCAT research panel commissioned item-level analyses of both the cognitive subtests and the situational judgement tests. These two reports are published to provide additional information regarding the 'measurement model' of the UKCAT for researchers.
Understanding the Dimensionality and Reliability of the Cognitive Scales of the UK Clinical Aptitude test (UKCAT): Summary Version of the Report
Dr Paul Tiffin, December 2013 – download report
Understanding the measurement model of the UKCAT Situational Judgement Test: Summary Report
Paul Tiffin & Madeline Carter, May 2015 – download report
Predictive value of the admissions process and the UK Clinical Aptitude Test in a graduate-entry dental school
British Dental Journal 2015, 218: 687-689 J. I. Foley and K. Hijazi
The study aims to assess the association between admissions performance and the UKCAT, and subsequent achievement within a graduate-entry dental school. It was conducted at the University of Aberdeen Dental School between 2010 and 2014. Student demographics, pre-admission scores (PAS), Universities and Colleges Admissions Service (UCAS) tariffs, multiple mini-interview (MMI) grades, UKCAT scores and percentiles were correlated with academic performance reported as the University Common Assessment Scale (0–20). This study suggests that student age, candidate performance at MMI and the UKCAT might be a predictor of academic achievement for graduate-entry dental students.
Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study
Medical Teacher 2015, 1-13 Jane Adam, Miles Bore, Roy Childs, Jason Dunn, Jean Mckendree, Don Munro, and David Powis
The study followed a complete intake of students throughout medical school to determine whether particular variables at the point of selection might distinguish those more likely to become satisfactory professional doctors. HYMS academic score, some UKCAT subtest scores and the total UKCAT score, and some non-cognitive tests completed at the outset of studies, together predicted outcomes most comprehensively. Tutor evaluation of students early in the course also identified the more and less successful students in the three domains of academic, clinical and professional performance. These results may be helpful in informing the future development of selection tools.
Predictive power of UKCAT and other pre-admission measures for performance in a medical school in Glasgow: a cohort study
BMC Medical Education 2014, 14:116 Nana Sartania, John D McClure, Helen Sweeting and Allison Browitt
Results from the study shows that UKCAT has a modest predictive power for overall course performance at the University of Glasgow Medical School over and above that of school science achievements or pre-admission interview score. The researchers go on to conclude that UKCAT is the most useful predictor of final ranking.
Predictive validity of the UK clinical aptitude test in the final years of medical school: a prospective cohort study
BMC Medical Education 2014, 14:88 Adrian Husbands, Alistair Mathieson, Jonathan Dowell, Jennifer Cleland, Rhoda MacKenzie
The study concludes that neither UCAS form nor interview scores were statistically significant predictors of examination performance but that conversely, the UKCAT yielded statistically significant validity coefficients in four of five assessments investigated. Multiple regression analysis showed the UKCAT made a statistically significant unique contribution to variance in examination performance in the senior years.
Comparison of the sensitivity of the UKCAT and A Levels to sociodemographic characteristics: a national study
BMC Medical Education 2014, 14:7 Paul A Tiffin, John C McLachlan, Lisa Webster and Sandra Nicholson
Researchers concluded that most of the sociodemographic factors that predict A level attainment also predict UKCAT performance. However, compared to A levels, males and those speaking English as a first language perform better on UKCAT. The findings suggest that UKCAT scores may be more influenced by sex and less sensitive to school type compared to A levels. These factors must be considered by institutions utilising the UKCAT as a component of the medical and dental school selection process.
The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of twelve UK medical schools
BMC Medicine 2013, 11:244 IC McManus, Chris Dewberry, Sandra Nicholson and Jonathan S Dowell
This study looked at 4,811 students in 12 UK medical schools who took the UKCAT from 2006 to 2008.
Researchers concluded that UKCAT scores and educational attainment measures were significant predictors of outcome. The incremental validity of UKCAT taking educational attainment into account was significant, but small. Medical school performance was also affected by a contextual measure of secondary schooling, students from secondary schools with greater average attainment at A-level performing less well.
This collaborative study in 12 medical schools shows the power of large-scale studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training.
Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: Meta-regression of six UK longitudinal studies
BMC Medicine 2013, 11:243 I C McManus, Chris Dewberry, Sandra Nicholson, Jonathan S Dowell, Katherine Woolf and Henry WW Potts
Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register.
Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance. The researchers suggest that some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills.
Forecasting the impact of the mode of use of the UKCAT on medical and dental school entrant demographics
Paul Tiffin, Lisa Webster & John McLachlan October 2013
The results of previous analyses suggest that the manner in which the UKCAT is used in the admissions process may influence the demographic characteristics of entrants. These findings were used to develop forecasts of what may happen to the demographics of the medical and dental students if medical and dental schools were to change their usage style of the test. Results suggest that if consortium medical schools currently using a ‘borderline’ or ‘factor’ approach to the UKCAT scores switched to using the test results as a ‘threshold’ for interview or offer there would be modest increases (roughly 5-7%) in the proportion of males admitted to ‘standard entry’ medical courses. In addition, if medical schools currently using a ‘factor’ approach to the UKCAT scores changed to a ‘threshold’ approach there may be an appreciable increase in the proportion of ex-state school students admitted to ‘standard entry’ medical courses (roughly in the order of 10%). If consortium dental schools currently using a ‘factor’ approach to the UKCAT scores switched to a ‘threshold’ approach then this may result in an appreciable increase in the proportion of ex-state school students admitted to ‘standard entry’ dental courses, roughly in the order of 10%.
The UKCAT-12 Study Technical Report Educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in twelve UK medical schools
IC McManus, C Dewberry, S Nicholson, J Dowell
The following Technical Report was previously submitted to the UKCAT Consortium to inform the Consortium of progress in analyzing the UKCAT-12 datasets. The full report is now being released, unchanged, to coincide with and to supplement the publication in BMC Medicine of a group of three papers, two of which analyse the UKCAT-12 data in more detail. The Report provides more details on some of the analyses and may be of use to those interested in UKCAT. It should be emphasized that the Report was a working document, that none of the conclusions were final, and indeed some of the conclusions and the methods on which they were based have been updated, in some cases as a result of minor errors having been found. The reader should therefore interpret the findings in that context and in conjunction with the three BMC Medicine papers.
Socioeconomic status of applicants to UKCAT Consortium Medical Schools 2009 – 2012
Published on the Medical Schools Council website May 2013 Lynda Cochrane, David Ridley, Bruce Guthrie, Jon Dowell
This analysis examines the socioeconomic status of applicants to the 26 medical schools in the UKCAT consortium. It assesses the impact of increased fees on the socioeconomic profile of applicants to medicine. It was produced by the Health Informatics Centre, University of Dundee, on behalf of the Medical Schools Council.
The UK clinical aptitude test and clinical course performance at Nottingham: a prospective cohort study
BMC Medical Education 2013, 13:32 Janet Yates and David James
The study built on earlier work, investigating whether 2006 UKCAT results predicted perfomance in the clinical components of the medical programme at Nottingham. The authors report that verbal reasoning and the UKCAT total score show modest correlation with clinical course marks. The authors go on to conclude that these are significantly less strong than the predictive ability of performance earlier in the programme.
Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study
BMC Medical Education 2012, 12:69 Jane Adam, Miles Bore, Jean McKendree, Don Munro and David Powis
This study shows numerous significant relationships between both cognitive and non-cognitive test scores, academic examination scores and indicators of professional behaviours in medical students. This suggests that measurement of non-cognitive personal qualities in applicants to medical school could make a useful contribution to selection and admission decisions. Further research is required in larger representative groups, and with more refined predictor measures and behavioural assessment methods, to establish beyond doubt the incremental validity of such measures over conventional cognitive assessments.
The UK Clinical Aptitude Test: Is it a fair test for selecting medical students?
Medical Teacher posted on-line May 2012 Paul Lambe, Catherine Waters, David Bristow
The aim of this study was to determine if differences in: access to support and advice, in modes of preparation, type of school/college attended, level of achievement in mathematics, gender and age influence candidate performance in the UKCAT and unfairly advantage some candidates over others. The study was carried out using questionnaires of applicants to study on an undergraduate medical degree course. Researchers found that differentials in access to support and advice, in modes of preparation, type of school/college attended, in level of achievement in mathematics, gender and age were found to be associated with candidate performance in the UKCAT. The findings imply that the UKCAT may disadvantage some candidate groups and that this inequity would likely be improved if tutors and career advisors in schools and colleges were more informed about the UKCAT and able to offer appropriate advice on preparation for the test.
Widening access to UK medical education for under-represented socioeconomic groups: modelling the impact of the UKCAT in the 2009 cohort
BMJ 2012;344:e1805 Paul A Tiffin, Jonathan S Dowell, John C McLachlan
The authors set out to determine whether the use of the UK clinical aptitude test (UKCAT) in the medical schools admissions process reduces the relative disadvantage encountered by certain sociodemographic groups. They were able to analyse detailed candidate and admissions data and look at how and how strongly different medical schools used the test. They concluded that the use of the UKCAT may lead to more equitable provision of offers to those applying to medical school from under-represented sociodemographic groups. This may translate into higher numbers of some, but not all, relatively disadvantaged students entering the UK medical profession.
Can the UKCAT select suitable candidates for interview?
Medical Education 2011: 45: 1041–1047 Rebecca Turner, Sandra Nicholson
This paper examines current selection practices and questions the role that the UKCAT may take, focussing in particular on whether UKCAT can select suitable candidates for interview. The rejection rate before interview of candidates with low UKCAT scores was 2.7 times that of candidates with high UKCAT scores. However, no relationship between overall UKCAT score and overall interview score existed within a pre-selected cohort of applicants with high UKCAT scores. The paper concluded that the UKCAT can facilitate the independent selection of appropriate candidates for interview but that it is not predictive of success at interview.
A mixed-methods study identifying and exploring medical students’ views of the UKCAT
Medical Teacher 2011 Vol. 33, No. 3 , Pages 244-249 J.A. Cleland, F.H. French, P.W. Johnston
This was a mixed-methods study using a paper-based survey and focus groups with first year medical students in Scotland in 2009–2010. Questionnaire data were analysed using SPSS, focus group data using framework analysis. The UKCAT was viewed unfavourably by first year medical students completing it pre-admission. These negative views seem due to concern as to the use of UKCAT data, and the fairness of the test. The authors conclude that more evidence as to validity and fairness of the UKCAT and how it is used in practice is required.
Predictive validity of the personal qualities assessment for selection of medical students in Scotland
Medical Teacher 2011, Volume 33 Jon Dowell, Mary Ann Lumsden, David Powis, Don Munro, Miles Bore, Biokanyo Makubate, Ben Kumwenda
The Personal Qualities Assessment (PQA) has been used within the non-cognitive sub-test of the UKCAT, having been developed to enhance medical student selection by measuring a range of non-cognitive attributes in applicants to medical school. This study pre-dates the introduction of the UKCAT with applicants to the five Scottish medical schools piloting the PQA in 2001 and 2002. The study aimed to evaluate the predictive validity of PQA through a longitudinal cohort study in which PQA scores were compared with senior year medical school performance. The most significant finding was that students identified by PQA as ‘not extreme’ on the two personal characteristics scales performed better in an OSCE measure of clinical performance.
Use of UKCAT scores in student selection by UK medical schools, 2006-2010
BMC Medical Education 2011, 11:98 Jane Adam, Jon Dowell, Rachel Greatrix
This paper summarises the way medical schools have used the test since its inception in 2006, categorises that use and demonstrates trends in use over time. Annual telephone interviews were conducted with UKCAT Consortium medical schools. The paper reports that four ways of using the test results have emerged (Borderline, Factor, Threshold and Rescue methods) with many schools using more than one method. The considerable variation in how medical schools use UKCAT supports the wish of the Consortium that applicants are clearly informed about how the test will be used in order that they can make best use of their limited number of applications.
Comparison of A-level and UKCAT performance in students applying to UK medical and dental schools in 2006: cohort study
BMJ 2010;340:bmj.c478 David James, Janet Yates, Sandra Nicholson
The object of this study was to determine whether the UK Clinical Aptitude Test (UKCAT) adds value to medical and dental school, and in particular whether UKCAT can reduce the socioeconomic bias known to affect A levels. The study suggests that the UKCAT has an inherent favourable bias to men and students from a higher socioeconomic class or independent or grammar schools. At the same time however, it does provide a reasonable proxy for A levels in the selection process.
The value of the UKCAT in predicting pre-clinical performance: a prospective cohort study at Nottingham medical school
BMC Medical Education 2010 10;55 Janet Yates, David James
The study aimed to determine whether UKCAT scores predict performance during the first two years of the 5-year undergraduate medical course at Nottingham. This limited study suggested that the predictive value of the UKCAT, particularly the total score, is low although section scores may predict success in specific types of course assessment.
Has the UK Clinical Aptitude Test improved medical student selection?
Medical Education 2010: 44: 1069 – 1076 Sarah R Wright, Philip M Bradley
The study aimed to determine whether the UKCAT had made any improvements to the way medical students are selected. The analysis studied the ability of previous school type and gender to predict UKCAT, personal statement or interview scores in two cohorts of accepted students. The results demonstrated that previous school type was not a significant predictor of either interview or UKCAT scores amongst students who had been accepted onto the programme. UKCAT scores were significant predictors of knowledge examination performance for all but one examination administered in the first two years of Newcastle Medical School whereas interview scores are not.
A comparison of the UKCAT with a traditional admission selection process
Informahealthcare 2009, Vol. 31, No. 11 , Pages 1018-1023 Nishan Fernando, Jennifer Clelland, Kathryn Greaves, Hamish Mckenzie
The aim of this study was to compare candidate UKCAT performance with medical student selection outcomes at the University of Aberdeen. Results from this study indicate that UKCAT scores show weak correlation with success in Aberdeen’s medical admissions process indicating that the UKCAT examines different traits.
Does the UKCAT predict Year 1 performance in medical school?
Medical Education 2009: 43: 1203 – 1209 Bonnie Lynch, Rhoda MacKenzie, Jon Dowell, Jennifer Clelland, Gordon Prescott
This study set out to identify whether UKCAT total score and subtest scores predict Year 1 outcomes at the Universities of Aberdeen and Dundee. UKCAT scores did not predict Year 1 performance at the two medical schools suggesting that studies of UKCAT’s ability to predict outcomes in later years of medical school and in subsequent qualification and practice as a doctor are indispensable in determining whether the use of the test as a selection tool is justified.