UKCAT SJT: a study to explore validation methodology and early findings
Fiona Patterson, Stuart Martin – download report
The paper summarises the initial evaluation of the SJT undertaken by the Work Psychology Group in 2014. The evaluation of the methodology and initial outcomes informed the development of a further study which took place the following year. Results from the second study are being prepared for publication.
Fair access to medicine? Retrospective analysis of UK medical schools application data 2009-2012 using three measures of socioeconomic status BMC Medical Education2016 16:11 Kathryn Steven, Jon Dowell, Cathy Jackson and Bruce Guthrie
This study used UKCAT data to examine variation in socioeconomic status (SES) of applicants to study medicine and applicants with an accepted offer from a medical school. The authors conclude that regardless of the measure, those from less affluent backgrounds are less likely to apply and less likely to gain an accepted offer to study medicine. Previously unreported variation between UK countries and between medical schools warrants further investigation.
The UKCAT research panel commissioned item-level analyses of both the cognitive subtests and the situational judgement tests. These two reports are published to provide additional information regarding the 'measurement model' of the UKCAT for researchers.
Understanding the Dimensionality and Reliability of the Cognitive Scales of the UK Clinical Aptitude test (UKCAT): Summary Version of the Report Dr Paul Tiffin, December 2013 – download report
Understanding the measurement model of the UKCAT Situational Judgment Test: Summary Report Paul Tiffin & Madeline Carter, May 2015 – download report
Predictive value of the admissions process and the UK Clinical Aptitude Test in a graduate-entry dental school. British Dental Journal 2015, 218: 687-689 J. I. Foley and K. Hijazi
The study aims to assess the association between admissions performance and the UKCAT, and subsequent achievement within a graduate-entry dental school. It was conducted at the University of Aberdeen Dental School between 2010 and 2014. Student demographics, pre-admission scores (PAS), Universities and Colleges Admissions Service (UCAS) tariffs, multiple mini-interview (MMI) grades, UKCAT scores and percentiles were correlated with academic performance reported as the University Common Assessment Scale (0–20). This study suggests that student age, candidate performance at MMI and the UKCAT might be a predictor of academic achievement for graduate-entry dental students.
Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study. Medical Teacher 2015, 1-13 Jane Adam, Miles Bore, Roy Childs, Jason Dunn, Jean Mckendree, Don Munro, and David Powis
The study followed a complete intake of students throughout medical school to determine whether particular variables at the point of selection might distinguish those more likely to become satisfactory professional doctors. HYMS academic score, some UKCAT subtest scores and the total UKCAT score, and some non-cognitive tests completed at the outset of studies, together predicted outcomes most comprehensively. Tutor evaluation of students early in the course also identified the more and less successful students in the three domains of academic, clinical and professional performance. These results may be helpful in informing the future development of selection tools.
Predictive power of UKCAT and other pre-admission measures for performance in a medical school in Glasgow: a cohort study BMC Medical Education 2014, 14:116 Nana Sartania, John D McClure, Helen Sweeting and Allison Browitt
Results from the study shows that UKCAT has a modest predictive power for overall course performance at the University of Glasgow Medical School over and above that of school science achievements or pre-admission interview score. The researchers go on to conclude that UKCAT is the most useful predictor of final ranking.
Predictive validity of the UK clinical aptitude test in the final years of medical school: a prospective cohort study BMC Medical Education 2014, 14:88 Adrian Husbands, Alistair Mathieson, Jonathan Dowell, Jennifer Cleland, Rhoda MacKenzie
The study concludes that neither UCAS form nor interview scores were statistically significant predictors of examination performance but that conversely, the UKCAT yielded statistically significant validity coefficients in four of five assessments investigated. Multiple regression analysis showed the UKCAT made a statistically significant unique contribution to variance in examination performance in the senior years.
Comparison of the sensitivity of the UKCAT and A Levels to sociodemographic characteristics: a national study BMC Medical Education 2014, 14:7 Paul A Tiffin, John C McLachlan, Lisa Webster and Sandra Nicholson
Researchers concluded that most of the sociodemographic factors that predict A level attainment also predict UKCAT performance. However, compared to A levels, males and those speaking English as a first language perform better on UKCAT. The findings suggest that UKCAT scores may be more influenced by sex and less sensitive to school type compared to A levels. These factors must be considered by institutions utilising the UKCAT as a component of the medical and dental school selection process.
The UKCAT-12 study: Educational attainment, aptitude test performance, demographic and socio economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of twelve UK medical schools BMC Medicine 2013, 11:244 IC McManus, Chris Dewberry, Sandra Nicholson and Jonathan S Dowell
This study looked at 4,811 students in 12 UK medical schools who took the UKCAT from 2006 to 2008.
Researchers concluded that UKCAT scores and educational attainment measures were significant predictors of outcome. The incremental validity of UKCAT taking educational attainment into account was significant, but small. Medical school performance was also affected by a contextual measure of secondary schooling, students from secondary schools with greater average attainment at A-level performing less well.
This collaborative study in 12 medical schools shows the power of large-scale studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training.
Construct-level predictive validity of educational attainment and intellectual aptitude tests in medical student selection: Meta-regression of six UK longitudinal studies BMC Medicine 2013, 11:243 I C McManus, Chris Dewberry, Sandra Nicholson, Jonathan S Dowell, Katherine Woolf and Henry WW Potts
Construct-level predictive validities were calculated in six cohort studies of medical student selection and training (student entry, 1972 to 2009) for a range of predictors, including A-levels, General Certificates of Secondary Education (GCSEs)/O-levels, and aptitude tests (AH5 and UK Clinical Aptitude Test (UKCAT)). Outcomes included undergraduate basic medical science and finals assessments, as well as postgraduate measures of Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) performance and entry in the Specialist Register.
Educational attainment has strong CLPVs for undergraduate and postgraduate performance, accounting for perhaps 65% of true variance in first year performance. Such CLPVs justify the use of educational attainment measure in selection, but also raise a key theoretical question concerning the remaining 35% of variance. The researchers suggest that some variance probably relates to factors which are unpredictable at selection, such as illness or other life events, but some is probably also associated with factors such as personality, motivation or study skills.
Forecasting the impact of the mode of use of the UKCAT on medical and dental school entrant demographics Paul Tiffin, Lisa Webster & John McLachlan October 2013
The results of previous analyses suggest that the manner in which the UKCAT is used in the admissions process may influence the demographic characteristics of entrants. These findings were used to develop forecasts of what may happen to the demographics of the medical and dental students if medical and dental schools were to change their usage style of the test. Results suggest that if consortium medical schools currently using a ‘borderline’ or ‘factor’ approach to the UKCAT scores switched to using the test results as a ‘threshold’ for interview or offer there would be modest increases (roughly 5-7%) in the proportion of males admitted to ‘standard entry’ medical courses. In addition, if medical schools currently using a ‘factor’ approach to the UKCAT scores changed to a ‘threshold’ approach there may be an appreciable increase in the proportion of ex-state school students admitted to ‘standard entry’ medical courses (roughly in the order of 10%). If consortium dental schools currently using a ‘factor’ approach to the UKCAT scores switched to a ‘threshold’ approach then this may result in an appreciable increase in the proportion of ex-state school students admitted to ‘standard entry’ dental courses, roughly in the order of 10%.
Download the medicine report
Download the dental report
The UKCAT-12 Study Technical Report Educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in twelve UK medical schools. IC McManus, C Dewberry, S Nicholson, J Dowell
The following Technical Report was previously submitted to the UKCAT Consortium to inform the Consortium of progress in analyzing the UKCAT-12 datasets. The full report is now being released, unchanged, to coincide with and to supplement the publication in BMC Medicine of a group of three papers, two of which analyse the UKCAT-12 data in more detail. The Report provides more details on some of the analyses and may be of use to those interested in UKCAT. It should be emphasized that the Report was a working document, that none of the conclusions were final, and indeed some of the conclusions and the methods on which they were based have been updated, in some cases as a result of minor errors having been found. The reader should therefore interpret the findings in that context and in conjunction with the three BMC Medicine papers.
Download the Technical Report
Socioeconomic status of applicants to UKCAT Consortium Medical Schools 2009 – 2012. Published on the Medical Schools Council website May 2013 Lynda Cochrane, David Ridley, Bruce Guthrie, Jon Dowell
This analysis examines the socioeconomic status of applicants to the 26 medical schools in the UKCAT consortium. It assesses the impact of increased fees on the socioeconomic profile of applicants to medicine. It was produced by the Health Informatics Centre, University of Dundee, on behalf of the Medical Schools Council.
The UK clinical aptitude test and clinical course performance at Nottingham: a prospective cohort study BMC Medical Education 2013, 13:32 Janet Yates and David James
The study built on earlier work, investigating whether 2006 UKCAT results predicted perfomance in the clinical components of the medical programme at Nottingham. The authors report that verbal reasoning and the UKCAT total score show modest correlation with clinical course marks. The authors go on to conclude that these are significantly less strong than the predictive ability of performance earlier in the programme.
Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study BMC Medical Education 2012, 12:69 Jane Adam, Miles Bore, Jean McKendree, Don Munro and David Powis
This study shows numerous significant relationships between both cognitive and non-cognitive test scores, academic examination scores and indicators of professional behaviours in medical students. This suggests that measurement of non-cognitive personal qualities in applicants to medical school could make a useful contribution to selection and admission decisions. Further research is required in larger representative groups, and with more refined predictor measures and behavioural assessment methods, to establish beyond doubt the incremental validity of such measures over conventional cognitive assessments.
The UK Clinical Aptitude Test: Is it a fair test for selecting medical students? Medical Teacher posted on-line May 2012 Paul Lambe, Catherine Waters, David Bristow
The aim of this study was to determine if differences in: access to support and advice, in modes of preparation, type of school/college attended, level of achievement in mathematics, gender and age influence candidate performance in the UKCAT and unfairly advantage some candidates over others. The study was carried out using questionnaires of applicants to study on an undergraduate medical degree course. Researchers found that differentials in access to support and advice, in modes of preparation, type of school/college attended, in level of achievement in mathematics, gender and age were found to be associated with candidate performance in the UKCAT. The findings imply that the UKCAT may disadvantage some candidate groups and that this inequity would likely be improved if tutors and career advisors in schools and colleges were more informed about the UKCAT and able to offer appropriate advice on preparation for the test.
Widening access to UK medical education for under-represented socioeconomic groups: modelling the impact of the UKCAT in the 2009 cohort BMJ 2012;344:e1805 Paul A Tiffin, Jonathan S Dowell, John C McLachlan
The authors set out to determine whether the use of the UK clinical aptitude test (UKCAT) in the medical schools admissions process reduces the relative disadvantage encountered by certain sociodemographic groups. They were able to analyse detailed candidate and admissions data and look at how and how strongly different medical schools used the test. They concluded that the use of the UKCAT may lead to more equitable provision of offers to those applying to medical school from under-represented sociodemographic groups. This may translate into higher numbers of some, but not all, relatively disadvantaged students entering the UK medical profession.
Can the UKCAT select suitable candidates for interview? Medical Education 2011: 45: 1041–1047 Rebecca Turner, Sandra Nicholson
This paper examines current selection practices and questions the role that the UKCAT may take, focussing in particular on whether UKCAT can select suitable candidates for interview. The rejection rate before interview of candidates with low UKCAT scores was 2.7 times that of candidates with high UKCAT scores. However, no relationship between overall UKCAT score and overall interview score existed within a pre-selected cohort of applicants with high UKCAT scores. The paper concluded that the UKCAT can facilitate the independent selection of appropriate candidates for interview but that it is not predictive of success at interview.
A mixed-methods study identifying and exploring medical students’ views of the UKCAT Medical Teacher 2011 Vol. 33, No. 3 , Pages 244-249 J.A. Cleland, F.H. French, P.W. Johnston
This was a mixed-methods study using a paper-based survey and focus groups with first year medical students in Scotland in 2009–2010. Questionnaire data were analysed using SPSS, focus group data using framework analysis. The UKCAT was viewed unfavourably by first year medical students completing it pre-admission. These negative views seem due to concern as to the use of UKCAT data, and the fairness of the test. The authors conclude that more evidence as to validity and fairness of the UKCAT and how it is used in practice is required.
Predictive validity of the personal qualities assessment for selection of medical students in Scotland. Medical Teacher 2011, Volume 33 Jon Dowell, Mary Ann Lumsden, David Powis, Don Munro, Miles Bore, Biokanyo Makubate, Ben Kumwenda
The Personal Qualities Assessment (PQA) has been used within the non-cognitive sub-test of the UKCAT, having been developed to enhance medical student selection by measuring a range of non-cognitive attributes in applicants to medical school. This study pre-dates the introduction of the UKCAT with applicants to the five Scottish medical schools piloting the PQA in 2001 and 2002. The study aimed to evaluate the predictive validity of PQA through a longitudinal cohort study in which PQA scores were compared with senior year medical school performance. The most significant finding was that students identified by PQA as ‘not extreme’ on the two personal characteristics scales performed better in an OSCE measure of clinical performance.
Use of UKCAT scores in student selection by UK medical schools, 2006-2010 BMC Medical Education 2011, 11:98 Jane Adam, Jon Dowell, Rachel Greatrix
This paper summarises the way medical schools have used the test since its inception in 2006, categorises that use and demonstrates trends in use over time. Annual telephone interviews were conducted with UKCAT Consortium medical schools. The paper reports that four ways of using the test results have emerged (Borderline, Factor, Threshold and Rescue methods) with many schools using more than one method. The considerable variation in how medical schools use UKCAT supports the wish of the Consortium that applicants are clearly informed about how the test will be used in order that they can make best use of their limited number of applications.
Comparison of A-level and UKCAT performance in students applying to UK medical and dental schools in 2006: cohort study BMJ 2010;340:bmj.c478 David James, Janet Yates, Sandra Nicholson
The object of this study was to determine whether the UK Clinical Aptitude Test (UKCAT) adds value to medical and dental school, and in particular whether UKCAT can reduce the socioeconomic bias known to affect A levels. The study suggests that the UKCAT has an inherent favourable bias to men and students from a higher socioeconomic class or independent or grammar schools. At the same time however, it does provide a reasonable proxy for A levels in the selection process.
The value of the UKCAT in predicting pre-clinical performance: a prospective cohort study at Nottingham medical school. BMC Medical Education 2010 10;55 Janet Yates, David James
The study aimed to determine whether UKCAT scores predict performance during the first two years of the 5-year undergraduate medical course at Nottingham. This limited study suggested that the predictive value of the UKCAT, particularly the total score, is low although section scores may predict success in specific types of course assessment.
Has the UK Clinical Aptitude Test improved medical student selection? Medical Education 2010: 44: 1069 – 1076 Sarah R Wright, Philip M Bradley
The study aimed to determine whether the UKCAT had made any improvements to the way medical students are selected. The analysis studied the ability of previous school type and gender to predict UKCAT, personal statement or interview scores in two cohorts of accepted students. The results demonstrated that previous school type was not a significant predictor of either interview or UKCAT scores amongst students who had been accepted onto the programme. UKCAT scores were significant predictors of knowledge examination performance for all but one examination administered in the first two years of Newcastle Medical School whereas interview scores are not.
A comparison of the UKCAT with a traditional admission selection process. Informahealthcare 2009, Vol. 31, No. 11 , Pages 1018-1023 Nishan Fernando, Jennifer Clelland, Kathryn Greaves, Hamish Mckenzie
The aim of this study was to compare candidate UKCAT performance with medical student selection outcomes at the University of Aberdeen. Results from this study indicate that UKCAT scores show weak correlation with success in Aberdeen’s medical admissions process indicating that the UKCAT examines different traits.
Does the UKCAT predict Year 1 performance in medical school? Medical Education 2009: 43: 1203 – 1209 Bonnie Lynch, Rhoda MacKenzie, Jon Dowell, Jennifer Clelland, Gordon Prescott
This study set out to identify whether UKCAT total score and subtest scores predict Year 1 outcomes at the Universities of Aberdeen and Dundee. UKCAT scores did not predict Year 1 performance at the two medical schools suggesting that studies of UKCAT’s ability to predict outcomes in later years of medical school and in subsequent qualification and practice as a doctor are indispensable in determining whether the use of the test as a selection tool is justified.