Authors: Sugeet Jagpal, MD; Leon Melnitsky, DO; Sam Zidovetzki, MD, MPH
Editor: Anne Messman, MD

A Case

Irena, a junior faculty member at a medical school, is asked to consult on the development of a new medical school in her state. In particular, she is asked to comment on what should be taken into account in the admissions process. Irena remembers feeling quite confused by the components of the application process, and wonders if she could possibly suggest new angles that would make it easier for future applicants to apply to medical school. Excited about the prospect of being able to think through this de novo, Irena accepts the consulting offer. 

As she reflects, negative memories of the application process come flooding back. She recalls having a letter of recommendation uploaded late, and the sheer panic that she felt at the time. She wondered back then, and still wonders to this day, if that missing letter of recommendation would have cost her her medical career. She also remembers working for days on her personal statement, only to find time and time again that interviewers had not taken the time to read it. One interviewer even told her that he did not look at personal statements prior to meeting the applicant – he trusted his ability to assess the applicants himself. He mentioned that in most cases he did not trust their ability to self report.

In addition, Irena was a biology major in college, and seriously thought about adding a minor in the liberal arts to make her application more well rounded for medical school. The added credits would have cost her additional money, and she chose against this as a preparatory course for the MCAT had already depleted her bank account. She ended up doing quite well on the MCAT, but would the money have been better spent on the minor in liberal arts? Would either of those have predicted success in medical school or a medical career? 

Irena sincerely hopes that she will be able to suggest an admissions process that will allow medical schools to select excellent candidates AND prevent unnecessary angst in thousands of college students. She enthusiastically dives into the data behind the admissions process. She is pleasantly surprised to learn that this is a debated topic, and her concerns have been echoed by others. She zeroes in on Siu and Reiter’s TAU approach, drawn to their separation of the cognitive and non-cognitive domains of the admissions process.

Background

In the highly competitive market of medical school applicants, it is essential for the admissions committee to use reliable and valid selection tools. Most applicants to medical school are highly motivated individuals who will stop at very little in order to gain acceptance. Currently there are multiple tools available at the disposal of the medical school faculty. Applicants’ grade point average, Medical College Admissions Test score and the multiple mini-interview are among the most practically relevant and statistically significant in correlation to medical school performance.

The TAU perspective recommends having little faith in the validity of the current application process. In particular the “Trust no-one” approach is suggested when it comes to evaluating an applicant’s references, file reviewers or interviewers.

The applicants’ inability to objectively self report is highlighted in this paper. It is unreasonable to expect medical school candidates under tremendous pressure to honestly and reliably self-assess on the application. Using repeated measures to judge the validity of the interview process is deemed highly valuable. For example, multiple interviews are recommended instead of the single interaction in order to effectively predict candidates’ potential performance.

Modern takes on this Theory

Letters of reference are a common part of an application, both for medical school and residency programs. Letters have been criticized as having poor predictive ability and poor inter-rater reliability.¹ Since the TAU approach was published, a trend toward standardized letters of evaluation (SLOEs) has emerged in many medical schools. The SLOE allows for a more objective measurement and is not just a personal narrative, which was the primary mode of assessment in previous letters of reference.⁴ In addition to the narrative portion, a typical SLOE will contain seven competency-focused questions which require the evaluator to rank the applicant in one of several tiers.⁴ This ranking is to be done by multiple evaluators relative to the applicants’ peers which brings the SLOE well within the TAU framework of repeated measurements.

Scholarly activity in the form of publications and presentations has also been cited as important in the application process. A survey of Emergency Medicine Residency Directors listed scholarly activity as being an important factor in assessing an applicant.⁵ Other specialties such as general surgery and orthopedics have also had scholarly activity mentioned as an important component of an application.⁶ Scholarly activity can be assessed in an objective way with different weight being given to the strength of the study, publication or presentation in which the applicant participated. If objective criteria are used, scholarly activity can be used quantitatively to add to the global assessment of the applicant. This quantitative measurement fits with the TAU framework by avoiding self report.

The TAU frameworks are becoming more important as an increasing number of applications to health professions are non-traditional, providing significant diversity in the components of the application. This has increased the difficulty of comparing applications relative to each other. Using the TAU approach for admissions allows admissions committees to appreciate the uniqueness of each individual application without compromising the validity of predicting future success. 

Other Examples of Where this Theory Might Apply

The TAU approach has far reaching applications to academic assessments beyond the application process itself. Clinical evaluations of residents and faculty are important indicators of performance and similarly should avoid self report. Classically Emergency Medicine resident assessment is done with one-on-one direct observation.⁷ This method of assessment is currently limited by evaluator bias and subjectivity. Although there are attempts to decrease this objectivity via interventions such as multiple choice testing or OSCEs, these are not repeated often enough in most training programs to be reliable. 

Increasingly, evaluations of residents are milestone based, providing a framework for all faculty to assess residents objectively. These competency based evaluations are completed on a more frequent basis, and compared to other residents within the same program and across other programs to give a more accurate assessment of a residents’ performance.⁷ This is more consistent with the TAU approach.

Early career faculty new to their role would also benefit from a TAU approach to assessment of their administrative skills, teaching skills, bedside diagnostic acumen, and other roles they have been assigned. Colleagues of new faculty, support staff, and trainees should assess early faculty in several standardized parameters including teaching ability and bedside clinical skills. Those in existing administrative leadership roles should also give feedback to early faculty on their leadership skills and organizational abilities, instead of relying on their self assessment of these abilities. The principle of using objective data as part of a TAU approach at assessment is applicable to many clinical and educational realms. 

Limitations of this Theory

The TAU approach has limitations as it allows for the possibility that future generations of applicants will go as far as modifying their applications solely in order to increase chances of admission thereby “cheating the system”. The “trust no one” theory can be interpreted as disrespectful and demeaning to the applicants and those that spend a significant amount of time writing required reference letters and various evaluations.

The TAU approach relies heavily on the MCAT as a predictor of success in medical school. It should be noted that in recent years, MCAT consulting and coaching has become increasingly popular. Those that can afford to pay $200/hour for a professional MCAT coach will have a higher chance at a better score, without having a higher chance of success in medical school itself. This also inherently increases the inequity of the admission process itself.

As far as mini interviews and use of “repeated measures,” there is a gray zone of uncertainty particularly when it comes to how defining how many mini interviews are required in order to conduct a reliable interview process. There is also no clear definition offered of the exact number of so called “repeated measures” regarding the number of interviewers that would satisfy criteria for multiple evaluations as suggested by the Siu and Reiter approach. 

Returning to the case…

After a deep dive into the TAU approach, Irena submits her report to the medical school admissions committee. She recommends that they focus on the candidates’ GPAs, MCAT, and also rely on multiple mini interviews. She also recommends a de-emphasis on subjective measures such as personal statements and letters of reference. Given the higher likelihood of a better matched candidate with repeated measurements, she recommends scheduling multiple mini interviews for each potential candidate. She does clearly state in her report that the committee should look at all subjective data with caution. 

She goes on to suggest an ideal interview process would involve initial screening of applications based on objective measurements such as GPA and MCAT scores. This would be followed by an interview day that involved multiple mini interviews, with only a brief review of personal statements and letters of recommendation as these were least likely to correlate with success. 

Irena strongly suggests that in order to be as objective as possible, the medical school should undertake a training process for their interviewers as well as the admissions committee. By having people meet and determine beforehand what to look for in a letter of reference, for example, there would be an opportunity to create repeatable measurements to create some objectivity in this data. Another example would be to determine how to quantify scholarly activity – this would allow the committee to speak more objectively about the candidates, in line with the TAU approach. She suggests a debriefing after the admissions process so that all of those involved in the process can give and receive feedback, allowing the interviewers and committee to function as a team and to become a reliable resource for each other in the future. 

In addition, she suggests that they track their enrolled students and adjust their admission screening process accordingly. By having an iterative process, she expects that they will be able to calibrate their admissions process with time and increase their satisfaction with their medical school class. In an ideal world, they would be able to track their admissions process to success in medical school and then success in the graduate medical education world, allowing them to fulfill their medical school’s mission. If a deficiency or need was identified, for example, in the residency milestones of their trainees they would be able to incorporate that both in their medical school curriculum as well as calibrate, if needed, their admissions process. This would be a very valuable resource for the medical school. 

The admissions committee is grateful for Irena’s report. They inform her that they will be asking for her help moving forward as they calibrate their process.

References

1. Siu, E. and Reiter, H. (2009). Overview: what’s worked and what hasn’t as a guide towards predictive admissions tool development. Advances in Health Sciences Education, 14(5), pp.759-775.
2. de Visser, M., Fluit, C., Cohen-Schotanus, J. and Laan, R. (2017). The effects of a non-cognitive versus cognitive admission procedure within cohorts in one medical school. Advances in Health Sciences Education, 23(1), pp.187-200.
3. de Visser, M., Fluit, C., Fransen, J., Latijnhouwers, M., Cohen-Schotanus, J. and Laan, R. (2016). The effect of curriculum sample selection for medical school. Advances in Health Sciences Education, 22(1), pp.43-56.
4. Jackson JS, Bond M, Love JN, Hegarty C. Emergency Medicine Standardized Letter of Evaluation (SLOE): Findings From the New Electronic SLOE Format. J Grad Med Educ. 2019 Apr;11(2):182–6.
5. Crane JT, Ferraro CM. Selection criteria for emergency medicine residency applicants. Acad Emerg Med. 2000 Jan;7(1):54–60.
6. Melendez MM, Xu X, Sexton TR, Shapiro MJ, Mohan EP. The importance of basic science and clinical research as a selection criterion for general surgery residency programs. J Surg Educ. 2008 Apr;65(2):151–4.
7. Colmers-Gray IN, Walsh K, Chan TM. Assessment of emergency medicine residents: a systematic review. Can Med Educ J [Internet]. 2017 Feb 24 [cited 2019 Jun 5];8(1):e106–22. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5344063