#admissionchat episode 11 welcomes Dr. Keith Wright, VP, psychometrics and assessments at the Enrollment Management Association (EMA). Dr. Wright answers the tough questions regarding fairness and bias in standardized tests such as the SSAT.
Questions discussed include:
- Why are standardized tests such as the SSAT a necessary component in a fair and equitable applicant review process?
- Are grades and GPAs a viable alternative to standardized testing for admission candidate review?
- How are SSAT questions created? What steps does EMA take to ensure the questions are fair for all test takers?
- What are some of the things EMA does to increase test preparation equity from an economic standpoint?
- What about critiques that standardized tests only measure cognitive abilities and don't factor other components of future success such as work ethic and teamwork?
Listen to the episode above, ask your smart speaker to play #admissionchat, or subscribe on your favorite podcast platform: Amazon Music, Apple Podcasts, Google Podcasts, Pandora, Spotify.
Transcript
Daren Worcester: Welcome to #admissionchat. Let's talk about fairness and standardized testing.
Greetings. I'm Daren Worcester, and in this episode, I'm joined by Dr. Keith Wright, VP of psychometrics and assessments at the Enrollment Management Association who leads the development of SSAT testing. Keith, thank you for joining me today and welcome to the podcast.
Keith Wright: Thank you, Daren. Looking forward to the discussion.
Daren Worcester: Excellent. Let's start off by having you tell us a little about yourself and your career and assessments and psychometrics for EMA.
Keith Wright: That's a great question, Daren. And when I think back to my career, I will put my career into three buckets. I started off in electronic engineering and software development as a software design engineer. Then I fell in love with just being back in higher ed. I started teaching engineering courses, and so I transitioned from being in the technology field and going back into academics again, teaching engineering courses. While in academics I said to myself, if this is a place where I want to stay, I probably need to get a PhD degree. The question was, I had been out of my master's of science and computer science graduate degree for about, I think at that point, maybe eight years. So the thought of doing a doctoral degree in computer science wasn't that appealing just because I knew it would be very challenging. The graduate degree in and of itself was challenging, and so I said, what do I love? So I love education, I love math and computer science, and that's an excellent blend for psychometrics, and that's what I do at EMA. That was the doctoral degree, and that's how I ended up at EMA from a psychometrics assessment perspective.
Daren Worcester: That's interesting. I actually really was wondering how it's such a specified niche, how you got down that road. So for our discussion today, we're talking about fairness and standardized testing, specifically the SSAT. And as we know, standardized tests such as the SSAT are part of the candidate review process for many schools. We're very quick to say they aren't the end-all and be-all for decision-making, but they are a piece of the admissions pie that helps schools identify whether students will be a good fit for their programs. With all your time and experience here, why do you feel the SSAT is a necessary component in a fair and equitable applicant review process?
Keith Wright: Another great question. So I'm going to have to go back to the Ph.D. degree because I would not consider me being at EMA very specific decision that I made. It happened haphazardly. And what I mean by that is during my doctoral degree and psychometrics, I was very, very interested in the area of standardized testing or assessment within education. I'm a lifelong learner. I think education, of course, is extremely important for many reasons. Most importantly, it helps you. It can definitely change the trajectory of your life and those outcomes. I really wanted to have a better understanding of whether or not standardized testing, whether or not they were fair. Being from the city of Chicago, the south side of Chicago where I attended Chicago public schools, I really wasn't sure. I didn't really have a good answer to that question on whether or not testing, standardized testing was fair.
And so my entire doctoral program was centered around trying to answer that question. If you know anything about a doctoral program, you look at a lot of literature, there's a lot of research, and it's very scientific in terms of approach in which you try to answer any research question. The methodologies that are used, my area, specific area, and my doctoral program were differential item functioning. That's a statistical technique that looks at whether or not test items differ for one group of kids versus another group. So you can group 'em by gender, ethnicity, so forth, and so on. And so after that, quite honestly, when I went into that research investigation, and you should never go into a research investigation with, say for instance, your mindset on the answer that you think you're going to get. Well, I did not get the answer that I thought I would get, and the answer was yes, standardized testing is fair.
And so when we speak of fairness in testing, well, it's not the test that's not fair. One thing I've learned being at EMA for the last—almost now 10 years—is the development process in which we use the rigorous process in terms of how we determine scores, the test administration of the test, and the score reporting. All of that is very, very rigorous, standardized, and fair. So a test in and of itself cannot not be fair. Maybe I can say that there's been an overuse and inappropriate use of test scores, but tests measure what they're supposed to measure. They're supposed to measure one's cognitive ability and math, verbal reasoning, and reading comprehension. And one thing, again, I can say that it's not the test that's not fair. The process is extremely fair.
Daren Worcester: Keith, you just surprised me there. I want to make sure I understood that correctly. So you actually approached this originally anticipating that you were going to find that tests weren't fair. Did I understand that correctly?
Keith Wright: 100% accurate. My entire research inquiry when I went into my doctoral program was to prove that standardized testing was not fair. So if I say my hypothesis wasn't a null hypothesis because I had already accepted what I would consider the alternative hypothesis, right? So a null hypothesis is when you're trying to answer the question—you don't make a determination yet until you determine what the research yields. And so of course it was a fair process that I undertook in terms of the research inquiry, but at the end of the day, I concluded, and the only reasonable thing based on objective research that I could conclude was the test is fair. It's not the test that's not fair. Maybe the human actors, maybe it's the human actors involved in how they go about looking at scores, but the test in and of itself is fair.
Daren Worcester: So would it be fair to say you being involved in this industry now is a can't beat 'em, join 'em approach?
Keith Wright: Absolutely. And my mission is to ensure that everything we do at EMA is to ensure fairness from A through Z related to everything we do around developing our assessments.
Daren Worcester: One outcome of the pandemic that we've seen is a shift to test-optional admissions for some schools. As a semi-objective observer from the outside, what has always puzzled me about this is how do they objectively evaluate a student's academic performance or cognitive abilities? When folks say we're shifting away, they often say, well, we've got grades, and we've got GPAs as alternatives to standardized testing. But with grades, the part that confuses me is you often have two teachers in the same school teaching the same course grading differently. One may offer tests while the other puts more emphasis on essays. One may offer extra credit while the other doesn't. So you can have two students of the same abilities coming out of that school with widely different grades in the same subject. So if grading can be different in the same school, just the thought of how widely they vary across the world just blows my mind. This is a very long-winded way to ask you what I think is probably a rhetorical question: Is there a realistic alternative out there to objectively evaluate a student's cognitive abilities?
Keith Wright: As you said yourself, how is it that you can have grades being used now or even grade point averages being used objectively? Can grades and grade point averages be a replacement over standardized testing? And when I think about it, is when you look at all the information that's in the student's application file, a lot of it is subjective. And so when we look at objectivity and what are the objective pieces of information in that student's application file, the only objective information would be anything that was standardized. And so when you think of grades and grade point averages, that's just not standardized. Standardized testing, and the reason we call it standardized is because it's standardized as related to the development of the test, the administration of the test, the scoring of the test, and the score reporting of the test—all those components are standardized, hence objective.
When you look at grades and to your point, just we can even take two students in the same classroom with the same teacher, and I can argue that the grades that one student gets over another student from that same teacher and that same classroom could be subjective because you have unfortunately a human actor involved. Now, I'm not saying there are some nefarious activities taking place with this teacher. No, I think we all come with our own biases—good or bad, even in how we assess students. I can say that because I've been in a classroom, I've been a teacher. So we try to have objectivity, but there are many factors that can happen throughout a semester that can create bias in your mind as a teacher with one student over another—good or bad. And so grades and grade point averages, Daren, as you just pointed out, they're not objective. They're very subjective, and they're not equal. And that's not just for me. Research has demonstrated that as well. And just take it even more extreme from what you said. You didn't even give an extreme example; just take me coming from Chicago public schools versus another student coming, say for instance from a school like Andover or Exeter. So you think that the courses that I took at my school are going to compare to the courses from Andover and Exeter? It's impossible.
Daren Worcester: That's a very good point, and I want to follow up on that one in a second. So hold onto that thought. Staying in the vein of the grading discrepancy, from what I've seen, I think it also creates confusion amongst parents and others when it comes to taking standardized testing. I've done the research, I've seen the chat board and the Facebook comments where parents have a child that does great in school, gets good grades, they're attentive by all measures, they're doing well, they're a good student, and nobody would say they're not a good student. But then they take the SSAT and they don't score as high as what the parent would expect them to score based on what they know of their child and their performance in school. The natural fallback for them, because they don't understand it, is to say, well, the SSAT must be biased.
We know the opposite is true. Your team goes to painstaking lengths, as you've mentioned already, to ensure questions are fair for all test takers. And I was kind of hoping we could use this time for you to walk us through the lifecycle of an SSAT question. Let's kind of let everybody understand what happens in this process, how are they created? What checks and balances do they go through? You mentioned you got into this trying to prove that tests weren't fair and now you're in it to make sure that they are. So what is it we're doing to ensure that the questions on the tests aren't biased in some way?
Keith Wright: Great question. It's going to take a while to go through that rigorous process. Parents, hopefully, you have time and hopefully the listeners can visualize what I'm trying to articulate. But as you said, it's a rigorous process. I would argue that it takes over a year to develop one test form for the SSAT. And so it's an iterative process, and so we're developing test items and test forms on a regular basis.
Daren Worcester: Can you define what a form is to make sure everybody understands that?
Keith Wright: So a test form is a complete SSAT form that consists of the 50 questions for the quantitative section, 40 questions for the reading section and the 60 verbal questions for the verbal section. So that's an entire SSAT form.
Daren Worcester: Okay, so basically it's the version of the test that you would be taking.
Keith Wright: Absolutely.
Daren Worcester: Awesome. Sorry to get you off base.
Keith Wright: No, that's important to help keep me honest with respect to some acronyms that I may be using just because I'm so familiar with these terms. And so again, it could take almost a year just to develop one form. Unlike me when I was teaching, I could develop the final exam in a week. So not to go back to your previous point with respect to grades and grade point averages, well, I could tell you I could develop a final exam for a course in a week, week and a half. That's not the case for, say, for instance, a well-developed standardized test or assessment. It takes a year because of the number of steps that are involved in that process. So first we have to start with content, right? So we have content experts, and we use the actual teachers and our independent private schools to develop our test questions.
The reason that is is because it's called content validity. So we would not want, say for instance, public school teachers developing test items for kids that are going to be in independent private school classrooms. They don't know the content, they don't know the difficulty level that will be expected, that will be expected. So we utilize teachers in the schools themselves to write the test questions. Now of course, these teachers are not experts in the test development process, which there's research and there's a methodology you go through even in developing, say for instance, standardized test items. So we have our staff, our expert test developers working with these teachers to develop the test items. Once these test items are developed, they have to determine the difficulty level of these test items. And so they have this subjective and somewhat subjective when they're determining the difficulty level because they're saying, okay, based on what I know I'm teaching in this class, I think this question is going to be at a medium difficulty level, or this is going to be a very hard question, say for instance for this seventh grader.
So then we actually tested in the field with the students, and that's called pre-testing. So before these test items even count for real scores, they get field-tested or pre-tested with students who are trying to answer these questions. So statistically, we get almost perfect precision as related to the difficulty level of these test items after say for instance, we do pre-testing, we get our item statistics on the difficulty level on these questions, and we also look to determine if these questions created any issues from how students may interpret it, the question. So questions get removed all the time if they don't pass our rigorous statistical process. And after they pass that pretesting process, then they go through another review with our test developers before they're even assembled on a test form where they are then administered for actual scores that are going to count.
Daren Worcester: Now, as I understand it, in those review processes, you guys are specifically looking for any potential for regional or cultural biases in the questions, is that correct?
Keith Wright: Yeah, but that's done before they're even field-tested. We even go through that rigorous process there and before they're even field-tested, because that's called our fairness review. So our staff, our expert test developers, they ensure all of that stuff has already been taken care of before they're even pre-tested and field-tested.
Daren Worcester: Okay. And the field testing is that the experimental section at the end of the SSAT that some students take? So families know your students take the SSAT, and sometimes—is it a 15-minute section at the end—where they're answering questions but they're not getting scored on them, and that's our pre-work to make sure those questions are going to be good.
Keith Wright: That's excellent. That's correct. And those questions appear towards the end of the SSAT, and we get excellent, excellent insight. Because these students take it very seriously as related to taking the SSAT, they put the same effort into answering those pre-test questions as they would the actual questions that are going to count. And so our statistics that we get back are excellent with respect to making a determination on the difficulty level of these questions because we often look at the pretest statistics and then when these same questions are then administered on a form where they're going to count, these statistics match almost perfectly.
Daren Worcester: Thank you. Thank you for explaining that and breaking it down. I know for me, coming into EMA, I was a student that didn't necessarily love standardized tests, so I didn't necessarily have the most positive opinion of it either. But understanding what they go through and everything that you guys put into it is definitely, for one, changed my impression of the whole piece
Keith Wright: And I could raise my hand and say, I'm with you on that, Daren, hence the whole Ph.D. investigation.
Daren Worcester: Yeah, that was a bombshell for me in this one. I didn't realize that's how you had gotten into it. I do feel like, Keith, that we can't objectively have this conversation and say that we're objectively having it without acknowledging the elephant in the standardized testing room. And you actually alluded to this earlier, and I tabled it, so let's go back to it. But that's the fact that the SSAT is an assessment for student readiness in what we know are rigorous private and independent school academic programs. As such, students who are already enrolled in these schools, maybe a middle school student testing up for high school or who have the means to hire a tutor, have an inherent advantage. So what are some of the things that we do at EMA to level the playing field from an economic standpoint?
Keith Wright: Another elephant in the room, let's acknowledge it's going to be very difficult for EMA to correct the ills of society wholeheartedly. But there are things that we do to try to assist traditionally underrepresented students. But before I even speak about what those things are, let's first address and not penalize students whose parents have the resources to provide those additional educational enrichment opportunities. That's what we should do as parents. And those students who get that additional enrichment opportunities, they're going to just be able to learn more and go to take in any type of assessment with a bit more confidence. And I look at, I think you used the term prep. I look at it as just think about even in K–12 public schools where they have after-school programs, well, that's the purpose of after-school programs because you typically cannot get everything that's needed and whatever the time period is for that day in school.
I've been out of school for a while, so I don't remember, but I know it was maybe from 8:30 a.m. until the bell was ringing at four o'clock. I don't remember. But you need to have things outside of the classroom to help you continue to move forward in your educational journey and to enhance your educational outcomes. I don't want to dismiss the importance of after-school programs. You call it test prep. I call it getting more instructional time with learning algebra and geometry and studying vocabulary words, increasing my vocabulary and my reading comprehension ability. So that in and of itself isn't a bad thing. It is just part of the academic program for those students. And so it's important. So what do we do to try to say, for instance, mitigate some of these societal ills out there? Because it's the societal factors that create the differences in standardized testing scores.
So let's recognize that. And that's why I think it's really important, and this discussion is very personal to me because I think we focus so much on the test and not on the societal factors that's creating the disparity. And so when we do that, we're not holding those who we should be holding accountable, such as our political leaders, our educators, even the families who we need to hold accountable to what needs to happen as well to ensure all students have an opportunity to be successful. But what we do at EMA, we have an excellent practice portal program where we have developed this practice portal to give students an opportunity to get exposure and familiar with the content. Say for instance, on the SSAT, the difficulty level of, say for instance, the content on the SSAT, often times for traditionally underrepresented students, they can access the practice portal for free, and they can use this to enhance some of their test-taking skills and just refresh, say for instance, their memory and get more knowledge about different topics and algebra and geometry, so forth and so on.
We have a fee waiver program where we allow students who qualify for the fee waiver program to be able to take the test for free. And I think very importantly since my time at EMA would be the education related to the use of standardized scores. And so I do a lot of speaking to schools to the admission professionals and their offices related to this discussion right here, Daren, right? So what is the standardized test score? How is it developed? What does this score mean? How should I interpret this score? Just helping say, for instance, admission professionals understand that there isn't really a huge difference, say for instance, an 80th percentile and a 90th percentile. I know families even probably put too much emphasis on those differences, whereas statistically, it's not really a great difference there. And so understanding how to interpret these test scores and use it holistically with the other information, those are all the things we do at EMA. Again, our practice portal fee waiver program, very important to me is the education related—say for instance—to how do you interpret and use these scores holistically in admission decisions.
Daren Worcester: You've hit on some very important topics there. I imagine how that is playing out. You use the example of large public school systems where maybe the resources and the opportunities aren't as great. Is that basically, and to use your percentile example, is that helping admissions folks understand that hey, a kid that scored an 80% in a public school system that's in a large city without the same resources, that is excellent and shows a lot of promise and it should be right up there with a student that had all these academic advantages that got a 90%, is that essentially what you're saying?
Keith Wright: Spot on. A hundred percent.
Daren Worcester: Excellent. So I do think people sometimes get a little carried away with looking at the percentages, talk to the school, and understand how they're interpreting that data and how they're using it, that they aren't necessarily just looking at it as a flat-line score. And that's all they look at.
Keith Wright: A lot of this discussion, Daren, I can just look at myself personally. So me, coming out of Chicago Public Schools, I was an excellent student honor roll student. I was in gifted architectural program. So coming out of high school, I thought I was in the top 3% of my class, but my score did not suggest that it should not have suggested that I was this top student because I didn't even know I had to take a standardized test to go to college and I was going to college. So there you go. I had two weeks to prepare for taking the ACT. And so how do you think I did with only two weeks of prep time when my counselor told me, you know you're going to college, Keith? And I was like, a hundred percent. She said, well, you know you have to take this test. I was like, what test?
So there you go. So my score did not reflect, say for instance, my ability, but I did need some extra opportunities in my first year to catch up because my school didn't have pre-calculus. And so that put me at a disadvantage wanting to move into engineering. So it just took me about a semester to catch up, but once I was able to catch up, I was on par with my other peers. But just think about it, if say for instance, I was not given out opportunity to go into additional courses, that could have really discouraged me if I was thrown right into the deep water without this opportunity to get on par with my other peers who came from more highly resourced institutions from a K through 12 perspective.
Daren Worcester: Yeah, that's a great point. And I share the experience coming out of a public school system and not having felt like in hindsight, seeing all the prep that a lot of students do, feeling like I really did put in the legwork or even knew to put in the legwork to prep for the SSAT, or excuse me, the SAT at that point. One other thing I do want to clarify, you talked about fee waivers. Just to make clear, so everybody understands a fee waiver does allow them to register for the SSAT and get the online practice for free. Somebody wants to see if they qualify for a fee waiver. How do they go about getting one?
Keith Wright: I don't know the exact process there. So that's another department within EMA. But I do know that they can speak to, say for instance, the school that they're working with in terms of whether or not they can get a fee waiver. There's a process that's involved. I'm not sure of all the different, say for instance, criteria that's used to determine whether or not one is eligible for a fee waiver or not. But I know over the years we have given over millions in fee waivers. So what that process is Daren and the criteria, that's another department that would be able to answer that question.
Daren Worcester: Yeah, you actually, you hit on the key factors there being that families inquire with the schools. EMA doesn't distribute fee waivers directly. We are funding them, but we don't distribute them directly. So if for families that are interested, contact the school, talk to them, and they will walk you through that part of the process. So Keith, I just have one last question, one last critique of standardized tests that I feel like we need to address here. And that's in regards to the SSAT being a predictor of future success. And one of the critiques of that is that, well, it only looks at cognitive abilities and not necessarily other qualities such as work ethic, teamwork, et cetera, that can certainly contribute to how successful someone is as they move on to other chapters in their life. What's your viewpoint on that?
Keith Wright: I agree with it. And again—
Daren Worcester: You agree with it?
Keith Wright: Yeah, I agree with it because as someone that I would consider myself a left-brain. So not to stereotype any one person, but I love math. So being in engineering and computer science, I can relate to this because we had to work really hard, my research institutions like Bell Labs where I worked at, because you had a bunch of smart people there from MIT, Georgia Tech, and these high-performing engineering schools. So that's who I worked with at Bell Labs. And so we were cognitively capable in math, but did we like to work together in teams? And all the non-cognitive stuff that I think about, probably not. So we have to develop those skills, but it's very important. I would say that we need those non-cognitive factors. They're just as important. Arguably some would say more important than the cognitive, but I don't like to say one is more important than the other.
I say always cognitive plus character. And so when I speak of character, I'm speaking about some of those non-cognitive factors that's important that research has shown, they are important for one success in their academic pursuit. So we speak of teamwork, resilience, intellectual engagement, initiative, open-mindedness, all those non-cognitive factors play a role in one's success. And that's why at EMA, we embrace and highly encourage all of our member schools to consider holistic admissions where they're not just looking at the cognitive piece, but also the non-cognitive piece. They both are important. And I can go back all the way back when Dr. Martin Luther King said one of his quotes was, we must remember that intelligence is not enough. It's intelligence plus character. That is the goal of true education. And that's where I am as well. And from my entire tenure at EMA, I've encouraged holistic admission—not just looking at one piece of, say for instance that application file.
Daren Worcester: And when you say holistic, you're talking the interview, you're talking the recommendations, you're talking everything down the line. But your team has also developed a complementary assessment for the SSAT to look at exactly that—at character, correct?
Keith Wright: That's correct. It's called the Snapshot. And so we have to be careful when we say character, right? Because we're not assessing a student's character from the perspective of when we think of one's character. So these would be just what I would consider, I like to use the term, we often use the term we use it interchangeably, but it's really non-cognitive factors or if you look at what we call it in the workplace, soft skills, those soft skills, I would also parallel non-cognitive factors in education to what we consider in the workplace soft skills that a lot of employers are putting even more emphasis on because again, they're hiring individuals with exceptional skills from a cognitive perspective, but they're lacking those soft skills. And as you know, we don't have teamwork. If we don't have collaboration and all that stuff, we're not going to be successful.
So yes, we developed an assessment to look at those non-cognitive factors that we feel are very important. I have to give credit to our schools. Our schools came to us and said, EMA help us with trying to assess what we're trying to assess without a standardized measure. So schools were trying to assess these non-cognitive factors themselves, but they understood that they all had different approaches and that probably wasn't fair. So you needed a standardized approach to say, for instance, to measure these important non-cognitive factors. And that's where Snapshot was born out of really from the school's approaching EMA. And we said yes, that's very important. And we were very excited to go down that journey. And I'm excited to say that I think Snapshot is an important assessment as well to provide that additional piece of information that's not cognitive Daren, as you just mentioned.
Daren Worcester: Excellent. Yeah. So I'll throw a plug in there and say that families can go to Admission.org to learn more about the Character Skills Snapshot. And we've kind of come full circle. We started out saying that the SSAT wasn't the end-all be-all, that it was part of a holistic admissions process. And at the end of our conversation kind of reiterating that and showing the other ways that we're looking at it. So Keith, I really appreciate you doing the conversation with me today. This has been very enlightening, and I'm sure a lot of families will find it helpful to really understand the why and the how behind standardized testing and a lot of work goes into taking the test and preparing for the test for families. So I think we owe it to them to understand what this is all about. So thank you for breaking it all down for us today.
Keith Wright: Great, and thank you, Daren. And just let me say I was really excited to do this podcast with you because it's my love for education as to the reason why I am in the field that I am in. Thank you.
Daren Worcester: And for all the families, we wish you all the best in the admissions process. Thank you for listening to this episode of #admissionchat and stay tuned. We'll have more episodes coming soon.
Announcer: Thanks for joining us for this episode of the #admissionchat podcast. #admissionchat is a production of the Enrollment Management Association. For more EMA resources to help families throughout the private school application process, visit Admission.org.
