By: Stuart Gurrea, Jéssica Dutra, and William K. Schwartz^†

The recent ruling for defendants Students for Fair Admissions v. The United States Naval Academy, et al. (“SFFA v. USNA”) underscores the limitations of regression analysis when applied to complex, multifaceted decision-making processes like college admissions. Regression analysis is used to quantify the impact of alleged gender, ethnic, or racial discrimination on outcomes beyond college admissions, such as hiring decisions or loan approvals. The court concluded in this case, which plaintiff Students for Fair Admissions has appealed, that relying solely on a subset of the information that USNA considered in its holistic admissions reviews likely leads to overstated estimates of the impact of race and ethnicity on USNA’s admissions decisions. The ruling explores the bounds on the probative value of statistical evidence of discrimination. The remainder of this article draws lessons on what those bounds are from which econometric arguments did and did not appeal to the judge.

An oversimplified regression model of admissions cannot reliably estimate the contribution of race to admissions decisions

Multiple regression analysis is a statistical technique used to quantify the relationship between an outcome of interest and several explanatory variables. In the context of race-based discrimination litigation, the outcome of interest might be college admissions, loan or hiring decisions, or quality of medical care. By including race alongside sufficient other relevant factors, researchers can estimate the independent effect of race on the outcome of interest. In SFFA v. USNA, the key statistical question was whether the regression model provided by SFFA’s expert included enough other relevant factors to isolate the effects of race and ethnicity on the Naval Academy’s admissions decisions from the effects of variables such as socioeconomic background.

USNA acknowledged considering race and ethnicity as one of many factors in its holistic review of applicants’ files. SFFA’s expert attempted to quantify the impact of race and ethnicity on admissions decisions using a logit regression model that explains the probability of admission with several factors, including race and ethnicity. Secretariat Managing Director Dr. Stuart Gurrea, serving as a testifying expert for USNA, argued that this model oversimplified the complex admissions process and produced unreliable estimates of the impact of race and ethnicity. The court agreed with this assessment, finding that SFFA’s model failed to account for many other factors the Academy considered in its admissions process.

Non-statistical evidence can reveal the omission of relevant variables from a model of admissions, which skews the estimate of the impact of race

Dr. Gurrea’s critique of SFFA’s regression model centered on the potential for “omitted-variable bias”—a misattribution of an omitted variable’s causal effect on the outcome to an included explanatory variable. (Economists’ and statisticians’ use of the word bias has nothing to do with prejudice.) Regression models, by their nature, are simplifications of complex, real-world phenomena, like college admissions, and may not capture all relevant factors. Omitting key variables correlated with both race and the outcome of interest, however, can lead to biased estimates of the impact of race. As a result, the true causal relationship between variables can be either overstated or understated. In this case, if SFFA’s model failed to account for factors that are positively correlated with both race and admissions decisions, the model would overstate the impact of race on the probability of admission. The court found that SFFA’s expert omitted some variables from the model intentionally and others because data for them were unavailable.

SFFA’s model fit the data well, correctly predicting a large proportion of admissions decisions. But strong predictive power does not permit one to conclude that race directly impacts selection. Models with good fit can—and often do—misattribute explanatory power among the explanatory variables included in the model; for instance, a model may identify race rather than a challenging life experience (e.g., coping with a chronic illness in the family) as the reason one applicant was admitted and another, otherwise similar applicant was rejected. This misattribution occurs if USNA selects for overcoming challenges, minority applicants tend to face them, and the model is blind to their presence or nature. The Academy developed an extensive record of documentary and testimonial evidence that identified information they accounted for in their admissions decisions but that was not included in SFFA’s model; this information included narratives contained in letters of recommendation, applicant’s personal statements, and interview notes. Both experts acknowledged that omitting this information likely skewed SFFA’s estimate of racial impact. Having established the likelihood of error, the remaining question was the error’s direction.

Non-statistical evidence can prevail over statistical evidence to assess the direction of statistical bias

SFFA’s expert argued that the model likely erred by understating the impact of race on admissions. This is because non-minority candidates were also likely stronger in attributes that the Academy considered via narratives but that the model did not, such as overcoming challenging life experiences. Consequently, in the view of SFFA’s expert, the Academy’s racial preference must be stronger than his estimates showed to make up for minority candidates’ weakness in the omitted attributes. To support this conclusion, SFFA tested the effect of sequentially adding available variables to the regression on the estimates of race’s impact. Because non-minorities compared favorably on these variables, their addition to the SFFA model relatively strengthened non-minority applications from the model’s perspective. As a result, including these variables increased the weight of race in explaining the admission of minority candidates. SFFA averred that the estimates were conservative.

Dr. Gurrea disputed SFFA’s assumption that minority applicants were inherently weaker in unobserved characteristics. The available variables that SFFA’s expert sequentially added were different in kind from the unavailable variables he omitted. For instance, showing that adding in SAT scores increases the estimated racial impact (because non-minority applicants have good SAT scores) said nothing about what would happen if he added in a measure of overcoming challenging life experiences (where minority applicants may have an advantage). Therefore, it was impossible to infer from observed characteristics that minority candidates’ unobserved characteristics were relatively weaker. The court found that omitted variables positively correlated with both race and admission included socioeconomic disadvantage, exceptional or challenging life experiences, and whether an applicant is from a congressional district underrepresented at USNA.

A model that suffers from omitted variable bias has limited probatory value

The court’s findings highlight the importance of rigor in data analysis for litigation when quantifying the impact of the consideration of race. The court concluded that SFFA’s estimates of the impact of race and ethnicity on USNA’s admissions process were unreliable. In particular, the court disagreed with SFFA’s presumption that minority applicants were weaker on factors not accounted for in their regression model. The Supreme Court ruled in Bazemore v. Friday that omitted variable bias does not disqualify a regression model from evidence of discrimination in expert testimony “absent some other infirmity.” Consistent with that, USNA defendants did not attempt to exclude SFFA’s model. But the trial court in this case still recognized the limited probative value of a model that suffers from omitted variable bias, ruling against the model on its merits. Moreover, the court agreed with Dr. Gurrea’s opinion that SFFA’s estimate was not only biased but also overstated because of the omission of factors that would positively influence the admissions of minority applicants more so than of non-minority applicants. We will be interested to see whether and how SFFA adjusts its approach in its similar, pending suits against West Point, and, already since the USNA ruling, against the Air Force Academy.

† Drs. Stuart Gurrea, Jéssica Dutra, and William Schwartz are a Managing Director and Directors, respectively, at Secretariat. Dr. Gurrea testified in SFFA vs. USNA on behalf of USNA, and Drs. Dutra and Schwartz were consulting experts for USNA’s counsel at the United States Department of Justice.