Maybe it’s Not “Men of Color”: Equity in College Mathematics, data part II

A recent post here looked at a summary of pass rates based on “Pell eligibility” and race, where Pell eligibility is used as an indicator of possible poverty.  Take a look at https://www.devmathrevival.net/?p=2791 .  The basic message was that the outcomes for black students was significantly lower and that part of this difference seems related to the impact of poverty.

Today, I wanted to follow that up with some similar data on the role of gender (technically, ‘sex’) in the outcomes of students, accounting for poverty and race.  This seems especially important given the national attention to “men of color” (http://cceal.org/about-cceal/).  As a social justice issue, I agree that this focus on MEN of color is important given the unequal incarceration rates.

However, this is what I see in our data for all Pell eligible students in math courses:

As for the prior chart, this reflects data over a 6 year period … which means that the ‘n’ values for each group are large (up to 10000 for ‘white’).  Given those sample sizes, almost any difference in proportions is statistically significant.  All three comparisons ‘point’ in the same direction — females have higher outcomes than males, within each racial group.

However, notice that the ‘WOMEN” of color have lower outcomes than men “without color”  (aka ‘white’). A focus on men of color, within mathematics education, is not justified by this data.  Here is what I see …

• There is a ‘race thing’ … unequal outcomes for blacks and hispanics, compared to white students.
  [This pattern survives any disaggragation by other factors, such as different courses and indicators of preparation.]
• There is a ‘sex thing’ … unequal outcomes for men, compared to women.
  [This difference is smaller, and does NOT survive some disaggregations.]

There is a large difference in ‘effect size’ for these; the black ‘gap’ in outcomes approaches 20 percentage points (about  2/3 of the white pass rate), while the ‘male’ gap is 5 percentage points or less (90% to 96% of the female pass rate).  In other words, it does not help to be a woman of color; it just hurts less than being a man of color.

I think that pattern fits the social context in the United States.  The trappings of discrimination have been fashioned in to something that looks less disturbing, without addressing the underlying problems.  We have actually retreated in this work, from the period of 40 to 50 years ago; there was a time when college financial aid was deliberately constructed as a tool in this work, and this was effective from the information I have seen.  Current college policies combined with the non-supportive financial aid system results in equity gaps for PEOPLE of color.

Most of us have a small role in this work, but this does not mean the role is unimportant.  If your department and institution are critiquing your impact on people of color, terrific; I hope we have an opportunity to share ideas on solutions.  If your department or institution are not deeply involved in this work, why not?  We have both the professional and moral responsibility to consider the differential impact of our work, including unintended consequences.

 Join Dev Math Revival on Facebook:

Equity in College Mathematics: What does the data tell us about poverty and race?

I am very proud of my department for our decision to do some serious work on equity.  We are having focused discussions at meetings and in hallways, we are bringing up equity in other discussions, and have examined quite a bit of data.  I want to highlight a little bit of that data.  This post will focus on the role of poverty in the pursuit of equity in college mathematics.

Like many colleges, my institution provides access to a centralized data reporting function (“Argos” in our case).  We can use this database to extract and summarize data related to our courses, and the database includes some student characteristics (such as race, ethnicity, and sex … self-reported).  In addition, the database connects to direct institutional records dealing with enrollment status and financial aid.  The primary piece of data from the financial aid record is a field called “Pell Eligible”.

As you know, Pell Grants are based on need; this usually means an annual income of less than $30,000.  Students are not required to apply, even if they would qualify for the maximum award.  However, we do know that students do not receive a Pell ‘award’ unless they have a low income.  For us, this “Pell Eligibility” is the closest thing we have to a poverty indicator.

When we summarize student grades by race and Pell Eligibility (across ALL courses in our department), this is the result.

This graph has two “take aways” for me.  First, poverty is likely associated with lower rates of passing.  Secondly, the impact of race on outcomes is even stronger.  Note that the “Pell” group is lower than the non-Pell group for all races, and that the “Black non-Pell” group has lower outcomes than the non-Pell hispanics or whites.

The situation is actually worse than this chart suggests.  The distribution of ‘poverty’ (as estimated by Pell eligibility) is definitely unequal: 70% of the black group is Pell eligible, while only 40% of the white group is Pell eligible (with hispanics at a middle rate).

I am seeing a strong connection between our goal of promoting equity and the goals of social justice.  As long as significant portions of our population live in poverty, we will not achieve equity in the mathematics classroom … awarding ‘financial aid’ does not cancel out the impacts of poverty.  In addition, as long as some groups in our population are served by under-resourced and struggling schools, we will not achieve equity in the mathematics classroom.  This latter statement refers to the fact that many states have policies like Michigan’s which allow those with resources to have a choice about ‘better schools’, while limiting state funding for public schools (and simultaneously attacking the teaching profession).

In our region, the majority of the black students attending my college came from the urban school district.  This urban school district had a proud history through the 1980s, with outcomes equal to any suburban school in the area.  However, dramatic changes have occurred … even though that district has made significant progress in recent years, there is no doubt that the urban schools are not preparing students for college.  Poverty plays a role within that school district, and the interaction between race and poverty is again unequal: more blacks live in poverty within the city than other races.

The social justice movement seeks to provide all groups with equal access to upward mobility, combined with a reasonably high probability of escaping poverty, based on a presumption of effort.  Barriers to progress are addressed as systemically as possible.  College mathematics is currently one of the barriers to progress in social justice.  Modern curricula do not solve this barrier, given the data I’ve seen (though we are early in that process of change).

If we see our role as separate from equity and social justice, we are enabling the inequities to continue.  This is a set of issues that we can not remain silent about.  Even if we are not committed to social justice, we need to work on these barriers for the good of our profession.  You might begin by discussing social justice issues with your friends or colleagues who teach sociology or anthropology, quite a few of whom have a background in ‘social problems’.

Join Dev Math Revival on Facebook:

Culture of Evidence … Does it Exist? Could it Exist??

Perhaps you are like me … when the same phrase is used so extensively, I develop an allergic-type reaction to the phrase.  “Awesome” is such a phrase, though my fellow educators do not use that phrase nearly as much as our students.  However, we use ‘culture of evidence’, and I have surely developed a reaction to it.

Part of my reaction goes back to a prior phrase … “change the culture”, used quite a few years ago to describe the desire to alter other people’s beliefs as well as their behavior.  Education is based on a search for truth, which necessarily implies individual responsibility for such choices.  Since I don’t work for Buzz Feed nor Complete College America, my priority is on education in this classic sense.

The phrase “culture of evidence” continues to be used in education, directed at colleges in particular.  One part of this is a good thing, of course … encouraging the use of data to analyze problems.  However, that is not what the phrase means.  It’s not like people say “apply the scientific method to education”; I can get behind that, though we need to remember that a significant portion of our work will remain more artistic and intuitive than scientific.  [Take a look at https://www.innovativeeducators.org/products/assessing-summer-bridge-developing-a-culture-of-evidence-to-support-student-success for example.]

No, this ‘culture of evidence’ is not a support for the scientific method.  Instead, there are two primary components to the idea:

• Accountability
• Justification by data

Every job and profession comes with the needs for accountability; that’s fine, though this is the minor emphasis of ‘culture of evidence’.

The primary idea is the justification by data; take a look at the student affairs professional viewpoint (https://www.naspa.org/publications/books/building-a-culture-of-evidence-in-student-affairs-a-guide-for-leaders-and-p  ) and the Achieving The Dream perspective (http://achievingthedream.org/focus-areas/culture-of-evidence-inquiry  ).

All of this writing about “culture of evidence” suggests that the goal is to use statistical methodologies in support of institutional mission.  Gives it a scientific sound, but does it make any sense at all?

First of all, the classic definition of culture (as used in the phrase) speaks to shared patterns:

Culture: the set of shared attitudes, values, goals, and practices that characterizes an institution or organization  (Merriam-Webster online dictionary)

In an educational institution, how many members of the organization will be engaged with the ‘evidence’ as justification, and how are they involved?  The predominant role is one of data collection … providing organizational data points that somebody else will use to justify what the organization wants to justify.  How can we say ‘culture of evidence’ when the shared practice is recording data?  For most people, it’s just part of their job responsibilities … nothing more.

Secondly, what is this ‘evidence’?  There is an implication that there are measurements possible for all aspects of the institutional mission.  You’ve seen this — respected institutions are judged as ‘failures’ because the available measurements are negative.  I’m reminded of an old quote … the difference between the importance of measurements versus measuring the important.

There is also the problem of talking about ‘evidence’ without the use of statistical thinking or designs.  As statisticians, we know that ‘statistics’ is used to better understand problems and questions … but the outcome of statistics is frequently that we have more questions to consider.

No, I think this “culture of evidence” phrase describes both an impossible condition and a undesirable goal.  We can’t measure everything, and we can’t all be statisticians.  Nor should we want judgments about the quality of an institution to be reduced to summative measures of a limited set of variables covering a limited range of ‘outputs’ in education.

The ‘culture of evidence’ phrase, and it’s derivatives (‘evidentiary basis’, for example) are used to suggest a scientific practice without any commitment to the scientific method.  As normally practiced, ‘culture of evidence’ often conflicts with the scientific method (to support pre-determined answers or solutions) and has little to do with institutional culture.

Well, this is what happens when I have an allergic reaction to the written word … I have a need to write about it!

 Join Dev Math Revival on Facebook:

Factors in Student Performance: Improving Research

Our data work, and our research, in collegiate mathematics education tends to be simple in design and ambiguous in results.  We often see good or great initial results with a project, only to see regression towards the mean over time (or worse).  I’d like to propose a more complete analysis of the problem space.

The typical data collection or research design involves measuring student characteristics … test scores, HS GPA, prior college work, grades in math classes, etc.  For classical laboratory research, this would be equivalent to measuring the subjects without measuring the treatment effects directly.

So, think about measurements for our ‘treatments’.  If we are looking into the effectiveness of math courses, the treatments are the net results of the course and the delivery of that course.  Since we often dis-aggregate the data by course, we at least ‘control’ for those effects.  However, we are not very sophisticated in measuring the delivery of the course — in spite of the fact that we have data available to provide some levels of measurement.

As an example, we offer many sections of pre-calculus at my college.  Over a period of 4 years, there might be 20 distinct faculty who teach this course.  A few of these faculty only teach one section in one semester; however, the more typical situation is that a faculty member routinely teaches the same course … and develops a relatively consistent delivery treatment.

We often presume (implicitly) that the course outcomes students experience are relatively stable across instructor treatment.  This presumption is easily disproved, and easily compensated for.

Here is a typical graph of instructor variation in treatment within one course:

We have pass rates ranging from about 40% to about 90%, with the course mean (weighted) represented by the horizontal line at about 65%.  As a statistician, I am not viewing either extreme as good or bad (they might both be ‘bad’ as a mathematician); however, I am viewing these pass rates as a measure of the instructor treatment in this course.  Ideally, we would have more than one treatment measure.  This one measure (instructor pass rate) is a good place to start for practitioner ‘research’. In analyzing student results, the statistical issue is:

Does a group  of students (identified by some characteristic) experience results which are significantly different from the treatment measure as estimated by the instructor pass rate?

The data set then includes a treatment measure, as well as the measurements about students.  In regression, we then include this ‘instructor pass rate’ as a variable.  When there is substantial variation in instructor treatment measures, that variable often is the strongest correlate with success.  If we attempt to measure student results without controlling for this treatment, we can report false positives or false negatives due to that set of confounding variables. Another tool, then, is to compute the ‘gain’ for each student.  The typical binary coding (1=pass 2.0/C; 0=else) is used, but then subtract the instructor treatment measure from this.  Examples:

• Student passes, instructor pass rate = .64 … gain = 1-.64 = .36
• Student does not pass, instructor pass rate = .64 … gain = 0-.64 = -.64

When we analyze something like placement test scores versus success, we can graph this gain by the test score:

This ‘gain’ value for each score shows that there is no significant change in student results until the ACT Math score is 26 (well above the cutoff of 22).   This graph is from Minitab, which does not report the n values for each group; as you’d expect the large confidence interval for a score of 28 is due to the small n (6 in this case).

That conclusion is hidden if we look only at the pass rate, instead of the ‘gain’.  This graph shows an apparent ‘decreased’ outcome for scores of 24 & 25 … which have an equal value in the ‘gain’ graph above:

The main point of this post is not how our pre-calculus course is doing, or how good our faculty are.  The issue is ‘treatment measures’ separate from student measures.  One of the primary weaknesses of educational research is that we generally do not control for treatments when comparing subjects; that is a fundamental defect which needs to be corrected before we can have stable research results which can help practitioners.

This is one of the reasons why we should not trust the ‘results’ reported by change agents such as Complete College America, or Jobs For the Future, or even the Community College Research Center.  Not only do the treatment measures vary by instructor at one institution, I am pretty sure that they vary across institutions and regions.  Unless we can show that there is no significant variation in treatment results, there is no way to trust ‘results’ which reach conclusions just based on student measures.

 Join Dev Math Revival on Facebook: