Are Those Tests Evil? (ACT, SAT)

So, I have been doing some work on my college’s data relative to passing a math class correlated with either a placement test score or a score on the ACT Math section.  I shared some correlation data in an earlier post … this post is more about validity of measures.

One issue that has been impacting 4-year colleges & universities is the ‘test optional’ movement, where institutions make admissions decisions based on factors other than standardized tests.  This is an area of some research; one example is at http://www.act.org/content/dam/act/unsecured/documents/MS487_More-Information-More-Informed-Decisions_Web.pdf if you are interested.  Since I work at a community college, all of our admissions decisions are ‘test optional’.

Michigan uses standardized tests (ACT or SAT) as part of the required testing for students who complete high school, and the vast majority of our students do complete high school in Michigan.  Curiously, less than half of the students have standardized tests on their college record.  This both creates some interesting statistical questions and some practical problems.

For the past several years, the ACT has been that test for Michigan high school students (a switch was made to the SAT this year).  We use the ACT standard for ‘college readiness’, which is a 22 on the ACT Math section.  That standard was determined by the ACT researchers, using a criteria of “75% probability of passing college algebra” based on a very large sample of data.

A problem with this standard is that “college algebra” has several meanings in higher education.  For some people, college algebra is synonymous with a pre-calculus course; for others, college algebra is a separate course from pre-calculus.

My institution actually offers both flavors of college algebra; we have a “Pre-Calculus I” course as well as a “College Algebra” course.  The College Algebra course does not lead to standard calculus courses, but does prepare students for both applied calculus and a statistics course.  The Pre-Calculus I course is a very standard first semester course, and has a lower pass rate than College Algebra.  The prerequisite to both courses is one of (a) ACT Math 22 (b) Accuplacer College Level Math (CLM) test 55, or (c) passing our intermediate algebra course; all three of these provide the student with a “Math Level 6” indicator.  We assign a higher math level for scores significantly above the thresholds listed here.

So, here is what we see in one year’s data for the Pre-Calculus course:

• ACT Math 22 to 25                      63% pass pre-calculus I
• CLM  55 to 79                               81% pass pre-calculus I
• passed Intermediate Algebra    71% pass pre-calculus I

The first two proportions are significantly different, and the first proportion is significantly different from the ‘75%’ threshold used by ACT.  One conclusion is that the ACT College Readiness standard is based more on other “college algebra” courses (not as much pre-calculus).

One of the things we find is that there is little correlation between the ACT and passing Pre-Calculus.  In other words, students with a 25 ACT Math are not any more likely to pass than those with a 22.  This is not quite as true with the CLM; the probability of passing increases (though slightly) with scores as they increase from the cutoff.

Now, a question is “why did so many students NOT provide the College with their ACT scores”?  Well, perhaps the better question … “Are those who did not provide the scores significantly different from those who did provide them?”  That is a workable question, though the data is not easy to come by.  The concern is that some types of students are more likely to provide the ACT scores (either white students or students from more affluent schools).

We’ve got reason to have doubts about using the ACT Math score as part of a placement cutoff, and reason to prefer the CLM for predictive power.

More of us need to analyze this type of data and share the results; very little research is available on validity issues of standardized tests done by practitioners.

 
Join Dev Math Revival on Facebook:

No Comments

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a comment

You must be logged in to post a comment.