"How Did We Do?" Part 1: Why the Metric Matters in School Data Presentations

December 16, 2017

Often district or school administrators ask their quantoid* minions how well their students did on a standardized assessment.

 

Administrator: "Just make some charts showing generally how we did."

 

Quantoid: "Sure. But how do you want it displayed? And what sort of numbers do you want?"

 

Administrator: "I'll let you decide...you're the expert!"

 

Quantoid: "Okay, then."

...

Quantoid: "Here are the charts as requested"

 

Administrator: "Why did you use the median score? Why not percent proficient?"

 

Quantoid: "Excuse me for a moment...I just need to slam my head in the door a few times... I'll be right back."

 

To be sure, I'm not blaming either party for such situations (though I do relate more to the quantoid side). Administrators should have some idea of what they want and quantoids should have a general sense of how to tell a story with numbers that fits their audience.

 

To that end, I present a few common metrics along with their advantages and disadvantages, along with some advice on the matter.

 

The Percent Proficient

 

This is usually defined as the portion of students who attained a score at or above a given cut score determined to be representative of "Proficient", usually by a standard-setting body of educators.

 

Advantages:
 
  • Easy to understand and explain

  • Relates to a cut score that has been carefully considered

  • Can be used to compare different grades and subjects

  • Can be used for higher-order analyses

 
Disadvantages:
 
  • Does not address the average or distribution of scores
  • May not account for error

  • The definition of "proficient" may change over time (i.e. raising cut scores)

  

The Average (Mean) Scale Score:

 

Advantages:
 
  • Easy to understand and explain

  • Addresses the central part of the distribution

  • Can sometimes be used for some higher-order analyses

 
Disadvantages:
 
  • Vulnerable to outliers

  • Does not take error into account

  • Does not describe the shape of the distribution

  • Cannot generally be used across subjects (or grades, unless scale is vertically aligned and lends itself to such comparisons)

  • Does not take cut scores into account (it's a relative measure)

 

The Median Scale Score:

 

Advantages:
 
  • Robust to outliers

  • Easy to understand and explain (depending on the audience)

  • Addresses the central part of the distribution

  • Can sometimes be used for some higher-order analyses

 
Disadvantages:
 
  • Vulnerable to inliers (though these are generally less common than outliers)

  • Does not take error into account

  • Does not describe the shape of the distribution

  • Cannot generally be used across subjects (or grades, unless scale is vertically aligned and lends itself to such comparisons)

  • Does not take cut scores into account (it's a relative measure)

 

Given the above, what is the best way forward?

I propose the following for presentations:
 
  • Use the percent proficient but also show the distribution shape. Here are some examples of how this can be achieved.

  • Start out with an overall set of graphs showing percent proficient by grade and subject.

 

Like this:

Sometimes, it helps to add reference lines showing state proficiency (or targets), in this case, per grade, in relation to local numbers.

 

Like this:

 

If all data needs to be in one chart, you could use a heat map (though the above is best in terms of principals of visual design).

 

Like this (note the lack of references to state results):

 

Next, and depending on your audience, you may want to display information about the distribution of scores in relation to cut scores. You could use a box-and-whisker plot, but that's often a one-way ticket to Snoresville for board members. Alternatively, you could use my current favorite chart: the probability density function or distribution plot.

 

Like this:

 

                 1 = "Not Proficient"       2 = "Partially Proficient" 

3 = "Proficient"               4 = "Advanced"

 

This can be "paneled" (put into a matrix) by grade and subject if needed. The "rug plot" on the bottom represents each score (student) as a small vertical line. It can be omitted, depending on the audience. One can see concentrations of such lines corresponding to the height of the curve above. In this case, many students were at level 3 ("Proficient"), but there were a few at level 1 ("Not Proficient") and some at level 4 ("Advanced"). It would be also interesting to plot the distribution of state scores in a different color on top of this (without the rug plot) if they were available.

 

In this way, one can supplement proficiency rates with an idea of the distribution of scores for a given grade level and subject area, for those who want to know. I do hope this advice is helpful for you and your school district. I will demonstrate another way to show distribution in a following post.

 

 

*Quantoids are what I call those of us who crunch numbers for schools and districts. It is a term borrowed from this article: https://eric.ed.gov/?id=EJ333021

 

 

 

 

Please reload

Recent Posts

Please reload

Archive

Please reload

Tags

I'm busy working on my blog posts. Watch this space!

Please reload

(734) 252-9040

Plymouth MI United States 48170

©2019 ExceLance, LLC