# Evaluating a Program With Minimal Data Points: Step 2: Year-to-Year Gain

Do We See A Year-to-Year Gain?

Many schools measure a program in yearly gains.  In one year, students should show a year of growth.  What we mean, of course, is the school year; that from September to June students will gain a grade level.  And we hope there is little regression over the summer.  We will discuss the difference between yearly and school year gains in our next post.  For now, let’s focus on yearly gains.

Are students learning?  After identifying where you students are (the previous post) your next task is to measure growth over a year.  To start, I suggest spring to spring because that measures where they are after a year of instruction.  Again, here is my spreadsheet from Grace Haven Elementary.  Note Column E, which simply takes the DRP (Degree of Reading Power) score from 6th grade and subtracts it from the 7th grade score for a single number I call “gain”:

It is really important to compare apples to applies.  One of the strengths of the DRP is that the scores are comparable from year to year.  Whatever measure you use, please make sure the measures match so that you are able to measure a year’s worth of growth.

What is not always clear is what a year’s worth of growth IS.  For example, the DRP offers an I90 score (the reading level a student is able to read independently with 90% understanding).  I would expect that every student gains by just being in the building.  By having a year of life under their belt.  But what does a year’s worth of gain look like?  At Grace Haven, they gained a bit over five (5) points on the I90.  That does not seem like a lot.

Except that, as we learned in Step 1. over half of the students were in the top three stanines*, or top 23% nationally.  Where do they have to go?  If half of our students cannot gain much, it dampens the possible growth of the group.

So take them out.  When we look at those who are not in the top stanines another picture emerges.  In the case of Grace Haven Elementary, the growth is mixed.  Some students gain a lot, some regress, and others stay put.  If you have the former, pat yourself on the back before moving into the tough analysis.  If the latter–stagnation–you might think about the questions those in regression need to be ask because your program is not where you want it to be.

Regardless, just because a student is learning does not mean a teacher is teaching.  Can we take credit for success?  Or failure?  We need to know the effectiveness of our program if we hope to increase that effectiveness.  What, for example, if the school year gains only make up for a huge summer regression?  What if students, after a year of work, only gain slightly?

When we look at the 12 kids who regressed, half did so over the summer.  But half did so over the school year.  So, over 180 days of instruction student reading actually regressed for 6 students.  Our school made them go backwards.  Both summer or school year regression are results we should be concerned about, but the latter points to something we can control but are not.

Of those 12 students, 3 are in the top stanines–they have nowhere to go.  Similarly, the glut of students with none to minimal gains are also in the top stanines nationally–they had nowhere to go.

Still, it raises a basic question: Does our program help kids raise their game, or does it rely on previously done work and simply maintain that.  If the latter, those who did not “get it” earlier are not getting what they need now.

Restatement: Introduction to These Next Few Blog Posts (Backstory for those coming to this post first).

We get a lot of data. It may come in the form of test scores or grades or assessments, but it is a lot.  And we are asked to use it. Make sense of it. Plan using it.

Two quotes I stick to are:

• Data Drives Instruction
• No Data, No Meeting

They are great cards to play when a meeting gets out of hand.  Either can stop an initiative in its tracks!

But all of the data can be overwhelming.  There are those who dismiss data because they “feel” they know the kids.  Some are afraid of it.  Many use it, but stop short of doing anything beyond confirming what they know–current state or progress.  And they can dismiss it when it does not confirm their beliefs. (“It’s an off year”)  Understanding data takes a certain amount of creativity.  At the same time, it must remain valid.  Good data analysis is like a photograph, capturing a picture of something you might not have otherwise seen.

This series of blog posts will take readers through a series of steps I took in evaluating the effectiveness of my reading program.  I used the DRP (Degree of Reading Power), a basic reading comprehension assessment, as my measure because it was available.  I’m also a literacy teacher, so my discussion will be through that lens–but this all works for anything from math to behavior data.

* A stanine (STAtistic NINE) is a nine point scale with a mean of 5.  Imagine a bell curve and along the x-axis you divide it into nine equal parts.  The head and tail is very small area (4%) while the belly is huge (20%).  Some good information can be found in this Wikipedia entry.