The Boy Problem: Noise Obfuscation of True Problems

While a bunch of us were chatting, our administrator noted how all of the discipline cases he is dealing with are boys.  He then noted how most of our failing students are also boys, and that most of our top students are girls.  Not alone in this observation, he pondered what is perennially proposed: A different program for boys than girls.

Bad idea.

The problem of using gender to model programs is that it offers no path to a solution, except if the problem is gender discrimination.

In the data world they call this issue “the signal in the noise.”  9780143125082The signal is the problem–behavior and academic achievement–while classifications (gender, race, age) are the noise.  We identify the boy problem because gender is an easy way to categorize people.  We notice it.  Evolutionary, we are built to recognize patterns as a means of survival.  But our intuition can cause us to fall to, create and reinforce stereotypes.  We create more noise, and lose the signal.  Every stereotype has some truth at its core, but it ultimately binds the person it is being done to–we put the person in a box.  And, it makes others blind to the real problem and its solution.

To say that our school does not serve boys is to say that there is only one way to be a boy.  It’s a box we put people in.  For the sake of argument, let’s use the stereotype that “boys need movement.”  And let’s say that 80% of boys need movement.  That leaves us missing 20%.  And if the needs are reversed for girls–only 20% need movement–that leaves that group missing out, too   Everyone is in a box.

Instead, we might build a program around students needing movement, and another program(s) around something else.  Now, those student who need it (half) get movement.  That 20% of non-movement boys are now free to pursue their needs, along with 80% of the girls.  Win-win.

We track gender, race, SES and the like for two reasons.  First, some elements of an identified group we can address–SES kids, by definition, need to be fed. But that use is limited. Second, we identify groups because some have been discriminated against, historically.  That was the reason NCLB required those designations.  When a group comes up short, this provides a place for schools to start the conversation–is the cause discrimination?

After that has been answered in the negative (hopefully) the use of such designations should then move to characteristics of the individual students in question and their needs.  Why are these students getting into trouble?  How large an issue is that, and how can it be addressed?  In looking back at NCLB the one area that using the data made a big difference was with SES students.  Interestingly, the solution was like the 80:20 above–many needed something the school was not providing (but not all), and some non-SES students benefited from those same programs.  Win-win.  It was not because of discrimination, but in starting there schools took a fresh look at a problem and identified the true root cause.  Focusing on gender instead places the needs of some on the larger demographic while excluding others–it is not an efficient solution and can create new problems.

So what is our school’s problem?  We are too indulgent.  In our desire to provide to students what they need to succeed we have failed to hold them accountable.  We take off limits but do not demand responsibility as part of the bargain.  For example, I allowed music last fall because students benefited as it canceled out distracting noise.  Now, it is the distraction.  And music has creeped into other activities and classrooms.  Snacks have become meals.  Fidgets have become toys.  Water and bathroom breaks are a right.  In the end, I should have a product.  Not always.  In opening the barn door first I have now set up my enforcement of academics as conflict, not an inspired goal.  Too often, the work reflects this shift in tone from support to scold.  The exception is the rule.  We indulge.

We have reached the tipping point: As we approach 20% of students being an exception, the exception becomes the rule.  When music left the classroom, using during essay writing only, it became an exception.  As 20% of classrooms had exceptions (hats, music, no lines, snacks) it becomes harder to hold to the rule.

There are two fixes for this.  The first is to hold the rule.  No hats.  No music.  Lines.  The second is to add responsibility and accountability to the privilege.  Currently, we ask nothing in return for privilege.  Why?  Because monitoring it is difficult and it sets up conflict.  But that’s exactly how we creep towards the tipping point–we let those with the least amount of respect for others redefine the rule.  When those 20% change expectations the number of rules pushed grow and the number of students breaking them grows, too.

Those kids are defending by packs of adults, all justifying why they need it and apologizing for transgressions.  It’s not the exception that is the problem, but the transgression and accountability.  We are starting with an assumption that this thing bestowed–music, gum, movement–is a right to be taken away, not a privilege to be earned.  This is the exact opposite of how the adult world works–the most responsible gets the privileges while those lacking control either get few rewards or confinement.  And we excuse them in academics, too, for the same reasons.  We are doing these kids a disservice.

All of this is hard.  You can’t penalize a kid for the lack of structure provided up until the moment they cross a line.  The Responsive and Developmental Design programs offer those systems and protocols, but they require time and commitment from the group.

I would argue that it appears boys are the problem because success involves awareness and impulse control.  Boys seem to have more of a problem with this, but they are not alone.  Because society excuses much of it based on stereotypes (“boys will be boys” and “you can’t expect a boy to sit for an hour”) they hear that and internalize it.  When we lean on punishment, though, we are teaching students not to be caught.  Our data does not account for those who appear to follow rules but who skate the line constantly–take a census of how many are not where most of their peers are, or doing outlier behaviors, and you’ll find “they have permission.”  Plus, we have behaviors that are more personal, and do not affect others.  Even with academics, Tier II is filled with those who flail openly and dramatically.  When we stop looking at the major behavior data, but instead account for minor behaviors and any deviations from the rule, a true picture of our ailments become clear.  The solution is not movement but accountability to norms and earned privilege.

This Wall Street Journal article is a nice summing-up of the balance between challenge and aspiration.

Counterfactual Reasoning, Black Swans, and Common Planning Conundrums

There are many nuts-and-bolts topics discussed (which is probably why you are here) in the name of counterfactual thinking and black swans.  All interesting, but you may by here for only one, so scroll down to what interest you.  In order, the topics are: Introduction, Configuration, Schedule, Placement, Last Thoughts.  Enjoy.

Counterfactual Reasoning and Black Swan Planning

A few weeks ago I wrote about counterfactual reasoning–that it is easier to prove something doesn’t work than it is that is does work.  For example, it is hard to prove that coming to school leads to learning, but we know truants fall behind.  This is important to remember as we plan–we can’t confirm much about the future, but we can plan with that blind spot in mind.

The idea becomes important when looking at the surprises we seem to encounter after we do a lot of planning for the next year (placement, schedule, resource allocation, etc.).  The theory of this is the “black swan“; that unexpected events have a major effect on plans.  Because of this, the theory goes, we should plan with the unexpected in mind.  The other half of the theory is that it is difficult to know what to plan for.  In short, just be ready for something.

Note: Much of this is inspired by  Nassim Nicholas Taleb’s Black Swan: The Impact of the Highly Improbable.  Much of this essay looks at the unexpected events that cause those disruptions every educator can relate to–nothing Earth shattering, but annoying nevertheless.

As educators, we tend to lock everything down–protocols, lessons, spaces, schedules, class lists–because we like to think we can predict everything.  For example, our school locked students into the three homerooms last June, but we did not take into account several unexpected developments.

The largest was we had asked students their World Language preference (French or Spanish).  Our students move through the school by classroom–Art, PE, Music and World Language.  During the summer we realized we could not honor most preferences because placement was based on a different criteria.  We punted.  But we also did not expect several students from one classroom to move away or that so many new families would move into our district.  We placed with insufficient information.  Then student preferences for WL created more turmoil, as parents demanded their kid get this or that language (they, of course, waiting for three weeks into September before doing so).  It also took a few weeks to figure out the WL/Tier II needs of new and leaving students.  Who knew?

As the old Yiddish proverb goes, “Man plans and God laughs.”

One of the mistakes people make is in trying to anticipate the black swan.  If we only anticipate more contingencies we think we can control it all.  No.  We certainly can straighten out many issues.  And we still have to make plans–schedules, room assignments, placement, etc.  How, though, can we plan while respecting the unknown?

Here are some examples of common issues facing schools.

Configuration

A few weeks ago I shared a configuration idea for our middle school.  For forever they have been multi-age, but they are contemplating going straight-grade “just for next year” because class sizes are getting too large and students are falling through the cracks.  There are several challenges facing the team, but the main one will be the influx or exodus of students.  I argue maintaining multi-age classrooms is the best safeguard against student numbers changing.

They are adding a fifth teacher to a four person team.  The Core is responsible for Language Arts, Math, Science and Social Science.  Currently, two teachers are licensed to teach STEM and two Humanities.  To alleviate class sizes they plan to hire a Science teacher.  In theory, this lowers the class average from 28 students to 23.  Of course, because the new teacher can only teach Science it does nothing to help the Humanities classes .  In theory, one of the STEM teachers will shift some of her load towards Math, but Math being taught straight-grade and other elements conspire against such neat divisions.

Using the our middle school’s straight-grade plan as an example, three teachers for the 7th and two for the 8th creates an automatic imbalance.  Of course, these can be mitigated (supports, aides, schedule).  But, should five students leave the 7th, or five join the 8th (or both), there is suddenly a difficult situation with no easy fix.  We can’t know if that scenario will play out, or be worse.  The plan, though, is crafted with little wiggle room should any black swan appear.

Using counterfactual thinking, instead create a multi-age made up of two teams.  Each team has a Science and Humanities teacher.  A single Math teacher instructs all students–a big load, but she is relieved of other duties and the two teams can focus on social-emotional middle school programming.

Imagine, now, five students leave the seventh–the release would be divided between two teams, offering a slight relief of two or three on each team.  In the case of the five joining the 8th, a much more devastating event, that burden would be shared and only result in two or three each.  In fact, if both scenarios play out, much of the effect is a wash.  Multi-age benefits can mitigate the other effects of population bubbles.  For example, each year teachers move (with their classrooms) and they need to relearn new curriculum and start new team protocols, etc.  Multi-age allows multi-dimensional flex.

But the question to ask is why that plan will not work?  I have shown the problems of straight grade (there are more, unwritten here), but that does not mean multi-age is better.  The black swan, counterfactual advice is to take each option and poke at it–with some of the more outlandish variables–until it reveals its flaws.  Then, put down other ideas, too, and poke them.  One area to look at is which “losses” are most tolerable and which would be devastating.  The plan with the most resiliency is probably your best bet.

Schedule

The difficulties of crafting a schedule are long and well known (we even know that we don’t know enough).  This is especially true in schools with many grades and diverse needs (we are a pre-K through 8).

I actually learned a lot recently on why crafting a schedule is so hard through an article on chaos theory (an article for later).  In short, such endeavors are dynamic and resetting course causes exponential problems down the path.   What we do know is that schedule craft works up to 80% of the elements are placed, but then conflicts trouble the last 20% of needs (that’s the dynamic nature of the process).  To fix it we begin to make compromises (which creates more conflict, exponentially).  Each year, someone (or many) is (are) unhappy.

Then, each year, we have a black swan surprise.  Last year, it was WL and everything that followed it.  Other grades had other emergencies.  Each fix seemed to bring more problems.  We live with it–we have no choice–but that 20% consumed a lot of resources and continues to create ripples (e.g., it is unclear if student preference for WL will be honored in 7th grade now that they have a year of one language under their belt.).

My solution is blocking out time instead of scheduling in time.  For example, one grade level figures four uninterrupted hours of seven school hours are needed for Core–90 minutes of Literacy, 60 minutes for Math, 60 for Science/Social Science and 30 for TA.  We demonstrate what’s important to an organization by the resources we put towards it (time, money) and four hours uninterrupted seems a reasonable commitment.  That leaves three hours for lunch, recess, Unified Arts (UA), band, mentor, counseling and any other non-Core activity.

How is a block resilient?  Let’s look at how our current system is not.  Predicting a seven hour day is hard; there are many variables.  It is also easy to be fooled by small numbers–three kids pulled out here, four there.  Currently, music lessons, mentor, speech and physical therapy, and other student needs are pulled from Core.  There are many reasons for this–part time employees and mentors have locked-in schedules, no one wants to deny a child their one Art class a week, and a few kids here or there are seen as non-disruptive.  But in my class of 20, one absence and three music lessons are 20% of the class.  Plus, recovery now becomes a negotiation not with two other Core teachers, but with three times that–pull from Art? PE? (And, because of the dynamics of the interwoven schedule, also lose 20% of their students and need recovery for them).  Again, here is the problem I am poking a hole in with our current schedule and how it was created.

By dividing the day into two blocks (4 hour, 3 hour) that do not overlap, students and teachers only have to negotiate within those blocks, with only those adults.  It is more likely I can flex with my Core teammate’s schedule because a) we have an aligned schedule, b) fewer pulls (classes) means fewer and smaller disruptions, c) fewer agents means less communication needed.  On Core, we can debate the virtues of Math vs. Science and time required to do it right, but with blocks we do not have to debate the death by a thousand cuts we currently face.  Four hours uninterrupted shows we value Literacy, Math, Science and Hive.  Three hours should be enough for others to negotiate what is best for students, among actors who have more in common (UA).  Or, to demonstrate that we want more for our students than the resources we have (or are willing) to give.  Again, this may be flawed.  Poke it.

Student Placement

Has there not been an issue with placement?  Do kids every come as advertised, stay friends (or enemies) with the kids from the previous year, or continue to need the same services?  Too often, we’ve even had to change the teacher assigned to the class (illness, pregnancy, number shifts).  For all the benefits we argue placement provides, those are the holes I poke in our locking kids in as we do.

Three years ago, the middle school wisely created two teams with a blank slate for placement.  First, Special Ed (SE) placed students on the two teams with a variety of rationales–service providers, friend/tormentor locations and the like.  This was open to debate/poking by the group, but we deferred to the experts.  Second, a small number of kids who do not receive services were separated or paired because of behavior or emotional needs.  Some kids need friendly peers, while others are a bit too friendly during class time.  All told, about 20% of placement was locked in like this.  That is an example of necessary planning with what is  known (at that time).  What’s important to note is what they did not do–lock in the remaining 80%.

Then, the sending teachers made two “balanced” teams.  More poking.  But the most important part was that no one was locked in until mid-September.  Each week the middle school would observe interactions, listen to students and families, and shift kids around.  Then, they locked in all kids, creating permanent teams and focused on creating identities for those teams and bonds within.

They waited because kids change–they move, they mature and their allegiances shift.  Because services did not have much flexibility (the nature of Spec Ed and part-timers), and some kids need reassurances (i.e., not being placed with a bully or knowing an adult face before the first day of school), those 20% never shifted.  But, as things shook out, a flexibility was built in for the other 80% so that movement was possible.  By the third week, the two teams were as solid as one is going to get.  It was a successful year.

Compare that flexibility with what happened when we locked in the grade last year that suffered at the hands of WL.  For what was gained (a faint sense of knowing their homeroom) much was lost.  While we should be aware of the anxiety we create with some unknowns, it can be fleeting compared to the year-long stress of a bad fit created by the black swan.  Again, the above might not survive a good poke, but our current spring lock-in is troublesome in the long term.

A Note About Counterfactual Poking

It is easy to spot flaws.  The question is a) if that flaw is true in any viable model, b) if that flaw has a major effect.

For example, one argument I have heard against postponing placement until a few weeks into the year is anxiety and parental concern.  Yet, when we locked in the placement of kids, one or two parents would inevitably call with concerns (or be more forceful about it).  Sometimes, that brought a dam break of other requests as parents micromanaged.  In short, parents and kids worry regardless.  The former is easily dealt with because the concerning elements are often soothed with more information, and solutions can be shared (including locking in needs).  The latter upends a carefully planned order, with results (according to chaos theory) that can prove exponential.

So, caution is necessary when weighing options.  People need to be objective that all plans have downsides–sometimes the same downsides–but some recover better.

Conclusion

The idea is that the we don’t know what we don’t know, but we still have to plan.  We want to poke holes in things early, and be honest about defects, before we choose the model that is best for students.

9 Steps to Creating a Schedule for a Complex Organization: Part Two

Schools are not the only organizations with a host of constituents demanding varied needs be met, but they offer a great case study.  I have broken these steps into two distinct parts: Theoretical  and Building.  Both are essential to success over 180 days.
Warning: It is easy to skip steps, especially the theoretical.  This groundwork is essential for the building portion to work.  Otherwise, you’ve got a “Blind Men and the Elephant” scenario.  Check out Part One here.
Part Two: Building: The mistake here is to see this as blocks needing to be stacked.  It’s true, in a sense, but when that mental shift happens the schedule produced becomes a series of pounding square pegs into round holes; lots of edges feel the rub.  At the same time, the water metaphor no longer works because a rigid line is needed in order to truly understand what works and what only seems to work (but, in the details, things are lost).
5. Schedule Everything: The big things become clear, and often have minor compromises; nothing is harder to roll with than additional surprises.  Often, these surprises are more disruptive than the larger issues.  For example, the one grade level at our school had 4.5 hours a day with students, or 22.5 hours in a week.  Band and Chorus was scheduled for three 40 minute blocks, or 2 hours, for 2/3 of our students (the remaining 1/3 were left for the teachers to plan for).  That left 20.5 hours of instruction for all kids.  Seems reasonable, except instrument lessons pull an additional 4 hours, and those are small clumps of kids that come-and-go at odd, rotating times–very disruptive.  When you craft the schedule, and put in 6 plus hours of music, it paints a very different picture.
But the time culprits are many: Guidance, mentors, Tier II, Special Ed…..  Each needs to pencil in a projected time.  The guidance counselor, for example, knows a bunch of his caseload.  If a special 3-4th grade block was identified based on expected need and time outside of non-negotiable times, that service would have a solid place to start and flexibility moving into September.
6. Populate the Schedule: Without a dry run, the schedule is an estimate.  That leads to grey areas, but grey areas add up to black over the course of the process.  While you might not have placement, you do know how many kids get pulled for X, Y and Z and can make reasonable assumptions about where they go.  For several years, one grade level had two sections of World Language–one French and one Spanish.  Then, the population bumped up.  It was a surprise that there needed to be three WL classes, not two as there had been.  That meant one Spanish and two French, which drove the UA groups, that had been driven by the homerooms–see, dominoes.  With each grade picturing their day–if not walking through it–many of the unexpected bits come into play.  Account for every kid and location.
7. Dry Run the Schedule: Even if each grade level imagines the day instead of physically walking through it, that’s something.  Did you take into account transition time?  The time it takes to put on snowsuits and take them off is substantial, especially if UA proceeds and follows recess.  For example, one grade’s schedule has kids go from UA to recess, which works well, but when they need their Chromebooks for UA after recess they go to the room, undress, get Chromebooks and walk to UA–five to ten minutes lost in a 40 minutes class.  Run the schedule in winter and warmer months, or whatever demands change in your locale.
8. All Teachers Sign Off: Every year, groups are asked to look and report back on if it works.  Every year, most people do.  But some don’t.  A few yeas ago, no lunch was scheduled for UA in the frenzy of crafting something that works; adjusting for it made the schedule a kludge, but since it was June everyone just went with it.  These things were not caught until after the schedule was set.  Everyone was mad at the group all year for the compromises.  Pet Peeve: There is often a member of the staff that claims to be helpless, and the nurturing environment of a school bends over backwards to accommodate, even as that person is rigid in their not being proactive or bending in retrospect.  Ugh.  At least with a signature, there is no buck-passing and everyone is forced to take some ownership of their part in the process.
9. First Schedule Wins (Bad): We like to believe the process is open and cooperative, but in reality the first schedule that works for 3/4 of the school rules.  Even with compelling arguments against it, little more than tweaks occur.  For that reason, the first few rules above are essential.  They need to be concrete as possible, and drive planning.
One way to counter this is to present multiple, diverse schedules.  Presented with three very different ways of doing something, people will see the possibilities, even when the schedule just doesn’t work.  By then going and adapting the best of several, you not only get more good ideas, but everyone sees how what was chosen is not the first presented, but the best of otherwise flawed plans.  It is otherwise easy to find fault without being tasked with a solution.
Every group needs to feel empowered.  There are a few ways to do this.  For example, allow each group to have the power to veto at each step, but once a step is adopted they lose that power.  So, if 90 minutes of literacy is a non-negotiable and adopted, the schedule bends to that even if means other desires are compromised when the final schedule comes out.  Each group should also have the power to create an alternative schedule.  Whatever, but something needs to happen to counter the “First Schedule Wins” problem.

9 Steps to Creating a Schedule for a Complex Organization: Part One

Schools are not the only organizations with a host of constituents demanding varied needs be met, but they offer a great case study.  I have broken these steps into two distinct parts: Theoretical  and Building.  Both are essential to success over 180 days.
Warning: It is easy to skip steps, especially the theoretical.  This groundwork is essential for the building portion to work.  Otherwise, you’ve got a “Blind Men and the Elephant” scenario.  Still, you can check out Part 2 here.
Part One: Theoreticals: At this point of development, the schedule is like a river: Water is fluid, bends around anything and fills every crevice.  Much like time.  Here, you will begin to construct the vessel that contains it (beginning with Start and End times of the day) and identifying those impediments it needs to flow around.
1. Vision: The schedule is a physical manifestation of the vision.  For example, if you want certain people to collaborate they need the same preps.  Or, if starting the day with a Restorative Justice circle in homeroom is important, than the day cannot begin before 8:20, so teacher have at least 15 minutes to start the day right.  Make a list of the vision(s) in all respects and then figure what elements of the schedule facilitate that.
2. Recognize What’s Just Tradition: Much of the schedule is the remnants of old visions, initiatives and needs.  In keeping what worked this year, we often perpetuate other patterns that no longer drive the schedule.  For example, we had Art scheduled on a day that worked for the part-time Art teacher, who worked at other schools on other days–when her situation changed, no one thought of changing the day.  Spending time just messing with assumptions both helps question how things are done and opens up new possibilities.  Often, these are impediments to work around that people do not even realize they are accommodating for; removing them frees up possibilities.  Even those that are preferential may not be as non-negotiable as an issue that comes up later than is mission critical to members.
3. Plan for Two Years (at Least): The class sizes are clear–if the 3rd grade needs three teachers, the 4th grade will need it next year.  You should also account for what the 4th grade looks like the year after, too.  Which class sizes are close to a bust (so that three kids moving in in August requires an additional teacher, classroom and the like), or might not be sustainable (i.e., that teach could be moved to a grade that needs her more)?  Plan teams, room assignments and the like towards that.  Also, think how part time staff will be used over that time.  It may not seem directly related to the schedule, but a) planning two years makes you be realistic about next year, and b) most schedules go south because small, unexpected changes that happen before September 15.  Best be ready.
4. Define Non-negotiables: What are bedrock points, that no one can touch.  For example, at one point 60 minutes of Math and 90 minutes of Literacy a day were non-negotiable in the schedule.  For any grade level, I would advocate for 3 hours a day uninterrupted–no lessons, pull-outs or other distractions; we are guaranteed all of our kids for that period of time.  Class size might be another (Art can only hold 25 students physically in the room).
Part Two: Building: The mistake here is to see this as blocks needing to be stacked.  It’s true, in a sense, but when that mental shift happens the schedule produced becomes a series of pounding square pegs into round holes; lots of edges feel the rub.  At the same time, the water metaphor no longer works because a rigid line is needed in order to truly understand what works and what only seems to work (but, in the details, things are lost).

Elo Balanced Grading

How do you fairly measure two students of different academic ability?

One way is to use a rubric.  It’s cold and fair, giving a score on where each falls on the assignment.  Aligned with the Common Core and it objectively shows if a student is on grade level.  But, if you have heterogeneous classes, how can you push that scoring to better reflect progress, disincentivise coasting  and add more options to your differentiated offerings?

Elo ratings.

Think sports.  On a traditional scoreboard, one team earns a win and the other earns a loss–it does not matter if the teams are evenly matched, mismatched, or how close the score is.  After the match, one team is 1-0 and the other 0-1.  Those of us who have watched underdogs come close–or triumph–know that the score does not always reflect what happened on the field.  Elo weighs those factors and results.

The Elo rating system is a method for calculating the relative skill levels of players in competitor-versus-competitor games.  Created by Arpad Elo, a Hungarian-born American physics professor, it was originally used for chess.  It has been adapted for other competitive sports, including football and soccer.

In short, Elo ratings give a score to two competitors prior to a match based on their ability, and update scores based on new results.  Let’s say a poor team goes against a very good one.  Intuitively, we expect the poor team to lose.  If the poor team loses, their Elo score goes down, but not by much–no surprise in the results.  The very good team rises in Elo, but, also, not by much–again, no surprise.  We expect that outcome.

If, though, the poor team wins, they get a bunch of points in the Elo, and the good team loses a bunch.  The poor team earns more in a victory than the very good team because their victory required playing above their norm.

If you use a more advanced algorithm, Elo can also account for the closeness in a score.  If the poor team should have lost by a huge margin, but the game is close, their points lost are minimal–it recognizes they played above their norm.  Similarly, the points gained by the very good team are also minimal, even with a win, because it should have been a blow-out.

Student Scores, Assignments and Differentiation

In using a rubric with a scale of 1 to 4, the struggling student will inevitably earn a 2.  That score is accurate, but does not take into account risk, progress or the difficulty of the task.  Similarly, a high achieving student typically collects 3s and 4s.  No surprise.  It is the sports equivalent of being 0-1 or 1-0.

With Elo the struggling student can score against the assignment, earning more for tackling one more difficult–even if they fall short.  So, if a student who typically earns a 2 does well on a particularly difficult assignment, they might earn a personal score of 2.5.  In addition, if the high flyer decides to coast, their scores will not be as stellar as if they chose a harder assignment.  Perhaps, a 3.5 personal score instead of an automatic 4.

Not that the rubric should go out the window–that is the objective measure.  It would make sense to have two scores–a cold rubric to have a norm, plus an Elo personal score.  The former would be the equivalent of the scoreboard, the latter the classic Elo.  This personal score would show the struggle.

Many teachers already offer differentiated assignments.  Now, they can make it clear what each’s is score.  Students could choose based not only on interest, but challenge.

Even more advanced, teachers could offer their normal assignments and use student scores to rate the assignment’s difficulty–the scores of various students would weigh it in relation to other students.  There is a bit of bell-curve to this, but that’s why the Elo is for the personal score.

Elo Teacher Scoring

Another application is in balancing grades across varying teachers.

When a local high school implemented Proficiency Based Learning (PBL), and Proficiency Based Grading (PBG, aka SBG) it did not go smoothly.  Those teachers who embraced the formative/summative model tended to produce higher grades, on average, than teachers who clung to the older rating system.  At issue was the lead-up to assessments, supports and allowance of retakes in the PBL classes–with the emphasis on mastery, all that mattered was getting it in the end.  Those old-school teachers tended to use one-and-done assessments, average everything into the grade, dock points for late assignments, and use a 100 points scale.

Whatever your position on PBL and PBS the debate exposed the age-old debate of “easy” teachers vs. the old-school hard -ss.  As GPA becomes more and more important for college acceptance (even if that is only a teenager’s perception) many students are avoiding hard graders and hard classes because of real and perceived grading discrepancies.  That’s a crime.

But what if teachers were graded using the Elo?  Each class could be averaged for GPA and an Elo assigned to that teacher based on that average.  Then, when a student earns a grade in that course, a second Elo score is used to indicate relative difficulty of the teacher.  A more advanced algorithm might take into account individual student GPA in relation to each class’ earned grade.  And even adjust teacher Elo based on that.  If not on the transcript, it might at least give administrators and teachers an idea of grade inflation or deflation.  Correlated with other measures–SAT and the like–it might show which teachers make students earn their learning, and which are just difficult.

 

 

Hattie, Homework and an Insightful Analytical Blog Post (and Cautionary Tale)

Our school is struggling with the issue of homework.

Recently, we have moved away from it.  Some parents equate more homework with academic rigor and quality.  Of course, the data says otherwise–but also supports that community perception of the school goes up when the homework load goes up.  Now, the rallying call is for a research based policy on homework.

I often start with John Hattie and his file reviews and meta-analyses because the guy covers everything.  From his file reviews you can also find great original research worth pursuing.

Rather than rehash, let me guide folks to an insightful analysis of Hatties’ conclusions in “Homework: What does the Hattie research actually say?” on Tom Sherrington’s headguruteacher blog.  Sherrington does a great job looking beyond the overall effective rating of d = 029, which implies that homework is not an effective (Hattie argues that a practice is “effective” when above d = 0.40, and even that statement is a simplification of his work).  Note: Sherrington’s analysis begins a bit technical, a reflection of Hattie’s technical meta-analysis that he’s analyzing.  Wade through it–it’s worth it.

In short, Sherrington notes that Hattie reports that homework is ineffective at the primary grade level (d = 0.15), but quite effective at the secondary level (d = 0.64).  Of course, these results have caveats about type of work and the like.  Read the article, as Sherrington goes in-depth with the details and their implications.

But Sherrington’s analysis is also instructive when we look at bias in analysis and reporting.  Up front, he makes his pro-homework views clear and shares a link to another piece of his, ‘Homework Matters: Great Teachers set Great Homework’.  The analysis that follows is, as I’ve said, pretty solid.  Until the end.

All of this makes sense to me and none of it challenges my predisposition to be a massive advocate for homework.  The key is to think about the micro- level issues, not to lose all of that in a ridiculous averaging process.  Even at primary level, students are not all the same.  Older, more able students in Year 5/6 may well benefit from homework where kids in Year 2 may not.  Let’s not lose the trees for the wood!  Also, what Hattie shows is that educational inputs, processes and outcomes are all highly subjective human interactions.  Expecting these things to be reduced sensibly into scientifically absolute measured truths is absurd.  Ultimately, education is about values and attitudes and we need to see all research in that context.

Let’s ignore the statement “All of this makes sense to me and none of it challenges my predisposition…” which is the definition of denial (or perhaps cognitive dissidence).  You would think, at least, Sherrington would concede that at the primary level homework offers little benefit (remember: d = 0.15 vs. d = 0.40 entering the effective range).  No.  Instead, he qualifies, “Older, more able students in Year 5/6 may well benefit…”  That word: May.  Ugh.

Through such phrases a bus runs through.  After presenting, breaking down and analyzing the data he a) ignores the data that refutes his viewpoint, b) presents an alternative based on no data.  You cannot do that!  It’s not good science!  If Sherrington wants to use his theory as a basis for more research, great.  Instead, he simply argues that the data says one things, he believes another and so he’s going with his gut.  At least he’s transparent about it.

But I have been in countless meetings and conversations like this: Someone concedes that a practice is not effective and then justifies its continuation.  Parents and educators will often come with opinion pieces that seem like common sense, but with little data.  With a little research these views are often refuted, and sometimes quite harmful practices.  Yet, the idea will persist.  As professionals and organizations we refuse to trust data–or even consider it.

The posts to Sherrington’s piece are a typical reaction to any presentation of data that challenges orthodoxy.  Some accept it, but many qualify their views and use that data Hattie presents in an interpretive way.  Again, a good study in identifying bias.

My suggestion in such situations is to turn it around.  First of all, if some primary kids “may” benefit, might it also be said that some secondary kids “may not” benefit?  Do they get penalized?  What’s the plan for them?  Remember–when there is a majority, there is also a minority.  Why do we make policy when the majority agrees with us, but ignore the majority when they disagree?

Second, all choices have consequences: When you choose one thing, you are giving up another.  To have homework, students and teachers are giving up something.  That might be something simple like time–a student with thirty minutes of homework loses thirty minutes of play time, for example.  Is the loss worth the gain?  Hattie’s analysis seems to indicate that little is gained at the primary level, so any loss might not be worth it.  Educators should focus on the trade-off.  Is it worth it?

For Sherrington, he seems as interested in getting in the content and practice in the face of limited class time.  He is trading off the student’s time for academic gain.  That might be a fair trade-off, especially if the students are part of the decision making process.  But there might be other inefficiencies in Sherrington’s lesson planning that can be exploited (I have no idea) that a student might want addressed before they give up their after-school time.

Is it worth it?  Really, that’s the essence of all of this data analysis.  Sherrington needs to respect that data a bit more instead of dismissing it.

Agency Formula: Who Should Be in a Meeting

Our middle school group were once great decision makers.  We were inclusive, thoughtful and decisive.

We had meetings, but over time we stopped rotating roles, or having formal roles, or taking minutes.  Eventually, nearly all decisions were made in the hallway or through impromptu gatherings.  Meetings were where we formally approved already made decisions.  If that.

Then our staff turned over.  And, we were charged with including new agents in the decision making process.  Suddenly, a decision made by a handful of teachers required an administrator, case workers, specialists, and whatever aides could attend.  Minutes became something to approve instead of a never read document created by a teacher who needed to do something with her hands to keep focus.  People kept wanting to meet.  Emails called for quorum.

We became a case study in Parkinson’s Law.

But not every decision requires “the team”.  Other than the 30 minutes slotted each week, we just did not have the time.  Not everyone needed to chime in (although they did) when a decision had little bearing on their world (which was often).  The formality of “respecting each member” and “respecting the process” became a drag.

So, I created a formula to determine who really needs to be in on the process:

o x c = a

o = ownership. How How much does the decision affect you directly? Is this your program, or do you have to live with the results of the decision?
c = capital invested. This can mean money, time or other resources. For an administrator, it might mean respect or political capital. We often talk about the one who “has to do the heavy lifting.”
a = agency. Relevance. The less relevant you are to the outcome or the work involved, the less of a need you have to be in the meeting or vote on the decision.

For o and c you rank a person on a simple 1 to 10 scale. Don’t get precious with decimals, as a rough number works fine. The closer to 100 a person is the more their voice needs to be heard.

If you don’t want to employ it yourself, you can always silently calculate the agency score for everyone around the table. Then compare it to who talks the most. It’s good data when you complain later.