LAUSD Presents A “School Performance Framework” To Parents, While Failing To Notify Any About It
10 Wednesday Jul 2019
Tags
Mathiness, School Performance Framework, School rankings, SPF, Summative assessment, value-added models, VAM, Yelp-style ratings
Share it
Two Aprils ago the District resolved to “Establish A Framework For Continuous Improvement” of its diverse complex of approximately one thousand schools, including traditional, charter, special education, special programming, etc.
The idea was to ‘identify and track a uniform set of measures for each school’s overall annual performance, that would enable district and parents alike to understand and evaluate school performance’.
A little off schedule (about nine months late), two trios of Educational consultants converged from Washington, DC (“Collaborative Commnications”) and Madison, WI (“Education Analytics”) to present some trial dashboard-like screens of the new School Performance Framework (SPF) to a collection of parents assembled during four daytime sessions, still ongoing this week (click here to attend).
The trouble is that at least for Tuesday’s session, fewer than a dozen parents were present, and the preponderance of these had heard of the meeting through “word-of-mouth”, via a shallow pool of friends and acquaintances demographically similar essentially by definition; not directly by invitation or message from the District. Questioned by a parent about outreach sharply, a District official explained that only parents already engaged with elite, District-level councils were notified during this holiday period, and the meetings were not advertised or managed even on LAUSD’s website. The demographic represented directly curiously reflected just those special-interest lobbying groups originally invoked (@1:27.07) 15 months previously at the time of the Resolution’s hasty passage: Parent Revolution and SpeakUp. Conspicuously absent were parents not previously actively-engaged with the District, or of a perspective not actively-curated.
The imported consultants took the temperature of potential parent-users regarding various select metrics {e.g. standardized test scores; school suspension rates; reclassification of English-language-learners, etc}; summary baskets holding a suite of these metrics {e.g., “school climate”; “growth”; “college/career readiness”}, and the overall mixture of these summary baskets {e.g. 40% contributed by basket A, 25% contributed by basket B, etc}. The resulting weighted composite scores are used to compute a single, yelp-like, five-star rating.
Because complex ecosystems like a school can be collapsed into a single, summative rating just like that. Because inherently non-comparable metrics like, say, reclassification rates of non-native speakers; and, say, student fitness; and, say, academic achievement among schools with significantly different proportions of special education students – can be manipulated onto a single quantitative scale and termed comparable thereby. Because the individuality of every unique learning child can be subsumed in a heterogeneous generalization that erases needs and personality: just because.
Most LAUSD metrics are by and large available already at the State’s Department of Education. And those that are not could simply be released by the District without the associated qualitative interpretation, color-coding, star-rating or ranking.
Because the artificial constructs of market baskets, and the subsequent weighting of these, all contribute dramatically to unstable summative interpretations. When these translate to serious choices or incumbent “high-stakes” like choosing a school, labeling failure or censuring a teacher, equivocal conclusions assume a false command. The American Statistical Association is very clear about Value-Added Assessment (VAA aka VAM) in this rare, 2014 public clarification: “VAMs should be viewed within the context of quality improvement, which distinguishes aspects of quality that can be attributed to the system from those that can be attributed to individual teachers, teacher preparation programs, or schools” [emphasis added].
Please read the whole paper: it is excellent, terse and should be heeded well. VAM by any other name – VAA, summative assessment, SPF – is a fraught technique with a ghastly history. We must never forget the horrible sequelae to the LA Times’ ingenuous embrace of its false promise. Artificially bolstering support for SPF from among astro-turf parent groups, camouflaging its pre-ordained, inexorable imposition by nominally conforming to policy mandates for stakeholder engagement with inadequately noticed or meaningfully heeded “parent focus groups”, serves absolutely no one well at all. Not students, not teachers, not the district, not statistical science.
Fact-based, quantitatively-evidenced science is the only kind that there is. But substituting a superficial mathy facsimile for the complex socio-developmental process of Education, is to betray our children’s educational prerogative to high-priced, commercial, partisan stakeholders.
12 Comments
July 13, 2019 at 5:04 pm
Kathy: Thanks for commenting, for showing up and thinking for yourself at all of these events whether elite, schoolsite or otherwise. Thank you, even, for agreeing with me sometimes! ;)
We’re all stakeholders; we’re all taxpayers. Parents who are also teachers; parents without papers and parents with PhDs. Parents with graduated kids and potential-parents who never had kids.
Who’s the “realest” parent aside, let’s focus on net need, and net outcome, for this proposed “tool”, the School Performance Framework (SPF).
Several distinct needs seem to have been expressed*, for (1) a single, “summative” assessment of a school and (2) information broken down by subcategories regarding performance of, say, specific sub-schools or -categories or demographics under one roof, say, magnet school, or SAS school, or English-language-learners, FRPM-students, AfricanAmerican students, etc.
The claim is that the State’s dashboard is confusing or insufficient.
The claim is that parents can and will — or variously cannot and will not — do the hard numeric evaluation of sifting through all this data and making decisions on their own.
There is a claim at the board level that the relative priority this sort of sifting reflects, should properly (but not, perhaps, uniquely) be asserted by our elected school board, hence the weighting schema.
And there is also that claim of need for One Single Number to rule them all, one number that summarizes a school’s relative worth. And the need for that was explicitly related to a “just parent’s” inability or unwillingness to come up with that number on their own, via their own resources, own personal network of associates, and in light of the perceived imperfections of the State’s dashboard.
From my point of view, which is as a former-LAUSD parent’s and biostatistician’s (and engaged educator of sorts), first, I am skeptical of the claimed need of several of these points. The State (CDE) dashboard exists and yet it is not the only tool by any means from the State with data and statistics. There are ready-prepared reports, there is DataQuest that can be tailored to the school level for a large variety of variables, reports can be tailor-made. There is a School Accountability Report Card (SARC) for each school, all this data is available for download and public-facing manipulation as well. If it is too complicated, it is not clear how or why LAUSD can or could do a better job, with the same consultants tasked with creating the State’s system. Why not engage actively in collaborating to improve the user-friendliness of that already-established centralized repository of data?
But what I suspect is really bemoaned as complicated is in fact the comparison, and not the extraction of numbers.
This matters because it is not just hard, there are limitations to the exercise. A relative comparison, carefully, may be observed, but that is distinct from an absolute determination.
So when a Single Number is desired to boil down an entity defined by the intersection of numerous factors — that cannot be done, technically. Unfortunately, perhaps — debatably sadly. But though you can force a computer to cough out a single number, that does not make it accurate. And also critically, it does not make the interpretation of that number valid.
What if I were curious about my experience exercising at an indoor track? Let’s consider the cost of my shoes: $100. Let’s consider how far I run: 1 mile. Let’s consider my heartrate: 95 beats/minute on average. Let’s consider the cost of the track’s construction: dunno, $12,365 per mile (totally made up), let’s consider the physical comfort of the track: 4 on a scale of 1-5. Now let’s construct my experience: {100+1+95+12365+4=12565}. Obviously these are different metrics with different meaning and scales so simply summing them means nothing. But forcing them into a single, “normalized” scale with technically comparable units doesn’t add meaning to a single number. It can be done, but it is not meaningful. More, constructing a second of these metrics for a different person running at a track one town over doesn’t improve the meaning: two wrongs don’t make a right. Just because we can do this does not mean we should; just because there is an overwhelming need or desire to does not justify it. The specific metrics can be compared between tracks, but not absolutely rated; they can be ranked but without any absolute — only comparative — meaning.
So then the question becomes: what happens when a measure is inaptly constructed, and subsequently misused. Whose fault is that? Should the desire to use the metric inappropriately be restricted or is that a violation of rights or freedom, etc? Who chooses?
Because as Dr. McKenna said at the 4/10/18 board meeting: ” … It may not be your intent [to label and stigmatize], but the harm will still be done”.
We may all want some one thing, but just wanting it doesn’t mean we can do it or get it. I would like to see world peace out there. It’s not happening. I would like to see malnutrition eliminated worldwide. Politically this is an insurmountable problem, though technically it is feasible. We may want a single number to explain one school, and we can generate that number. We can even fool ourselves into believing it describes the school accurately, but it does not.
Therefore, what does it mean to demand this “simplifying” tool that we cannot technically or even in good faith generate? It means your need is so compelling (and purse so high) that eventually you will find someone who will tell you they have constructed this number, and in your denial, you are willing to believe it is so.
It means you must elevate the imperative of that claimed need uncritically. It means being unwilling to accept that what is being presented is not as claimed.
It’s not a question of whose parent-profile is more valid. It’s a question of what can be done, what needs to be done, and who tells you what.
I take the perspective of the professionals who derived the numeric models for these comparisons. It’s here, or at least what’s here is from the society professionally tasked with interpreting the stratagem: https://tinyurl.com/ASAonVAM
I don’t take the perspective of the ideology that states because someone wants it badly enough let’s make it so even if only by perception.
That may or may not be what a “just parent” wants. It’s my experience that “just parents” can and do, in large numbers, understand the trickiness of “fit” and assessment and bring a variety of tools to bear in constructing their own. Adopting someone else’s priorities is dogmatic, technically questionable, and denies the prerogative of stakeholder’s individuality – for fit, priority, worthiness, voice. It is anti-democratic and it is counter productive toward sustaining a system of excellence for all, across the system.
*And I must disagree with you for several reasons that “two ‘just parents’ didn’t even get a chance to speak”? First, the facilitator in my opinion was highly skilled and professional, seemingly intent on allowing the conversation to be even. I do not think she would, and I do not think she did, allow domination of the conversation. The overall track of the conversation definitely did not go as according to plan, but she was highly cognizant of this, explicitly acknowledged and accepted as much. Second, I do not even recall anyone present who refrained from commenting; I believe every individual spoke, but some focused intensely on different issues. Last, I know no one to make such an effort at engagement (as you say, to the Valley, mid-week) and then deferentially opt out of, um, engaging. I don’t believe that happened, I saw no suggestion that it did, and have no indication it would have been tolerated, either.
July 13, 2019 at 9:58 am
Sure, people, including myself, occasionally get published in community newspapers, because: Democracy.
You bemoan LAUSD’s failure to notify parents about an input session — and your proposed fix? Round up a group of Teachers! (who also happen to be parents) and former LAUSD parents who are highly educated, then dominate the discussion with jargon and professional concerns. While doing so, the Latina woman and her husband, who arrived a bit late, never had an opportunity to speak.
The “just a parent” comment differentiates between parents who are also employees, and parents affiliated with community-based organizations whose funding reflects a particular political agenda. None of the invited parents at that session qualify as such. The District actually did a good job reflecting the demographics of the District at that meeting — there were parents of students in Special Education and 3 Latinx parents.
A current LAUSD parent’s comments rubbed you the wrong way? Oh well. She is a Latina mom, and precisely the type of “Prospective Parent User” of the draft SPF that this session was intended for. Her lovely HS, nestled in the Reseda Rancho area, where houses sit on big lots and homeowners keep horses — this HS appears on the CDE List of Shame of those schools whose students in particular subgroups (English Language Learners and Students With Disabilities in this case) perform among the 5% lowest in CA. This school, in our lovely Valley community, needs to do better. It was formerly my daughter’s school, and it remains my friend’s school. We don’t complain just for the fun of it. We have a personal interest in wanting to see improvement.
Why, you ask, is there a need for any SPF tool at all? Parents can just go to the CDE website and scroll through that bewildering list of every school in the whole State. Prospective Parent Users should suck it up and do their own darn research! Wow, I scrolled through that list; trying to identify the two subgroup categories this school fails to serve well was really difficult to sort out. With perseverance I finally did. It’s not parent friendly.
Yet you try to position yourselves as advocates for parents?
July 13, 2019 at 1:20 pm
For many years now, I have recognized the futility of public comment in fora such as this, but your total misrepresentation of what happened at that meeting compelled me to make the comment to which you now responded.
Alas, I should not have bothered because your reply is more of the same: total fabrication and spin.
I suspect your mind is already made up, and, therefore, this reply is more for the benefit of the readers of these rants, er, comments, than to establish a dialog. It is probably best that such a dialog take place face to face because words on a page can seem so harsh. Given that, please read these comments as a factual response and not an attack on your persona. I will attempt to respond to your points as well as I can:
It is disingenuous to say that getting published is Democracy. It is not. Rather, it is access to media for the select few. You and your co-authors are not simple folk and anyone who knows how op-eds get generated and published know of what I speak.
Personally, I don’t bemoan anything. I am aware that access is granted to those who seek it and/or cultivate it. Of the four self-identified current and former teachers who were present at the meeting we attended, two have children and one has a grand-child attending LAUSD classrooms. Their children will be part of this experiment. Are you suggesting they have no right to be there?
Nobody rounded these teachers up, certainly not UTLA. They heard it through the grapevine and chose to attend. You seem to believe they have no place at the table because they know too much. Are you serious? Should you be shown the door because your professional expertise on LCSW informs you of what goes into school climate issues?
As for that Latina woman, she certainly had every opportunity to ask questions. All she had to do was raise her hand. Please remember that she questioned the PCS representative fairly thoroughly.
You claim that the District did a good job reflecting the demographics of the District. Indeed, there were at least three parents whose children currently receive Special Education Services. However, both you and your friend were once members of the Parent Advisory Council and you yourself were a very active member of the Community Advisory Committee. The other Latina woman was also a member for years of the Community Advisory Council. Clearly, those invited are or were members of “high-level District Committees.” The demographics you speak of are clearly accidental.
Please don’t misrepresent what I have written and stop pretending that your friend is a “regular” parent. You chose to complain that a certain person continuously insisted on discussing technical aspects. However, you conveniently failed to note that your friend, a public relations professional, skillfully monopolized the conversation towards her own view point. Of course both of you have personal interests and that is why you were there. Why then insist that others don’t have the same right?
Please stop making things up. I do not question a need for an SPF. I question how it is being implemented because it will not answer any of the questions you want answered. In fact, if you burrow into it, you will find that Special Education issues will not matter in the end (and its counterpart, Gifted, are not even included). Why is that? Because any correction/manipulation to these students’ scores will be buried by the averaging process designed to give you that single number describing the school’s growth and/or achievement. Worse, if they were to create a separate number for the population you are interested, the sample size will likely be too small to be meaningful. Yet, you choose to believe the data wranglers’ words. Indeed, that is your right. But it is also my right to point out that the Emperor has no clothes, and, believe it or not, help you get what you want: recognition of appropriate programs.
For the record, I do not pretend to be a parent advocate. I am, instead, an advocate for the community as a whole, not just my particular social or economic group. I suspect that after your last child leaves LAUSD you will be in the same position as I am, still advocating for what you passionately believe is best for children. Or will you meekly accept being voted out of the island?
Lastly, it is counterproductive to continue this “conversation” here. Feel free to have the last word.
July 12, 2019 at 3:28 pm
Kathy complains that she did not get heard.
Maybe Kathy didn’t because her friend kept driving her own agenda (“I just want a single number so I can tell which school is best for my child… but I also want that number to celebrate the excellent SAS program at my child’s school”).
Kathy complains that there were not enough “true” parents. Whose fault is that? Besides, the invitation stated that “We are seeking to engage with a variety of parents, principals, and community stakeholders in the process to ensure this tool meets the needs of our communities.” I am a part of the community and am certainly entitled to be informed and question the process.
As for Kathy being “just a parent volunteer,” no, that’s not entirely true. She has been a member of at least two LAUSD committees. Those are indeed elite committees. LAUSD brass might not listen to them, but they are primus inter pares.
Kathy has also been a vocal antagonist of teachers and a supporter of value added measures for a long time. She is well acquainted with Academic Growth over Time, SPF’s previous incarnation. The fact Kathy kept quiet when the same nostrums were trotted out is indicative that she has not changed one bit even though she no longer shares her thoughts as freely as she once did (e.g., https://patch.com/california/encino/letter-to-the-editor-a-call-for-common-sense )
Wait, I take that back: here is something more recent http://laschoolreport.com/lausds-parent-committee-volunteers-we-need-someone-who-knows-the-district/
Both of those writings demonstrate Kathy is certainly able to contribute to the din, wade into the politics, and expertly sling the jargon.
Just a parent volunteer, indeed.
July 12, 2019 at 11:25 am
Not trying to put guardrails around who should or shouldn’t have attended that elite “Parent Input Session” at LDNW (btw, thanks for traveling to the Valley mid-week). Yes, LAUSD needs to do a better job with parent engagement so it’s not just the same 15 people showing up.
However, most of those notified by “word of mouth” that day turned out to be parents, or former parents, who are also teachers. Is that true parent engagement? After all, UTLA ensures that teachers are a part of all the district-convened task forces they want to be involved with. Teachers and admins are also welcomed on the CAC; members submit an application to join, it is hardly elite.
The majority of the other parents in attendance that day — the ones who are “just parents” — were the ones who most accurately reflect the demographics of the school district. Two of them didn’t even get a chance to speak. What I saw was a highly educated group of folks dominating the conversation with concerns about quantitative vs. qualitative data, summative ratings, “fact-based, quantitatively-evidenced science,” and lord help us, concerns that principals might use data to ask schools to improve.
Speaking for myself, I am in no way part of some “special interest lobbying group,” “astro-turf parent group,” nor am I a “high-priced, commercial, partisan stakeholder.” I’m just a parent volunteer. None of the parents in attendance that day belong to any such group.
It is true that, before it launched, I attended several Speak Up meetings. I heard they were forming to represent parents and decided to check it out. When I concluded it wasn’t being run in a way that I believed reflected the true parent voice, I stepped away. People change their minds sometimes.
However, I don’t think your merry band of underdogs reflected the true parent voice, either. Perhaps lumping us parents with “astro turf groups” makes us seem less threatening. As an informed and involved parent — a parent who shows up — I think for myself. Nobody pays me. Nobody manages me. Nobody organizes me. My opinions are my own; sometimes I even agree with you.
As “just a parent,” I know it is very hard to get heard above the din, the politics, and the jargon.
July 10, 2019 at 7:18 pm
The meeting location has meeting auditorium that can accommodate at least 60. We were forced into having our Special Education Community Advisory Committee meetings there when Sharyn Howell allowed PCSB to takeover “facilitation”. It caused participation to plummet and PCSB personnel bullied us and tried to control our agenda – all without being members or officers or even connected to Special Education in any way. PCSB is an LAUSD-created entity with absolutely no authority over the State and Federally mandated committee. We should not have been handed off so disrespectfully. We are accountable to the Board of Education and every single member listened to attorneys telling them to keep out of it instead of helping us when we were having so many problems.
Anything that happens at PCSB is not parent driven, it’s LAUSD driven.
July 10, 2019 at 10:02 am
Not sure I agree they’re “idiots” or “stupid” — strategic is more like it. And I’m not loving the results of an informal survey of parents on District-level cmtes – I’ve encountered not a one so far who has received notification.
That aside, I seriously doubt they will turn away parents who just show up. I could be wrong of course, but that would reflect poorly in a way that the meeting cap would not. I can understand trying to manage size logistically. But managing interest is another matter.
July 10, 2019 at 9:57 am
Now those idiots have capped the meeting at 15. A great representation of families!!!! They don’t want feedback, they want to satisfy some overlord that they held a meeting and now they can call it “input”. They are so stupid!