Act 1: Damage Done

While the content of this blog appears to be directed at teachers, it is just as important to parents of children in public school. The intent is to illustrate an incorrect and invalid method of labeling teachers as “ineffective,” which, in turn, incorrectly and invalidly assesses your child.

The History
In the 2012 regular session, Louisiana legislators passed House Bill 974 which, when enacted, became Act 1. Many people refer to the passing of this bill as the beginning of the destruction of the teaching profession. The bill was a direct response to meeting the requirements to qualify for President Obama’s Race To The Top (RTTT) program. Among the most noted requirements were:

1. States had to adopt a common set of academic standards which would allow accurate comparisons between states. Louisiana had already adopted the Common Core State Standards.
2. States had to enact laws that tied teacher compensation to the performance of their students on state assessments. There could not be any laws on the books that prevented this.
3. States had to remove, or make it difficult to earn, tenure of public teachers.
4. States had to transfer some authorities from locally elected school boards to the locally employed superintendent.

Individually, these requirements aren’t monumental changes, but grouping them together completely changed the landscape of public education, and making the changes a requirement to receive RTTT funds was a coercive attempt to get states to adhere to an agenda. Louisiana received just over $12 million from the RTTT program.

The Details
Faced with the need to implement an evaluation plan that tied a teacher’s compensation to the achievement of their students, the creators of the bill welcomed, with open arms, the first person to step forward with a plan. Rayne Martin, at the time, was the executive director of Stand For Children, Louisiana (SFC).

SFC is a non-profit organization that started out as a small organization that focused on providing educational resources, housing and healthcare assistance, and other resources, to under-privileged students in urban areas. When the RTTT program was introduced, they applied and received a large grant for their programs, and then, applied and received millions of dollars from the Bill and Melinda Gates Foundation. From 2009, forward, they became a major catalyst for the sweeping reforms in states and currently operate in eleven states.

Fresh off of a successful run of reforms in Illinois, SFC was present in the development and passage of Act 1 from start to finish. Martin not only helped to devise the teacher evaluation plan, she spoke to the legislators throughout the process about the merits of the plan included in the bill. With no other plan to consider, and little to no knowledge about evaluating teachers, the bill passed with ease.

The irony of SFC’s involvement in this process is illustrated in a “white-paper” published in the following year and made available on their website. The paper, titled Education Policies and Practices and the Quality of the Teacher Workforce, makes the following statement.

“Any educator evaluation system will involve trade-offs between transparency and accuracy. For example, a transparent, easily understood performance measure that teachers may be more likely to trust could result in greater teacher effort. A more accurate model that is less transparent may be easier for teachers to dismiss as simply statistical mumbo jumbo. That said, policymakers may wish to exercise caution with some methods of translating student growth into teacher effectiveness estimates that do not measure confidence in the system (that is, no standard errors are associated with the estimate of teacher performance), as this may be legally problematic in cases where the performance measure is used as a factor in making high-stakes personnel decisions.”

What you will see in the following paragraphs is that SFC, now, strongly warns legislators about the problems with accuracy, and the legality, of using the very model that their Louisiana Executive Director successfully sold to the Louisiana Legislature.

The Evaluation
The evaluation model presented by Martin is what is referred to as the “Value-Added Assessment Model” aka VAM. The purpose and design of VAM is to measure a teacher’s influence and effect on the academic growth of “individual” students. Why? Because one of the premises of education reform is that given access to high-quality education, in a high-quality school, by a highly effective teacher; every student can learn.

The most common argument of those opposed to VAM is that no self-respecting, professional educator can argue that point. Of course, every student can learn; however, there are barriers to learning and achievement that exist outside of the classroom such as poverty, broken homes, lack of adequate healthcare and on, and on. What they don’t know is the VAM design includes an extremely complex formula that takes all of these things into consideration. A properly executed VAM can accomplish the following:

• Recognize that not all students are likely to make the same growth from year to year.
• With reasonable accuracy, isolate a teacher’s effectiveness on a single student.
• Take into consideration an individual student’s socio-economic status to predict growth.

Here is the formula that is used to accomplish this as explained in an excerpt from research performed by the Education Policy Center at Michigan State University. Click here to read the paper.

“Value added models attempt to model the achievement process over time and are based on the broad notion that achievement at any grade can be modeled as a function of both past and current child, family, and schooling inputs general formulation, the model can be expressed as:

Aig = fg(Eig, . . . , Ei0, Xig, . . . , Xi0, ci

where Aig is achievement of student i in grade g, Eig is a vector of educational inputs including teacher, school, and classroom characteristics, and in some cases a set of teacher indicators, Xig consists of a set of relevant time-varying student and family inputs, ci example, motivation, some notion of sustained ability, or some persistent behavioral or physical issue that affects achievement), and the uig is an idiosyncratic, time varying error term. In this very general formulation, the functional form is unspecified and can vary over time.”

Did you catch that? Probably not. It would likely take a statistician to understand the formula. Because of the complexity of the design, the process and assessment must meet a strict set of criteria. The research quoted above also asserts that the accuracy of  VAM is greatly reduced, and likely voided, when the following requirements aren’t met.

1. Students must remain in the same grouping from year to year.
2. Groups must be purposely placed with the lowest performing groups going to the most effective teachers.
3. Requires a starting point (pre-test) and a stopping point (post-test) to show growth.
4. Assessments should be proven to be valid and reliable to accurately assess growth over time.
5. Assessments should be developed and scored by test publishers who have the ability to accurate calculate the VAM formula.

Now, let’s look at these requirements for a bit. I can say, with fair certainty, that there are most likely no school districts in Louisiana meeting the first two requirements. With that in mind, I decided to survey all of the districts to see if any were not meeting the other requirements. By the time I had received input from six districts (which will not be named), it became apparent that there is a complete and total absence of the formula in their assessment plan, and even if it were present, the districts fail to meet most, if not all, of the requirements to maintain accuracy. Here are the most common problems found listed by the requirements above.

1. Non-existent.
2. Non-existent
3. In most districts, at the elementary and middle school levels, a pre-test and post-test is used. At the high school level, particularly in subjects requiring “end of course” testing, there is mostly an absence of the pre-test. Some districts, in some subjects, develop their own pre-test. Either way, in the absence of a pre-test, or when the pre and post tests aren’t produced by the same source with equal reliability and validity, there is no possible method to calculate student growth.
4. Only in high school EOC courses are there tests that meet this requirement; however, the lack of a pre-test makes it meaningless.
5. Again, only in some EOC courses at the high school level.

Going through the motions, while not meeting these requirements, when assessing a teacher’s effectiveness as it relates to an individual student is a complete and total waste of time. The VAM model is completely disfigured, and the teachers evaluated under this plan are unfairly rated. It is punitive in design, and has not only resulted in an enormous exodus of teachers from the profession, but also has resulted in the loss of jobs because of an ineffective rating.

Aside from not meeting the criteria above, there was one thing that I found most troubling in the responses and samples that were provided to me. Most districts are using an extremely simplified formula to calculate a teacher’s effectiveness, and in many cases, it doesn’t represent student growth, at all. Knowing this, I was shocked to learn that many districts are unable to correctly use their simplified formula. For example, in one high school level course where a “district created” pre-test and post-test were used, I found the following sample Student Learning Target (SLT) used to assess teacher effectiveness.

SLT: At the conclusion of the post-test, 70% of the students with 80% attendance rate will show 20% growth.

Now, from the same teacher, a sample of an actual score in the class. The score reflects the number of correct answers out of 30.

Student	PreTest	Target Growth	PostTest	Score Needed	Target Met?
John Doe	9/30%	20%	12/40%	15/50%	N

First, I’ll point out the problems with the SLT and how it doesn’t rate effectiveness. The 70% is misleading. What if the lowest 30% of the students actually decline, but the other 70% meets the goal? Is that teacher effective? No! The teacher is not addressing the needs of the students who need her the most, but the teacher met the expectation. Furthermore, if the purpose is to determine the effect on individual students, this does no such thing. This is an example of how a bad teacher could be rated as good by not evaluating the growth of each student.

Second, The group of students is identified as having an 80% attendance rate. Us this percentage rate, in a 175 day school year, a student can miss 35 days and still be included in the data. Act 1, as specified in Revised Statute 17:3902, states that any student with more than ten absences shall be excluded from the calculation. This is an example of how a good teacher can be rated as bad by including students with low attendance who likely score lower.

Last, but not least, I want to address the actual formula and calculation used. I want to stress that this is evidenced in every district and school that responded to my inquiry.

Student, John Doe, got 9 answers correct with a score of 30%. His target growth is 20%. The table indicates that to meet his target, he needs to score 50%, or score 15 correct answers which relies on the 100% scale as a point of reference. This calculation is incorrect. To improve 20% over his original score, he would only need to get 11 answers correct. This student scored 12 correct answers receiving a 40% which reflects a 33% growth which is more than enough to meet his goal. If he had actually met the goal indicated in the table, he would have made a 66% growth.

So, this should have every teacher wondering if they have been correctly evaluated. As I mentioned before, I don’t care to name the districts that provided information. The districts aren’t the problem…well, except for incorrect use of the simplified formula. The problem exists at the State level. A change needs to be made in the method of evaluating teacher effectiveness. There are better ways.

While I feel strongly that you cannot accurately measure a teacher’s effectiveness on an individual student, a teacher’s effectiveness can be measured in a manner that is much simpler than this, and certainly more accurate. In fact, any level of accuracy would be better than this.

In the next blog, I will illustrate a viable alternative method that needs to be considered by the legislature in order to move forward with the goal of improving education outcomes by improving the educational workforce, and as a result, restore some dignity to the teaching profession.