One of the standard education reform proposals that gets suggested every time somebody brings up the condition of American public education is that teachers should be offered some form of performance incentive, whether in the form of “merit pay” programs on a continuing basis, or bonuses for reaching particular targets. This is one of those ideas that economists swear ought to work, but a new RAND Corporation study found that, well, they don’t. This is based on a three-year pilot program run by the New York City school district, which offered selected districts bonuses of up to $3,000 per teacher for meeting certain school-wide performance targets.Taking the key findings from their research brief, we have:
The program did not improve student achievement at any grade level. The researchers found that the average mathematics and English language arts test scores of students from elementary, middle, and K-8 schools invited to participate in [the Schoolwide Performance Bonus Program] were lower than those of students from control schools during all three years of the experiment. However, the differences were very small and statistically significant only for mathematics in year 3 and were not significant when the researchers controlled for testing effects from multiple years and subjects. Similarly, researchers found no overall effects on state Regents Exam scores for high school students in the first two years (year 3 data were not available for analysis). The program’s effects did not differ among schools of different sizes or according to bonus award distribution plan.
The program also did not affect school Progress Report scores. Across all years and all categories of scores for the Progress Reports (environment, performance, progress, and additional credit), the researchers found no statistically significant differences between scores of treatment and control schools. The lack of effects held true for elementary, middle, K-8, and high schools.
The program did not affect teachers’ reported attitudes, perceptions, and behaviors. The researchers found no differences between the reported practices and opinions of teachers in treatment schools and those of the control group. The survey responses about instructional practices, effort, participation in professional development, mobility, and attitudes from the two groups were very similar, with no statistically significant differences. Furthermore, the vast majority of teachers who received bonuses said that the bonus did not affect their performance.
That’s pretty much the definition of total failure. So, what’s going on here? Some miscellaneous items from looking through the report (the full study is free to download):
— One of the suggested reasons why this didn’t pan out is that the bonus was not seen as sufficiently valuable. The level of $3,000 per teacher (“teacher” here meaning “teacher represented by the local union,” which cooperated with the study) is generally acknowledged to be not that impressive. The report quotes a district official saying:
We] looked at the literature . . . and that number [$3,000] . . . is [about] fve percent of [the salary of] a fourth-year or ffth-year teacher. And that’s all we could aford. In an ideal world, it would have been more. I think there’s a little bit of literature about this that says to really change behavior–and that’s the only reason you would do something like this–that number needs to be bigger. But [we] thought intuitively that that number was big enough to have some meaning.
(page 37 of the report, page 75 of the PDF). A teacher’s union official described it as “the minimal amount we thought was necessary.” So, you could argue that the system would work, if only the bonuses were bigger. This quickly gets into the problem of where the money is going to come from, though.
— The money was disbursed to schools for meeting school-wide targets for test scores and the like, and then distributed to teachers within the schools according to a plan developed by a Compensation Committee for each school. The committees consisted of the school principal, another person chosen by the principal, and two people chosen by the teachers of the school. The committees were free to distribute the bonus money equally or not, but were forbidden from distributing it based on seniority. The majority of the participating schools distributed the money more or less evenly among all teachers in the school. Only 60 of the 198 districts included some measure of individual performance as a factor, leading to higher awards for some teachers and lower awards for others.
You could argue that the somewhat communal nature of the whole process reduced the incentive for individual teachers to improve their own performance, since they got the same amount of bonus money for being just good enough for their school to meet its target as for being really outstanding. This is pretty weak, though, as the schools that included individual performance in their distribution plans didn’t do any better or worse than the schools that spread the money around evenly.
(You can also argue that giving more money to the people who were already really good is exactly the wrong thing to do if you want to improve results. The way to make schools better as a whole is to make the worst performers do a better job, because they’re the ones pulling down the average.)
— I really like this bland summary of one possible reason for the program’s failure:
Finally, the lack of observed results could have been due to the low motivational value of the bonus relative to other accountability incentives that applied to all the schools. Many teachers and other staff acknowledged that other accountability pressures, such as receiving high Progress Report grades or achieving Adequate Yearly Progress targets, held the same if not greater motivational value as the possibility of receiving a financial bonus. While the bonus might have been another factor motivating SPBP staff members to work hard or change their practices, they would probably have been similarly motivated without it because of the high level of accountability pressure on all schools and their staffs.
This basically translates to “The extra money is nice, but we’re already working as hard as we can.” Which shouldn’t actually come as a surprise to anyone. This isn’t an accidental quirk, it’s the central principle of American public education: the people who go into education as a career are not the ones who are strongly motivated by money. If they were, they’d go into some other field, because education doesn’t pay very well.
“But just two bullet points up, you said they were making $60,000 on average,” you complain, being sharp enough to notice that if $3,000 is 5% of a fifth-year teacher’s salary, they must be making $60,000 a year. “$60,000 a year is pretty good.”
Yes and no. Keep in mind, this is $60,000 a year in New York City, so that doesn’t go as far as $60,000 a year a lot of other places. Also keep in mind that that’s the salary for a person with four years of experience, with a Master’s degree. That’s generally two years of education beyond a four-year college degree, which is only one year less than a law degree. A lawyer with four years’ experience would usually be making considerably more than that.
And there isn’t a huge amount of room for advancement. That $60,000 figure is pretty close to the median value for most NYC schools, according to these data tables, and the 95th percentile in those districts is under $100,000. Someone who really cares about making as much money as possible would be a fool to go into education.
So, in the end, the results of the study aren’t terribly surprising. Offering performance bonuses doesn’t magically produce huge improvements, because the people who choose teaching as a career aren’t in it to maximize their personal wealth. They’re motivated by other factors, and already working plenty hard.
Is this inconsistent with my long-standing contention that teachers ought to be paid more? No, because the whole point is that if you paid teachers more, you might draw in a different selection of people, some of whom would be more capable than many of the people now teaching in public schools. There are people out there who could be great teachers, but who don’t consider education as a career because of the money.
Anyway, there’s a whole bunch of data in this study, which I confidently predict will make absolutely no difference. Those who favor performance pay as an education reform solution will find enough loopholes in the test program (not enough bonus money, not enough individual incentive, not enough union-bashing) to write it off as meaningless. Those who think the problem is more complicated than anything that can be fixed with a little bonus money will nod along happily, but we didn’t need to be convinced of anything in the first place.
Well, if the problem is that the 3000 dollar bonus isn’t enough incentive since it only represents 5% of their annual base pay, the obvious solution is to make that 3000 dollar bonus a larger percentage of their salary. [/conservative]
I didn’t hit the “$3000 is five per cent” level until I had been teaching for 31 years. Money is not a big motivating factor for teachers, especially those just starting out. I nearly cried when my daughter went into teaching. She and her husband are both working, just had a son, and would be very hard pressed to get by without a ton of support (mostly child-care and auto repair) from family members.
I hope that first comment was tongue in cheek (hee-hee).
Mad Mike has had some posts which call into question whether teacher performance metrics in use in this country have any meaning. He pointed to one case where the metric came out for a particular teacher as “much worse than average”, with the confidence limits ranging from “slightly above average” to “worse than useless”. It’s possible that the experiment failed because the assessment metrics aren’t precise enough to measure a non-null result.
Of course, according to Dan Pink – performance based incentives only work for very low level tasks.
Here is his TED Talk – pretty interesting.
http://www.ted.com/talks/dan_pink_on_motivation.html
Not only might higher teacher salaries attract a larger and better pool of candidates, it would help with retention of teachers as well. Even smart, talented people need some experience in the classroom to become effective teachers, and even if they didn’t get into for the money, once they figure out that it’s going to be difficult to manage kids and a mortgage on a teacher’s salary, too often the best start looking elsewhere.
Match the low salaries with the increasing amount of top down micromanagement, and retention becomes even more difficult. Would you stay in a job that required a masters, didn’t pay well, and scripted what you taught?
My mother has been a middle school science teacher for about 15 years. She has a master’s degree, a multi-subject and two single subject(science and math) teaching credentials. She has much of the knowledge and skills that we are trying to attract into schools. Yes, she “only” works 10 months out of the year. But during those 10 months, she is at school 7am-5pm most days, if not later, with a 25 minute lunch (which is often mostly taken up working with students). Most school days and on weekends, she brings home papers to grade, and plans her lessons on her own time. During her summers, she takes classes and goes to conferences, paid for out of her own pocket, to make her a better teacher. And she makes less money than I do, at 33 years old, working in private industry.
What all these incentive programs fail to realize is how much of education is out of the teachers’ hands. At my mother’s school, a large percentage of the children live in poverty (around 75% qualify for free or reduced priced lunches), and the majority speak English as a second language (most of those do not speak English at home). There is very little parental support or involvement. Many of the children come into the school woefully unprepared for the expected level of learning, and do not have the basic supplies to perform the required schoolwork – she spends a lot of time teaching skills they already should have learned, which means there is less time to teach the things they are expected to learn for the year. The administration refuses to back up the teachers on matters of discipline or consequences for failure to meet academic expectations. Add onto this hostility from students and parents toward the subject matter (she teaches science). Many of the kids are simply not interested in being at school, and are not swayed by poor grades or other negative consequences. Class sizes are usually between 35 and 40 students. Supplies and equipment are severely limited – many teachers spend a good amount of money every year to have the necessary supplies in the classroom. My mother teachers physical science – if she wants to do labs, she has to buy the supplies, because there is no money in the budget for it. They are constantly expected to do more, with less and less funding.
Under these conditions, it is no wonder good teachers are leaving the profession. No matter how great the teacher is, or how much you are paying him or her, without addressing these issues, achievement is not going to improve significantly. Penalizing teachers for poor test score performance (or rewarding them for high scores) does not address the real reasons for low achievement – many of the reasons for low achievement are out of their control. Instead of addressing issues of funding, poverty, and early education, and organizational issues that hinder success, the teachers are blamed and punished.
Two thoughts:
1. Always bothers me when we try a program, but do it half-ass, then study it and find out it didn’t work. The conclusion that survives is just that it doesn’t work generally, when in fact, it wasn’t tried.
2. It also occurs to me that we have some nonlinearities at play here, in that the effect should depend upon the original level. By this I mean that we attract to the teaching profession now a mix of the following: people who care about education rather than money, and people who want summers off. It’s unlikely that these people — who have already made their career choice based on factors other than money — are going to suddenly become very sensitive to a monetary bonus.
If, instead, we already used higher pay to attract higher caliber teaching candidates, and then offered them performance bonuses, those bonuses might induce the desired behaviors more effectively.
I have to mirror Jen’s comment. Too many factors are outside the teacher’s control. Mike the Mad Biologist covers a lot of these factors regularly – poverty among students is easily the best predictor of classroom performance, for example.
It’s also worth noting that most of these measures of teacher performance are simply not good statistical measures, and have way too much variability to really amount to beans.
Wait a minute. The report says that all scores went DOWN in the schools with the bonus system in place, and the drop was statistically significant for math after three years?
That would mean this change is actually a bad one, and some states are going to make their schools worse … if you take those data as seriously as the proponents expected to.
Weird, weird, weird. “If the teacher worked harder, then the kids would be smarter. ” No … if the *kids* worked harder, they would be smarter. These economists seem to model everything as a sausage machine, with a handle that can simply be cranked faster.
Weird, weird, weird. “If the teacher worked harder, then the kids would be smarter. ” No … if the *kids* worked harder, they would be smarter. Maybe. These economists seem to model everything as a sausage machine, with a handle that can simply be cranked faster.
It doesn’t seem that ridiculous to me. I mean, the teachers are not the only place where “working harder” would change education, but surely teachers that spend more time working with students and preparing etc. are “working harder” and surely the results of that extra effort appear in classrooms.
Also it’s pretty much the only way the govt. can really tackle the problem. A lot of it rests on the parents, and a lot of it rests on the students. Getting students to care more has always been a problem, one that I don’t think anyone has solved. Teachers are likely the only angle of approach that is really practical.
(Again, “working harder” is not only for teachers and not all teachers can work much “harder” than they do now. Other typical disclaimers go here)