A little while back, Jonah Lehrer did a nice blog post about reasoning that used the famous study by Gilovich, Vallone and Tversky, The Hot Hand in Basketball (PDF link) as an example of a case where people don’t want to believe scientific results. The researchers found absolutely no statistical evidence of “hot” shooting– a player who had made his previous couple of shots was, if anything, slightly less likely to make the next one. Lehrer writes:
Why, then, do we believe in the hot hand? Confirmation bias is to blame. Once a player makes two shots in a row – an utterly unremarkable event – we start thinking about the possibility of a streak. Maybe he’s hot? Why isn’t he getting the ball? It’s at this point that our faulty reasoning mechanisms kick in, as we start ignoring the misses and focusing on the makes. In other words, we seek out evidence that confirms our suspicions of streakiness. The end result is that a mental fiction dominates our perception of the game.
Here’s where things get meta: Even though I know all about Tversky and Gilovich’s research – and fully believe the data – I still perceive the hot hand. I can’t help but watch the NBA playoffs and marvel at the streakiness of shooters, from Kobe to Rose. (Personally, I’d love to see an analysis of Ray Allen. If that man doesn’t show the hot hand, then it really doesn’t exist.) And I’m not alone in my stubborn skepticism. Red Auerbach, the legendary coach of the Celtics, reportedly responded to Tversky’s statistical analysis with a blunt dismissal. “So he makes a study,” Auerbach said. “I couldn’t care less.”
Elsewhere in blogdom, Paul Waldman looks for ways the study could be wrong, and Kevin Drum pokes fun at him for doing so, noting that “it’s interesting how unwilling most athletes are to accept the results of this study.”
It’s not hard to see why people who play basketball find this result surprising– shooting a basketball in a game situation is a complicated process involving lots of factors– balance, timing, sight lines, defense (though, to be fair, the study uses data from the NBA in the early 1980’s, so it’s probably safe to exclude the effects of defense…)– and it seems difficult to believe that all of those combine to give a single unchanging chance of success. And if you’ve played a lot of basketball, you know that there are days when one or more of those things just doesn’t feel right– where you’re rushing your shots for some reason, or the ball keeps slipping, or something like that.
The study itself looks pretty solid, though. There’s only one real weak point to it, that I see, which is that not all shots are equal.
By that I mean that they appear to have used a single number– the overall field goal shooting percentage– to characterize each player’s shooting, and looked for deviations from that. Anybody who has played the game, though, knows that different sorts of shots have different probabilities of success. Even great players tend to shoot a fairly low percentage from 3-point range– taking a look at this fan-generated list of great shooters, a lot of them are under 40% from three. Larry Bird shot 37.6% from three-point range for his career, and Reggie Miller 39.5%, for example, and Andrew Toney, the player in the study called out as an example of a “streak shooter” who wasn’t really shot just 34.2% from three.
All of those guys, though, have overall shooting percentages that are significantly higher– 49.6%, 47.1%, and 50.0%, respectively. That’s because they shot a considerably higher percentage from closer in– 50.9%, 51.6%, and 51.2%, respectively. So it can’t really be true that all shooting is described by a single percentage– there’s quite clearly a difference between different categories of shots, at least between very long shots and not-so-long shots. This is not factored into the statistical analysis of the “hot hand” paper, or if it is, they didn’t bother to tell anybody.
That leaves a tiny bit of a loophole for believers in the “hot hand,” especially when you take into account fan psychology. That is, when people talk about a player being “hot,” they don’t generally mean that he’s hitting lots of lay-ups. “Hot” in basketball circles generally refers to hitting a lot of shots for which the expected percentage is low. This is why if you go down lists of great shooters, you don’t find any dominant centers, despite the fact that guys like Wilt Chamberlain (career 54.0% shooting) or Kareem Abdul-Jabbar (career 55.9% shooting) blow away the shooting percentages of anybody on the list of great NBA shooters, percentage-wise. Those guys are shooting most of their shots from very close to the basket, often with a large size advantage over their opponents, and so they don’t tend to be deemed “hot” or “cold”– to the average fan, those are shots you’re expected to make.
A player is generally said to be “hot” after hitting either a couple of long jump shots in a row, or hitting some improbable-looking shots in the lane. Very few fans or commentators would describe a guy as having a hot night if he hit six open lay-ups in a row– instead, they’d likely talk about how bad the defense was. You’re judged to be “hot” when you hit a couple of three-pointers, or rattle in a couple of off-balance floaters in the lane.
(Similarly, free-throw shooting doesn’t really affect fan perception of “hot” or not– if an otherwise poor shooter hits several in a row, it will be remarked upon, but nobody has ever gone into work the day after a game and said “Man, Larry Bird was hot last night– did you see the way he hit all those free throws?”)
It’s possible, then, that the difference in types of shots could mask a real deviation from chance. That is, the tests that they did on NBA team statistics looked at total shooting percentage, and failed to see any evidence of “hot” shooting, but it could be that there really were cases where guys shot a higher percentage than would be expected for the subset of shots that “count” toward being “hot”, but the difference was masked by shot selection effects. A career 33% 3-point shooter going 6-for-9 might qualify as “hot,” but it could be masked statistically if he also went 5-for-10 from two-point range, putting him only a little above average.
They attempt to control for this by an additional experiment, in which players from the Cornell men’s and women’s varsity teams were asked to shoot from a distance where they shot about 50%, and they looked for correlations in their shooting. And, interestingly, this was the one area where they did find a correlation– player 9 on the men’s team showed a significantly higher probability of hitting the next shot after making a couple in a row (83% for a shot after making three in a row, compared to 54% overall).
Do I think this is the real explanation of the difference? No– I suspect that “people are bad at statistics” is the real explanation for the perception of “streak” shooting. If you were to repeat their analysis tracking 3-point shooting percentages separately, I doubt the results would be any different (in large part because it would be difficult to get the same level of statistical certainty– of the three players cited above, Reggie Miller was the only one to take more than 10% of his shots from long range. And characterizing all two-point shots by “degree of difficulty” would be way too difficult to undertake, unless you found yourself with a really obsessive graduate student who was also a basketball fan.
But the recent discussion (plus the fact that I’m taking a little time off from hoops, and find myself missing it) got me to read the paper, and that jumped out at me as the possible loophole, so I thought I might as well get a blog post out of the deal…
GILOVICH, T. (1985). The hot hand in basketball: On the misperception of random sequences*1 Cognitive Psychology, 17 (3), 295-314 DOI: 10.1016/0010-0285(85)90010-6
Actually I think that paper points up what is wrong with how some do statistical reasoning. It does constrain a few unrealistic models about shooting basketballs, but it does not limit any realistic models that include things like degree of difficulty of the shot, defensive adjustments, shooting adjustments (to change the percentage when one is cold). In fact it is fairly easy to create a more realistic model which has high correlations between shots based on type and defense (i.e. streaks) that has statistics that look just like the ones in the paper.
So the statistical power of the paper is there but the real use of statistics – to constrain the domain of possible -realistic- physical models to exclude something is asserted in the paper, but the choice of models excluded and the effort to use an actually realistic model of shooting and basketball play was fluffed and so I think the paper is a failure and does more harm than good.
This paper shows that bad model building trumps good correlations and means we don’t really know how important streakiness in shooting really is.
A very different sort of analysis looked at “momentum” in basketball, not at the level of individual shooters but at the level of team performance: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1279747/pdf/jaba00017-0143.pdf
They don’t quite agree with Gilovich et al. (you forgot Valone and Tversky), but there are enough differences in methodology that it’s a bit of an apples and oranges situation. Mostly, it supports your point that there are more ways the concept can be examined.
I think physiologically speaking, a “hot hand” is plausible. Shooting a basketball is a fairly complicated biomechanical event that requires many steps to go right. Anything that will cause a player to misfire on the steps can affect his ability to hit a shot and anything that allows him to focus better will help his ability to hit a shot. I think its plausible that on a given day, a player might feel more relaxed and thus shoot better. One of those things that will let a player relax can be hitting shots early in the game. One thing I would like to see is how these stats change based on score and time on the clock, as both of those increase the tension felt by the players.
I agree that the “he made a couple so he’s hot/he missed a couple so he’s cold” is a common fallacy among sports commentators, but on the other hand, back when I was playing (high school) I can recall times when the ball felt good in my hands – just the right amount of friction – and I felt that every shot could go in; as well as other times when the ball felt slippery or my timing was off and I had no idea whether any shot would hit. Once, in practice, I went to retrieve a ball from the bleachers, about halfway up, a place I had never shot from before, and I just knew from my grip on the ball that I could put the ball in from there, and did – nothing but net, as they say.
However, on the third hand, these days in infrequent pickup games the feeling is long gone, and I just try to concentrate on good mechanics. Sometimes they go in, and sometimes they don’t.
Coincidentally, Jason Terry made nine of ten three-point shots and Peja Stojakovic made six of six to close out the Lakers yesterday. They looked pretty hot to me. Paul Pierce looked good against the Heat the day before, also. Could be statistical flukes, I know.
@JimV
Peja and Terry were also WIDE OPEN. (Which relates to Chad’s point.) Good NBA shooters shoot a ridiculous percentage in shooting drills, which is essentially what those guys were doing yesterday.
Chad, there are more analysis tools available these days that could be combined into a more definitive hot hand study. For instance, there are stats sites that compare a team’s field goal percentage to its expected field goal percentage, based on where the shots were taken. This can tell you if a high FG% is due to the team getting good shots, or just getting lucky (or “hot,” if you insist) and hitting a bunch of long two-pointers.
Also, a study presented at this year’s Sloan Sports Analytics Conference (held at MIT, gaining mainstream notoriety) used video analysis tools to determine how the “openness” of a shot increased the chances it would go in. (Tight defense drops shooting percentages by 12 percentage points.)
Now, we just need someone to combine these results with a hot hand study, and we may be closer to the definitive answer (that no athletes will believe).
We could generalize this to other sports as well. What about hitting streaks in baseball? Or perhaps even more to the point, a pitcher who has overwhelming stuff on one day – http://articles.boston.com/2011-05-08/sports/29523168_1_second-no-hitter-blue-jays-justin-verlander – but gets shellacked a few starts later?
Could there be a second parameter for shooting percentage? something like the standard deviation from average. Intuitively, I think of a streaky shooter as one more likely than our hypothetical average shooter to shoot either better or worse than their average, whatever it is. So, if you have John Starks and Hubert Davis on your team, Starks is streakier and has game averages of 30-60% centering on 45%, where Davis always shoots at least 40% but never more than 50%. All numbers invented whole cloth.
Regardless of your methodology, you can’t evaluate the ‘hot streak’ claim based on one performance. I just did a quick model; based on Terry’s career 3-point percentage and the length of his career, one would expect to have seen one or two games like this from him (possibly none or three, very unlikely more than that) over the course of his career, if the success rate is entirely random. So looking at last night tells you nothing–you need to look at his career & see if he’s had more shooting nights like that than you’d expect. If he’s had, say, five or more, then we’d start questioning the null hypothesis.
If that was his career game, he did pick a good night to have it. 🙂 And yes, the Lakers forgetting how to defend on the perimeter didn’t hurt.
NBA “defense” in the early 80’s. Ha! Wave a cape and shout Ole! as your man drives the lane while everyone else on both teams takes a seat in the bleachers.
Wish I had thought of that while watching the Lakers self destruct over the past week.
I used to go bowling a lot. There are no defenders in bowling. It’s just you and the alley. Many years ago the stars aligned for one glorious summer and I averaged about 200. You have to get a lot of strikes to average 200. I knew my personal bowling mechanics very well, and I could feel it if I was a little bit off. I would try to correct things, and could feel It (the perfect stroke) getting closer and closer over a few frames, until, WHAM, I would roll a perfect strike, and then another, and then, another.
Or so it seemed. I kept detailed stats that summer, and it turned out that the probability of my getting a strike right after I had gotten a strike was the same as after I did NOT get a strike. No difference.
And yet, I STILL believe I could feel the hot hand.
I wish to take exception with the conclusion based on statistical analysis that there is no such thing as a “hot hand”. I think that such analysis ignores the very real factor of human variability. There are both psychological and physiological factors involved.
I am 70 years old. I play tennis, have been playing tennis for about 40 years. I am rated at slightly above average for my age.
When I play my regular weekly games (several times a week) with friends, on a familiar court, I sometimes get very comfortable. I feel like I can hit my forehand anywhere I want, into any small opening. And I do it repeatedly. I get hot.
However when I play in a league or a tournament with strangers, on a strange court, I have a lot of trouble executing that same shot. I get tight – literally. My shoulder does not release through the stroke as it should.
As any tennis commentator will tell you, everyone gets nervous at certain moments in a match. The difference is in whether you can learn to execute under that pressure.
All players are subject to pressure. Some can ignore it and execute the shot over and over just as it was practiced. When a player feels confident, does not tighten up, he can execute a shot many times successively – that is “hot”.
for some reason I thought of this:
[Rosencrantz and Guildenstern are riding horses down a path – they pause]
Rosencrantz: [to Guildenstern] Umm, uh…
[Guildenstern rides away, and Rosencrantz follows. Rosencrantz spots a gold coin on the ground]
Rosencrantz: [to horse] Whoa – whoa, whoa.
[Gets off horse and starts flipping the coin]
Rosencrantz: Hmmm. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads. Heads.
[Guildenstern grabs the coin, checks both sides, then tosses it back to Rosencrantz]
Rosencrantz: Heads.
[Guildenstern pulls a coin out of his own pocket and flips it]
Rosencrantz: Bet? Heads I win?
[Guildenstern looks at coin and tosses it to Rosencrantz]
Rosencrantz: Again? Heads
(later)
Rosencrantz: [flips coin which lands as ‘heads’] 78 in a row. A new record, I imagine.
Guildenstern: Is that what you imagine? A new record?
Rosencrantz: Well…
Guildenstern: No questions? Not a flicker of doubt?
Rosencrantz: I could be wrong.
(from Rosencrantz and Guildenstern are Dead)
The study also focussed on pro NBA basketballers ie the best of the best, and not on recreational players.
It seems plausible to me that, for the non-expert, it is more likely that the hot-hand is real than for experts. Experts have trodden the neural pathways to perform well on many occasions. Non-experts are more likely to be stumbling in and out of the right sequence of movements/mind-events to succeed.
Maybe the audience focusses on hot-hand in watching professional games because they are used to the experience of it in their own non-expert play, even though it doesn’t happen for experts.