Weak instruments are weak

More than a decade ago, back in the advanced econometrics course at George Mason, Thomas Stratmann taught us about the weak instruments problem. Looks like one of these was worse than I'd thought.

Suppose you wanted to know whether education affected earnings. And you saw that people with more education had higher incomes. But suppose that you also thought that people who were smarter (or more conscientious) would earn more regardless of education AND that those same people were more likely to select into more education. You then have an endogeneity problem: just running a wage regression with education on the right hand side doesn't really give you the effect of education on earnings. It tells you how much more those with more education earn compared to those with less education, but it's a total effect rather than a partial: it includes all the effect of selecting into more education as well as the effect of the education.

So, then, what to do? You need something that will affect the amount of education that somebody gets that isn't itself a predictor of income. Something exogenous to income that affects education. If you have that, you can use an instrumental variables approach to try to estimate the effect of education on income via the effect of the exogenous instrument on education.

And the typical early example is the old Angrist and Krueger measure of quarter of birth as instrument for education. Compulsory schooling laws in the US meant you couldn't quit school until you were some minimum age. Quarter of birth could then affect how much schooling you got: those who turned 16 early in the school year could drop out earlier than those who turned 16 later in the school year.

Then Prof Stratmann told us about the weak instruments problem. Quarter of birth is only really weakly correlated with education - it explains very little of actual differences in education levels. And so bad stuff can happen in the estimation.

So we've known for a while that quarter of birth might not be a great instrument. But it would still get used. Bound, Jaeger and Baker complained about it in 1995; I think we went through their critique in our 1999 metrics class. Jaeger got angry in 2009 when quarter of birth was still being cited as a decent instrument.

Fast-forward to today's news from the Freakonomics blog: Buckles and Hungerman have shown that season of birth is endogenous. (gated, ungated) Ka-boom.

Season of birth is associated with later outcomes; what drives this association remains unclear. We consider a new explanation: variation in maternal characteristics. We document large changes in maternal characteristics for births throughout the year; winter births are disproportionately realized by teenagers and the unmarried. Family background controls explain nearly half of season-of-birth’s relation to adult outcomes. Seasonality in maternal characteristics is driven by women trying to conceive; we find no seasonality among unwanted births. Prior seasonality-in-fertility research focuses on conditions at conception; here, expected conditions at birth drive variation in maternal characteristics, while conditions at conception are unimportant.

If different types of parents select different seasons in which to have kids, then there goes the exclusion restriction: IV just plum don't work right if the instrument affects outcomes via channels other than the one specified. I think that Dubner at Freakonomics understates the importance here when he says the finding's interesting because so many studies have found an association between birth month and later outcomes: the importance is rather that we have something much more serious than a weak instruments problem when using quarter of birth to get around endogeneity problems in estimating the effect of schooling on income.

@KevinMilligan points out that folks have moved toward regression discontinuity design for this kind of work anyway.

@colbycosh Certaily hurts QofBirth IVs, but most stuff now uses tight discontinuities. Season of birth effects no prob if trend smoothly.
— Kevin Milligan (@kevinmilligan) August 12, 2013

Agreed! But I still wonder whether loud cries of "WE TOLD YOU SO!" were coming from Jaeger, Bound or Baker's offices when the Buckles and Hungerman paper came out....

Update: I'm going to give the last word on this one to Dr. Phil:

Finding a solid IV is a grueling task. Make it a little more fun by turning it into a scavenger hunt for the kids.
— Dr.Phil of Economics (@DrPhilofEconomi) August 13, 2013

Shame elasticity of demand

Here's a fun one. New Zealand supermarkets have started putting in self-checkouts. Turns out customers buying embarrassing or personal products are more likely to choose those checkouts than ones staffed by clerks.

Pak 'n Save Moorhouse Ave was the first supermarket in New Zealand to have them installed, in 2006.

Foodstuffs chief executive Steve Anderson said 20 to 30 per cent of transactions were now through self-checkouts. However, they accounted for a far lower proportion of customer spending.

Customer research showed people who used the self-checkouts tended to buy fewer items and were often mothers with children, the technology-minded or introverted.

"Embarrassing products", such as personal or indulgent items, were more commonly bought at the self-checkout.

Anderson said there had been some savings for the business, but "nothing major". It had not saved on employee numbers. "The idea was more about customer choice and convenience," he said. "You still need staff there." [emphasis added]

So here's a potential IV study for somebody. If people are more likely to buy condoms when these kinds of self-checkouts are introduced, and there's no correlation between self-checkout installation and underlying sexual behaviour trends, you could use the availability of self-checkouts as an instrument for condom use. Ta dah!

It would likely be a weak instrument, alas; the correlation isn't likely to be that high.