Weak instruments are weak

More than a decade ago, back in the advanced econometrics course at George Mason, Thomas Stratmann taught us about the weak instruments problem. Looks like one of these was worse than I'd thought.

Suppose you wanted to know whether education affected earnings. And you saw that people with more education had higher incomes. But suppose that you also thought that people who were smarter (or more conscientious) would earn more regardless of education AND that those same people were more likely to select into more education. You then have an endogeneity problem: just running a wage regression with education on the right hand side doesn't really give you the effect of education on earnings. It tells you how much more those with more education earn compared to those with less education, but it's a total effect rather than a partial: it includes all the effect of selecting into more education as well as the effect of the education.

So, then, what to do? You need something that will affect the amount of education that somebody gets that isn't itself a predictor of income. Something exogenous to income that affects education. If you have that, you can use an instrumental variables approach to try to estimate the effect of education on income via the effect of the exogenous instrument on education.

And the typical early example is the old Angrist and Krueger measure of quarter of birth as instrument for education. Compulsory schooling laws in the US meant you couldn't quit school until you were some minimum age. Quarter of birth could then affect how much schooling you got: those who turned 16 early in the school year could drop out earlier than those who turned 16 later in the school year.

Then Prof Stratmann told us about the weak instruments problem. Quarter of birth is only really weakly correlated with education - it explains very little of actual differences in education levels. And so bad stuff can happen in the estimation.

So we've known for a while that quarter of birth might not be a great instrument. But it would still get used. Bound, Jaeger and Baker complained about it in 1995; I think we went through their critique in our 1999 metrics class. Jaeger got angry in 2009 when quarter of birth was still being cited as a decent instrument.

Fast-forward to today's news from the Freakonomics blog: Buckles and Hungerman have shown that season of birth is endogenous. (gated, ungated) Ka-boom.

Season of birth is associated with later outcomes; what drives this association remains unclear. We consider a new explanation: variation in maternal characteristics. We document large changes in maternal characteristics for births throughout the year; winter births are disproportionately realized by teenagers and the unmarried. Family background controls explain nearly half of season-of-birth’s relation to adult outcomes. Seasonality in maternal characteristics is driven by women trying to conceive; we find no seasonality among unwanted births. Prior seasonality-in-fertility research focuses on conditions at conception; here, expected conditions at birth drive variation in maternal characteristics, while conditions at conception are unimportant.

If different types of parents select different seasons in which to have kids, then there goes the exclusion restriction: IV just plum don't work right if the instrument affects outcomes via channels other than the one specified. I think that Dubner at Freakonomics understates the importance here when he says the finding's interesting because so many studies have found an association between birth month and later outcomes: the importance is rather that we have something much more serious than a weak instruments problem when using quarter of birth to get around endogeneity problems in estimating the effect of schooling on income.

@KevinMilligan points out that folks have moved toward regression discontinuity design for this kind of work anyway.

@colbycosh Certaily hurts QofBirth IVs, but most stuff now uses tight discontinuities. Season of birth effects no prob if trend smoothly.
— Kevin Milligan (@kevinmilligan) August 12, 2013

Agreed! But I still wonder whether loud cries of "WE TOLD YOU SO!" were coming from Jaeger, Bound or Baker's offices when the Buckles and Hungerman paper came out....

Update: I'm going to give the last word on this one to Dr. Phil:

Finding a solid IV is a grueling task. Make it a little more fun by turning it into a scavenger hunt for the kids.
— Dr.Phil of Economics (@DrPhilofEconomi) August 13, 2013

Minimum prices and mortality risk

The latest Stockwell piece on alcohol minimum pricing and alcohol-related fatalities seemed a bit fishy.

I was mostly worried about how they ran a panel study that had zero cross-sectional variation in their main regressor of interest and where the main source of time series variation was CPI adjustments to measured prices, but it looked like there were plenty of other holes for truck-driving expeditions.

George Mason statistician Rebecca Goldin goes for a scenic tour, noting another rather serious problem. Standard t-statistics don't really do the job if you're pouring over multiple lags to look for effects. She writes [ht Forbes]:

What’s troubling here is that they break the data down into many quarters and categories, run multiple statistical tests, but don’t adjust for multiple testing. This results in a table spotted with statistically significant results even as basic statistics tells us this method will produce spurious results.

A close look at the table is suggestive that spurious results are indeed at hand. This table looks at 16 quarters following a minimal price increase, and whether there is a correlated increase or decrease in deaths among acute, chronic or wholly attributable alcohol deaths. The authors point to a statistically significant decrease in wholly attributable deaths in the quarter that a price increase was implemented, as well as in the second and third subsequent quarters. (but not in the first quarter, nor the 4th-15th quarters).

But it also shows a significant increase in acute deaths in the third and fifth quarters after a price increase, and then a statistically significant decrease in acute deaths in the 8th quarter after the price increase. Chronic deaths saw statistically significant decreases in the 8th, 9th and 13th quarter after a price increase, but not in the other 13 quarters. This all suggests that the results could be, at least in part, the result of simply running a lot of tests on a lot of data – and without adjusting for multiple tests, randomness can creep in. Though the results do lean toward decreased death, picking out the most extreme of the results (as the media and the authors of this study did) may be misleading. In fact, if the 1,388 wholly alcohol attributable deaths occurred evenly over the quarters, these numbers refer to trends in about 87 deaths each quarter – trends that would be highly sensitive to a small number of deaths.

So while there seems to be an overall trend of decreased death with increased prices, the failure to account for multiple testing means there could be true correlation or there could be just a statistical fluke.

Eventually, somebody's going to take a proper shotgun to the stuff Stockwell's been up to. When Chris Auld was doing their econometrics, I didn't worry about their empirical results. Things seem to have gone a bit adrift since then.

Trusting Secret Data: Dunedin edition

Unless you've run the regressions yourself, it's often hard to trust empirical results. A lot of results are fragile - small changes in specifications, either changing date ranges or adding seemingly irrelevant variables, can change results. And endogeneity always makes inference hard in social sciences.

First best is running things yourself. If the data and code are available for replication and extension, then it's impossible to maintain outright fraud. And fragile results will eventually be commonly known to be fragile. Boing Boing recently ran a very nice summary in two parts on why we see so many contradictory results on guns and crime: all the specifications are fragile. On some questions, teh science isn't strong enough to give much policy direction.

When the data is not publicly available, we really have to trust the people running the specifications. There can be good reasons for keeping data private. Most self-interestedly, if the researchers did a ton of work in putting together the dataset, they might well want to get a few papers based on it before letting everyone else pile on. It's not an edifying reason, but even if the data collection involves public funding, it doesn't seem unreasonable to give the survey team the first few kicks at the can. On the other hand, if the data cannot be shared without violating the privacy of individuals who answered the surveys, and if there aren't clever ways of anonymising the data so that it cannot be used to identify individuals, then that's a pretty good reason to be stingy with the data. 

But, there potentially being good reasons for secret data certainly isn't sufficient reason to trust the consequent analysis. And so we come to Dunedin vs Rogeberg.

The Dunedin Longitudinal Survey is secret data, likely for the good reason of wishing to maintain the anonymity of survey respondents. We really have to trust the people running regressions on secret data. The Dunedin group recently found that early marijuana use predicts IQ decline. Ole Rogeberg wondered whether the results were due to cohort selection effects: the kids most likely to try marijuana early could well have had different results even had marijuana not existed. Rogeberg provided Dunedin a list of tests that might sort things out. He summarises things here:

I’ll start with a short recap: Researchers published article august 2012 arguing that adolescent-onset cannabis smoking harms adolescent brains and causes IQ to decline. I responded with an article available here arguing that their methods were insufficient to establish a causal link, and that non-cognitive traits (ambition, self-control, personality, interests etc) would influence risks of adolescent-onset cannabis use while also potentially altering IQs by influencing your education, occupation, choice of peers etc. For various reasons, I argued that this could show up in their data as IQ-trends that differed by socioeconomic status (SES), and suggested a number of analyses that would help clarify whether their effect was biased due to confounding and selection-effects. In a reply this week (gated, I think), the researchers show that there is no systematic IQ-trend difference across three SES groups they’ve constructed. However, as I note in my reply (available here), they still fail to tell us how different the groups of cannabis users (never users, adolescent-onset users with long history of dependence etc) were on other dimensions, and they still fail to control for non-cognitive factors and early childhood experiences in any of the ways I proposed. In fact, none of the data or analyses that my article asked for have been provided, and the researchers conclude with a puzzling claim that randomized clinical trials only show “potential” effects while observational studies are needed to show “whether cannabis actually is impairing cognition in the real world and how much.”

I'll be interested in seeing the Dunedin group's reply when it comes out. Rogeberg points to evidence the Dunedin group themselves have provided showing reasonable cohort differences between users and non-users. It's not implausible that these kinds of differences are responsible for at least some of their measured marijuana effect. It would be simplest if Dunedin would send Rogeberg the data and make him sign a data confidentiality waiver; it's not particularly plausible that a Norwegian labour econometrician wants the data for identifying NZ individuals. But if they're not willing to do that, they should at least be willing to put his code to their data. 

I'll be downgrading my trust in all the Dunedin Longitudinal results if they don't handle this well; secret data requires trust.

There is away around all of these kinds of problems though.

The General Social Survey in the US would have almost as many privacy concerns as the Dunedin survey. And yet they're able to make available an online resource letting anybody with a web browser run analysis on the data, for free. I regularly set assignments in my undergrad public choice class where students without much in way of metrics background have to go and muck about in the data.

As I understand things, the Health Research Council funds the Dunedin surveys. There's a worldwide movement toward open data in government-funded projects. HRC could fund a Dunedin Longitudinal equivalent of the GSS browser analytics. Anything personally identifying, like census meshblock, could be culled. Nobody would see the individual observations. And any cross-tab that would reduce to too-small a number of observations could return nulls. But folks with concerns about Dunedin studies could do first-cut checks without having access to the bits that might cause legitimate worries about privacy. Somebody like Rogeberg should be able to run a t-test on whether those who later go on to report marijuana use differed on other important variables prior to their starting consumption.

We all could trust Dunedin results more if we could check things like this. And it shouldn't be hard to set this up either: see how SDA coded things for GSS, put in the Dunedin data instead, then host it. If HRC did this, one small bit of funding could have a whole ton of researchers publishing useful studies based on the already-funded survey instead of having to fund Dunedin for every new regression. Let the Dunedin group keep any new wave of the survey under wraps for a couple of years before releasing it to the online analytics so they get a fair kick at the can. After that, why not open things up?