Where do they all go?

Neal Caren's put together a beautiful little infographic showing the flow of graduates from various US majors into different occupations, using data from the last two waves of the American Community Survey.

So we can see that of the 173,849 graduates in Economics:

• 37,043 became accountants and auditors
• 32,303 became financial managers
• 23,872 became chief executives
• 36,842 became managers 
• 43,789 became lawyers

So just under 14% of the kids graduating in economics became CEOs. Now there were 160,706 people identified as chief executives in the dataset. So 14% of econ grads became just under 15% of all CEOs.  Accountants were also overrepresented among CEOs. General business majors were underrepresented among CEOs; they were more likely to be retail salespersons or first-line supervisors of non-retail sales workers than to hit the head office. 

Alas, the chart isn't great for major-selection purposes. We can't see the proportion from each major reporting unemployment, for example. But it can help with Avenue Q's question. What can you do with a B.A. in English? By plurality: go on to teach at elementary and middle schools. 

HT: Fab Rojas

Signalling and education

U Vic's student magazine asked me for comment on signalling and degree inflation; Wilbur Townsend and Nick Cross's story is now out.

I agree with Nick that there has been degree inflation in New Zealand; most folks around the traps suggest the big push happened in the 1990s. It's largely leveled off since then. Or, at least at Canterbury, I've not seen much evidence of it since I've been here. It could pick up again once the full effects of the various Universities' new taught masters programmes start coming through.

Here they channel Bryan Caplan:

Why would people get a tertiary education if it doesn’t add value? We believe it is about sending a signal to potential employers. The ‘signalling model of education’ holds that the wage premium from having a tertiary degree comes from the signal of a person’s abilities relative to others. It’s based on the observation that most formal tertiary education is taught by professors with little experience outside the ivory tower, and that most courses are concerned with academic theory and not tailored to specific occupations. Why would these be any use in the real world? Research into the ‘transfer of learning’ suggests that education does little to actually improve intelligence or critical-thinking skills in the long run. Consider the way you approach your own education. Most of us spend weeks cramming for our final exams, but aren’t at all worried if we forget much of the material the second the exam is finished. Most of us rejoice when a class is cancelled because it means we have less to learn. But there is no proportionate reduction in fees for the cancelled class—shouldn’t we be outraged? If education is about building human capital, why isn’t our focus on accumulating as much knowledge as possible? If it’s about signalling to employers, we can understand that the most important thing to come out of our education is the piece of paper at the other end.

It's also been fun seeing a bit of evidence of degree inflation in the data. In the main US work on political knowledge, income matters far less than education in predicting political knowledge. But, here in New Zealand, they both seem to matter. A lot of very successful people here got their start when all you needed was a high school degree, and so household income is the better proxy for intelligence for that cohort. The main bout of degree inflation in America happened decades earlier.

Here's what I sent Nick, which he of course needed to truncate to leave room for others:

“Degree inflation is real, but signalling isn’t everything that’s going on in education. The signalling model says that the value of education mostly comes from the little piece of paper certifying not that the graduate has learned anything but rather that the graduate is the kind of person who is able to put up with a few years of grinding pointless work without quitting, can complete assignments on time, and is smart enough to have made it through. While it would be easy to provide cheaper signals of intelligence, there isn’t a lower-cost way of signalling the ability to put up with years of grind-your-way-through assignments. So it isn’t implausible that much of what goes on at University is providing that signal. But that can hardly be everything: there are wide differences in salary for graduates with different degrees, and at least some of that is due to the specific things taught in those degrees.” 

“At least at Canterbury in Economics over the last decade, I have not seen great evidence of degree inflation. In a degree-inflation model, we would have a higher proportion of students progressing from the Bachelors to the Honours degree, and then more moving on from the Honours degree to a Masters, but little change in where the graduates wound up. But about the same proportion of our majoring students go on to pursue an Honours degree now as was the case a decade ago, we haven’t really changed the standards for entering Honours, and our Honours students place as well at Treasury, the Reserve Bank, the bureaus, and the trading banks as they ever have. I expect that this will have changed in a decade’s time: a lot of places are repackaging Honours degrees as Masters. If employers only pierce the credentialing veil imperfectly, then we could wind up with Masters being the new Honours. While we have substantial degree inflation in the overall population as compared to, say, fifty years ago, I don’t think there’s been much change over the last decade. When I got here, it took an Honours degree to land a decent entry position at Treasury; that’s still the case now.” 

“The problem with the signalling equilibrium is that anyone not pursuing a University degree is lumped in with everyone who would not be able to complete one if they tried. And so employers infer something about an applicant’s likely abilities from that he hasn’t gone to University. The only real way of solving the problem would be to reduce the number of funded places at University while providing ample scholarships to students of very high ability but limited means, but it’s not immediately clear that this is better than what we currently have. It’s not crazy to think that New Zealand could be better off if, say, a quarter of the students currently coming to University for business degrees went instead to the polytechs for vocational training. I’ve not seen any conclusive case for or against it, but it’s not immediately implausible.”

The introduction forms a bit of a depressing assessment of New Zealand's intellectual environment.

Canterbury University economist Eric Crampton is the closest thing New Zealand has to a ‘public intellectual’. He blogs at offsettingbehaviour.blogspot.com, and regularly appears in New Zealand media to bring an economic perspective to issues such as alcohol regulation, breakfast in schools and the Christchurch earthquake recovery.

I'm not sure that's entirely accurate. Or, rather, the NZ Herald last year reckoned there were at least a couple dozen of us. 

The provocative conclusion, for a student magazine, I'm sure will draw interesting letters to the editor.

Breakfast

A few months ago, Social Service Providers Aotearoa asked me to review the literature on school breakfast programmes and provide an assessment of whether public funding of school breakfast programmes offered value for money. I spoke on the issue in Wellington and in Christchurch in February. As the government seems to be looking at the Mana Party's proposals around food in schools, it seems worth posting things here as summary.

I was only looking at school breakfast programmes, and so I can't here comment on school lunch programmes. I'm not sure why we'd expect results to vary greatly, but it's worth having the caveat.

Anyway, on my best read of the literature, it's hard to make a case for that we'd get any great benefit from the programmes. Rather, we often find that they don't even increase the odds that kids eat breakfast at all. Many shift breakfast from at-home to at-school, but among those who hadn't bothered with breakfast before the programme, not many wind up starting when schools provide it. You can then get kids reporting that they're less hungry as consequence of the programmes, but it's awfully hard to reject that the main thing going on is that kids are eating at 9 at school instead of at 7 at home and are consequently less hungry when asked at 11.

You can get some substantial results from school breakfast programmes in third world countries. But even there we need to watch for displacement effects: the benefit of the programmes is often the implicit income subsidy provided. In those cases, we can see evidence of families cutting back on food expenditures for the kid getting breakfast at school in favour of spending on the other kids; in the link provided, there's reasonable crowding out in a UK lunch programme. And if that's the benefit, cutting a cheque to the families instead just might be better.

In all the studies, I wish that there were a control group where the parents were just given cash equivalent to the per-student cost of putting on the programme. All of these kinds of programmes should be assessed against that kind of counterfactual to establish whether we're getting benefits from the programme, or from the implicit income transfer.

Here are a few typical pieces.

Devaney and Fraker, 1989, found that school breakfast programmes did not increase the likelihood of kids' eating breakfast at all. It did increase calcium intake and reduce consumption of cholesterol and iron - breakfasts provided at school differed from those they'd be getting at home.

Gleason, 1995, similarly found that school breakfast programmes did not influence the likelihood of students' eating breakfast.

Alderman and Bundy, 2011, concluded that food in schools isn't a great investment but could complement other investments - they focused on developing countries.

Bhattacharya, Currie and Haider, 2006, seems to be the touchstone for those advocating school breakfast programmes. They found improved nutritional outcomes in blood serum tests of kids participating in school breakfast programmes compared to the same kids during school holidays when they weren't getting the school breakfasts. But they also found no effect on the likelihood of eating breakfast. And I worry a bit about their identification strategy: because it's poorer schools who got school breakfast programmes, we might expect that there could be relevant differences in how parents respond to school holidays that might affect the difference between school/not school outcomes for reasons other than the programme.

Waehrer, 2008, in an unpublished study funded by the USDA's RIDGE programme, found that school breakfast participation reduced the likelihood of eating breakfast. We could imagine this happening where the kids don't really want breakfast anyway, the parents stop making them eat it at home because there's the programme at school, and then they skip it when they get to school. The study could have similar identification issues to the Bhattacharya piece noted above; they identify on weekday-weekend differences, but cohorts might respond differently to weekends.

Shemilt, Harvey, Shepstone et al, 2004, found pretty mixed outcomes in a messy randomised control trial. They wound up abandoning the RCT part of the analysis and just going for regressions. They found some evidence of worsened outcomes of having attended school breakfast programmes on a few behavioural measures, but I'm again not convinced that they've pinned down causality. What they seemed most sure of was that school breakfast programmes had kids eating more fruit, so I guess there's that.

There were a couple of pieces claiming reasonable benefits from school breakfast programmes too.

Powell, Walker, et al, 1998, ran a really nice randomised control trial in Jamaica. Kids in the programme got breakfast, those not in the programme were given a small piece of orange. So they're able to isolate socialisation effects from breakfast effects. They found that the treatment group saw small increases in nutritional status, achievement, and attendance; they suggested that "greater improvements may occur in more undernourished populations." I'm not convinced that we're in that category.

Murphy, Pagano et al (1998) found that moving from selective to universal school breakfast programmes had some benefits, but also had some odd results. Before intervention, "hungry and at-risk children were slightly, but not significantly, more likely to participate in the school breakfast program than nonhungry children", and that more than half of the hungry and at-risk kids rarely or never participated in voluntary school breakfast programmes. So stigma associated with voluntary programmes can substantially affect uptake. But, when the programmes were made universal, hungry and at-risk kids were only "somewhat more likely to increase their school breakfast participation than non-hungry children... although this difference was not statistically significant." So what do we then make of results showing some improved average outcomes at school but no particular increase in breakfast-eating among those who are hungry? I wonder if all the effects here point to that eating later in the morning rather than earlier is better. I'll talk more about this below.

Dotter, 2012, finds that universal in-class school breakfasts increase the number of children eating breakfast at school compared to voluntary programmes that could have stigma effects, but I couldn't see that the paper measured whether there was an effect on total breakfast consumption. And while Dotter finds increased school performance in schools with universal school breakfast programmes, I can't see how the paper distinguishes between an "eating at all" and an "eating later" effect. Why does this matter? Imagine an alternative policy where schools allow a designated morning tea break at 10:30 where kids bring in their own snacks. This would be cheaper than full school breakfast programmes and just as effective, if the main channel of effectiveness is having a fuller tummy at the time of instruction because breakfast was later.

Frisvold, 2012, found that state mandates requiring schools to provide school breakfast programmes increase availability of those programmes and consequently increase test scores: the paper reports math score increases of nine percent of a standard deviation and reading score increases of five percent of a standard deviation. Again, there is no significant effect on the total days per week that a student eats breakfast, suggesting substantial displacement of breakfasts that would otherwise have been eaten at home. The paper claims that the effect is through a nutrition channel, with kids eating healthier breakfasts. But I can't see how they're distinguishing the nutrition channel from my suggested "they're eating later in the morning and so are less hungry at 11" channel.

So, some bottom lines:

• School breakfast programmes really don't seem to increase the likelihood of that kids eat breakfast at all;
• To the extent that they improve outcomes in some studies, we really can't tell:
• whether the effect is from changing the timing of breakfast, in which case we should instead have a morning tea break;
• whether the effect is any better than just giving those families an equivalent cash transfer.

I spent an hour in Wellington and Christchurch walking through these findings. I hope we don't throw a pile of money at school breakfast programmes; the money could well be better spent. That also seemed to be the conclusion of a New Zealand study: Mhurchu et al, 2012, who found that the only effect of a randomised control trial of school free breakfast programmes here was that kids self-reported being less hungry.

Update: Lindsay Mitchell points to a presentation on the New Zealand trial. She also points to data showing child poverty rates have been dropping.

STEM shortages

The American econo-blogosphere has been wondering whether there really is any shortage of workers in Science, Technology, Engineering and Maths. Dan Kuehn and coauthors at the Economic Policy Institute suggest that the absence of particular wage inflation for graduates in STEM disciplines means there are no particular shortages of STEM workers relative to other workers; Tyler Cowen notes that this may be evidence of a shortage of complementary inputs for those workers. Kuehn agrees, but notes that policy then should be targeting those complementary inputs rather than boosting STEM numbers.

The American policy debate differs from the Kiwi one. There, they're talking about increased immigration for STEM workers via H1-B visas; EPI doesn't think they're needed. Here, we're wondering how much emphasis the tertiary education system should put on STEM disciplines relative to the others.

I've done a bit of digging around in NZ Stat; unfortunately, the publicly available data doesn't seem sufficiently disaggregated to say much about relative shortages. Median weekly income for all occupations went up by 6.7% from 2009 to 2012. Professionals' incomes increased by 9.4%; technicians and trade workers' income went up 4.9%; community and personal service workers' median weekly income went up 12.7%; managers' median weekly income went up 14.4%. By broad category, there doesn't seem to be much evidence of the kinds of salary ramp-ups that would be consistent with strong shortages in STEM disciplines. If anything, skilled managers seem to be in shorter relative supply. It could be that the technicians category is too broad and is missing out on very large increases in science technician salaries; maybe the 'scientist' category is a high-wage-growth lump within the broader professional category. The ANZSIC classifications are sufficiently finely grained, but I'd have to make the data request.

But suppose that we found STEM shortages here evidenced by large salary discrepancies. While salary ramp-ups can be evidence that we need to make it easier to import skilled workers, they're not really sufficient basis for increased government spending on STEM training. We'd rather expect that higher salaries would draw capable students into STEM subjects and away from disciplines with worse employment prospects. If there were binding constraints built into the funding system, like caps on the maximum number of students who could be funded to pursue those disciplines, then those caps could be revisited. But instead it seems the government wants to pour more money per student into STEM training. Where the gains from STEM training are internalised in higher wages, I'm not sure that's such a great play.

The best case for encouraging STEM workers over others is if those disciplines generate greater social benefits relative to the worker's wages;* maybe you could make some productivity spillover case for it. But even in that case, we ought to be intervening where the intervention can have best effect. If tech workers require a lot of complementary inputs and are mobile, does it make more sense to highly subsidise STEM disciplines in our universities and hope that the New Zealand labour market finds use for them, or to subsidise the wages of qualified STEM workers, or to subsidise the outputs that the government thinks actually generate whatever positive externalities they think come from technological innovation? If our workers are mobile and if there are agglomeration effects in play, then other countries will always be able to outbid us for STEM workers graduating here; I'm less than convinced that large funding boosts for STEM disciplines are the best way of achieving whatever the government's trying to achieve. If it wants more scientific research, funding lab outputs directly or establishing prizes for great new innovations bids up the wages of STEM workers most likely to produce those innovations and also brings in the complementary investments.

If the story is asymmetric information about job prospects and salaries post-University leading to inefficient student choices among disciplines, that seems already adequately addressed by government initiatives to highlight employment prospects for various degrees.

* The alternative case would work as follows. Government wants to maximise its total net tax take. If it thinks that young peoples' time preferences are too high relative to the government's time preferences, or even just if it discounts non-wage elements in students' utility functions, it might want to push people onto lower-welfare-but-higher-lifetime-earnings paths. That's more an Olson despot, NZ Inc. model. If government were maximising a broader social welfare function, it would put more weight on the externality case and look for areas where social benefits are high, wages are low, out-migration is unlikely, and where increased provision can be more efficiently ensured with training rather than wage/output subsidies. Another alternative case: it's a second-best solution when they're constrained to provide student loans at zero percent no matter what discipline the student chooses while repayments are keyed to income.

Exempting the elderly

New Zealand and Christchurch having been a bit depressing lately, other than last night's rather nice 77-44 vote in favour of allowing same sex marriages in addition to civil unions, it is sanity-enhancing to look a bit at lunacy elsewhere.

While New Zealand funds education from general taxation revenues and ensures that schools in needier areas receive enhanced funding, Manitoba, and much of North America, mainly funds its schools out of local property taxes. There are potentially good Tiebout reasons for doing this as some people prefer paying more in taxes for better schools, it also makes it harder for schools in poor areas to provide decent services.

From 2015, Manitoba seniors will be exempt from the education portion of the property tax bill. Says Finance Minister Stan Struthers, "These seniors who paid a lot of taxes over their working careers as they raised families, I think they deserve a break."

Let's leave to one side for now the obvious equity problem here that letting the relatively wealthy elderly off the hook for a bundle of services that they don't currently consume might well invite relatively poorer young people to request exemptions from the portion of their taxes that go to pay for government services never consumed by young people: the Canada Pension Plan, Medicare coverage for Altzheimers', government-funded old folks' homes, the Royal Canadian Air Farce... I'd be pretty surprised if net government funding didn't already strongly favour the wrinklies.

The bigger problem is the long-term fiscal outlook when the political incentive is to pull back from taxing the wealthy elderly. Now there are reasons why property taxes on the elderly can be politically unpalatable. Suppose that you've owned your house for 60 years, its value is bid up by a bunch of families coming in because of the local school, and you suddenly can't cover the property tax bill. This will be a problem for any land tax system, any capital gains tax system that taxes unrealised gains on the family home,* or systems supporting local education out of local levies on property values.** There are plenty of ways that the elderly homeowner could re-optimise: reverse mortgages could tap the equity, or the homeowner could sell to someone who values the local amenity and move somewhere else in town.

But if policy should be moving towards a tax base that can withstand a large proportion of taxable-income poor but asset-rich individuals with strong demand for government services, Manitoba's move isn't encouraging.

New Zealand at least has been making some moves in the right direction. We've been shifting the tax base from income to consumption. We don't tax capital gains, and for good reason. But whenever retirees cash out some of their portfolios to fund current consumption, they pay the 15% GST. I'd also expect that we'll be scheduling increases to the age of superannuation eligibility after the next election.

* Systems taxing only realised gains have other problems. And systems exempting the family home entirely have still other problems. Seamus has a rather lengthy series of posts on the topic, some of which are here indexed.

** Note also that we can also get big problems in rural school districts: townies are income rich but have less land wealth; farmers are land rich but will have incomes that fluctuate a lot year to year. If the school levy is a fixed proportion of the land value, the farmers bear a very large share of the school district's costs. I have no particular view as to what's fair here other than a generalised view that a progressive consumption tax would be rather better than all of this mess.

The plucking of low-hanging fruit

Tyler Cowen argued that some 20th Century growth was due to educational growth: in the early 20th Century, an awful lot of very smart people didn't get much education. By the 1970s, the gains from getting that pool of highly skilled labour into higher return occupations had largely already been exploited. Here's The Economist's take:

What low-hanging fruit? Educational fruit, for instance. Decades ago, the rich world was moving most of its population from very low levels of education to university educations, and was shifting lots of potential geniuses from the fields to the factories to the research labs. That was relatively easy to do and it fueled a big growth boom. These days, improvements in educational attainment are quite hard. Nearly half of all young people now go to college. Moving marginal students into college will require a lot of effort and won't yield large gains. That doesn't mean it isn't worth doing, he says. It simply means that easy gains from education are gone.

That move hadn't started in the rural Manitoba of the 1930s.

From my uncle Daniel Hacault's eulogy to my grandfather, who died 3 January aged 87:

Years ago I had talked to George Hutlet about what kind of student Dad was. George smiled and said he was always in trouble with the nuns and that he doodled a lot. Not very studious at all. Dad got his Grade 8 and left school to work on the farm, not an auspicious beginning for someones life, maybe not too smart-maybe not too bright. REALLY?

Dad was a self educated man. When we were young there were always books around that he read on algebra, welding and finance that he used to round out his education. Dad took a correspondence course in drafting and I can remember the mechanical drawings of machinery he had built or was about to build rolled up on his desk. His education was built on problem solving and common sense.

Lewis and I found a micrometer he had made when cleaning out the shop on the farm.  He probably made it in the 1950's. We measured it with a digital caliper and it is still accurate to a thousandth of an inch. It is on display in the hall.  Dad's metal lathe abilities are legendary and we still needed him to give us advice on how to center a work piece last year. Dad's self education instilled in his children a desire to learn either a trade or achieve higher education in university.

As you can see from the story, Dad was very mechanically inclined. He could fix or build almost anything. I can remember taking phone calls for the neighbors looking for “TI PAUL” to fix or build some farm machinery. I heard a story that he even built a double burner still for some of his more adventurous customers. His mechanical abilities filtered down to his sons in varying degrees. His boys followed in his inquisitive footsteps and dismantled his accordion – just to see how it worked. I think this put an end to his musical career as a polka band leader.

 I didn't know about the still. But it makes me smile.

Let Them Come

Adam Ozimek challenged econ-bloggers to spend more time writing about immigration.

I'd previously suggested a small move that New Zealand could make that would increase immigration at relatively low political cost: automatically grant permanent residence to international students completing a serious degree at one of our better universities. Adam argues for giving more international student visas for study in the U.S. It's already pretty easy to get a student visa to study here in New Zealand; our problem is that, as second or third country-of-choice behind the US for many international students, demand for our product seems pretty elastic to exchange rates.* One way of differentiating the New Zealand tertiary degree would be to bundle it with permanent residence.

International students graduating from tertiary institutions currently qualify for the Graduate Job Search Visa which gives them a year to find a job on graduation. On receiving a qualifying job offer in a field relevant to their qualification, the graduate can then apply for the Graduate Work Experience Visa, or try directly for immigration as a skilled migrant. Striking out as an entrepreneur would be tough to square with the immigration requirements. The working graduate can then move to residence. Each of those steps involves risk, and especially from the perspective of prospective students. Sure, NZ Immigration is likely to count your job offer in computer programming as relevant to your degree in engineering, but you can't really bank on it if you're weighing up your options from abroad. And the sure thing can be awfully attractive when you're already taking on a fair bit of risk by looking abroad in the first place.

On the downside, current University incentives to retain international students by chiselling on quality would have greater real-world effect; TEC would have to watch for the usual forms of gaming.

So:

• Advantages:
• More high-skilled immigrants for New Zealand in a world that's becoming more competitive in attracting skilled migrants;
• Wellington is severely constrained in offering greater tertiary funding, in part because of the costs of the zero percent student loan programme. Increasing demand for New Zealand tertiary education among foreign students would allow greater cross-subsidy of domestic students, mostly through higher programme quality, so long as the gains weren't extracted by the university bureaucracies. 
• Disadvantages:
• NZ students would face more competition in the labour market after graduation. That's offset by benefits to employers;
• Increased demand for housing to the extent that overall migration flows are substantially affected - we need to fix the supply side of housing anyway.
• Red Herrings:
• Eligibility for aid (DPB, unemployment benefit, sickness benefit) requires having been in New Zealand for two years after becoming a permanent resident; residence would only be granted on graduation. It's rather unlikely that anybody's grand scheme would be to slog through a university degree, hang around for two years, then live on the benefit;
• The government can withdraw permanent residence from those convicted of serious offences within ten years of being granted their first residence permit - deportation remains a real option.

So, why not encourage more international students to choose New Zealand by bundling permanent residence with our degrees?

* There are creative ways around this that I've not yet seen anybody try. Foreign students pay more than their marginal cost - that's how the whole NZ tertiary system is able to stay afloat. But it means that slightly reducing the cost to foreign students isn't always the end of the world. I've wondered why no New Zealand university has yet tried the trick of taking on exchange rate risk on behalf of their international students. The larger body is the better bearer of that risk and can more efficiently lay it off with better access to competitive quotes on exchange options contracts. We have a pretty variable exchange rate: since I've been here, the New Zealand dollar rose from about $0.60 US to $0.75, back down to $0.60, up to $0.80, down to $0.50, up to $0.75, and now up in the $0.80-$0.85 range. We quote international students $NZ tuition fees that, over the course of a student's enrolment, could double or halve relative to the student's home currency.

There are a few ways of insulating foreign students against that currency risk. For an additional up-front fee, you could guarantee that tuition rates in the student's home currency would not appreciate by more than some manageable amount relative to changes in domestic tuition rates, then buy option contracts on the exchange markets to lay off that risk. You could allow students to pay up-front for future years' tuition to lock in current exchange rates if current exchange rates are favourable, with a discount for early payment of future years' tuition charges and subject to refund if the student drops out. You could even offer international tuition fees in the student's home currency, guaranteeing that enrolled students would see no increase in that tuition schedule relative to the increases enjoyed by domestic students, for the three or four years of their degree - again, laying off the risk in international markets.

What's the use of National Standards data?

The anonymous blogger at Emil's Demesne* provides a few useful insights on how we might use National Standards data.

For readers outside of New Zealand, school-by-school data on student achievement rates will soon be published. An early version of that data came out this week when Fairfax compiled results from those schools willing to release their data early; the rest will wait until the government publishes the comprehensive dataset. Emil begins:

My own prejudices. I am a policy analyst with nil expertise in education. I support publication of national standards data. I am agnostic as to whether national standards should have been introduced in the first place, but furious at those “public servants” who sought to obstruct implementation of a lawful Government policy: no integrity.

I’m mainly interested in what the standards data can and can’t tell us, and how they might be used to improve education outcomes. The lens I’m trying to look through is how would I tackle this if I were at the Ministry of Education?

He follows up:

Where cross-school comparisons might come in useful is in identifying stand-out schools so that successful and/or innovative practices can be, where appropriate, replicated more widely. Obviously, what works in a school within one particular cultural, social, and economic context won’t necessarily work for a school that’s in a totally different one. You do your data analysis and your case studies, then you devise sensible categories and work within them.

So long as moderation is at least better than hopeless, in time, we’ll also learn quite a bit more than we already know about the impact of social and economic factors on academic attainment during the early years of school. This is important, because it’s these early years that are assumed to matter most. National standards data, for all its flaws, is or can be made rich enough to support meaningful research that will help us improve how we teach our children.

Depending on whether or not I can be bothered, I might write up some stuff on why I don’t like deciles as analytical tools. But not this week.

The post has nice context around why we shouldn't expect standardized testing or strong grade moderation in New Zealand. What should parents be looking at rather than just a school's ranking?

I’d look for signs that schools are using the standards sensibly on a student level. Especially for students not at standard, I’d want to see individualized learning plans, with achievable benchmarks/milestones. Ideally, these plans would be designed in and as a collaboration between teacher, student, and caregiver. Give the student a sense of direction and ownership: here’s what we want you to be able to do, here’s our plan for getting you there, and here’s how you’ll be able to feel your own progress along the way.

Over time, I’d look for signs that schools are using standards, in conjunction with the learning plans, to do some value-added appraisal of teachers. I would incorporate this formally into professional development structures. I accept that there’s only so much a teacher can be reasonably expected to do for kids who turn up hungry, have caregivers with significant reading difficulties, or who switch between schools a lot. (These are, incidentally, things that are thought to correlate pretty tightly with decile.) Placing appropriate weight on factors like this is something that I think standards will be able to do over time, even if they’re fairly messy.

Helpful advice. Our oldest starts in at proper school in February next year.

* Emil previously brought us chapter 1 of a short story chronicling how the Wellington bureaucracy might respond to a zombie outbreak, partially as follow up to a fun Twitter hashtag. I look forward to the story's second chapter.

Education regressions

Luis at Quantum Forest has been doing some great work with the schools data. I'm picking up on it with a few regressions. Raw performance data for the schools isn't all that instructive as schools have very different raw materials with which to work. But it would be nice to know how well a school does given the decile and ethnic mix of students coming into the classroom. So let's check that.

I'm using Luis's data here, modified slightly to work in Stata: I replaced the NA cells with . so that Stata would read things as numeric variables rather than strings. My do file and dta file are up at Dropbox.

Like Luis, I started by generating a variable giving the proportion of students either meeting or exceeding the standard in each of reading, writing, and maths. I then ran a few simple linear regressions with analytic weights equal to the total school roll: the dependent variable is an average and the schools average over different numbers of students.

Covariates are decile, decile squared, the number of students per full time teacher equivalent, proportion of each of {Maori, Pacific, Asian, International, Melanesian [MELAA, which could be an acronym for Middle-East, Latin America and Africa, says Kiwi Poll Guy in comments; he's likely right], Other} students (European dropped), indicator variables for each of {minor urban area (town), secondary urban area (suburb), main urban area (rural dropped)}, indicator variables for single sex boys and girls schools (co-ed dropped), an indicator variable for state schools (integrated schools dropped), an indicator for boarding schools, and indicator variables for each of the main types of school {composite, contributing, intermediate, secondary (full primary dropped)}.

Results are in Table 1, below. But first, a caveats. As best I understand things, these grades are not moderated. So any effects here could be saying either that some schools do a better job in teaching, or that some schools engage in grade inflation.

Table 1: Full sample: Reading, Writing, and Math
	(1)	(2)	(3)
	Reading	Writing	Math
decile	0.0485***	0.0363***	0.0437***
	(6.53)	(3.67)	(5.27)
Decile squared	-0.00224***	-0.00110	-0.00195***
	(-4.39)	(-1.62)	(-3.41)
Students per teacher	0.00282*	0.00362*	0.00215
	(2.41)	(2.35)	(1.65)
Proportion of Maori students	-0.0664*	-0.0848*	-0.112***
	(-2.20)	(-2.13)	(-3.34)
Proportion of Pacific students	-0.115***	-0.114**	-0.0676
	(-3.52)	(-2.63)	(-1.86)
Proportion of Asian students	-0.0796**	-0.0787*	0.00448
	(-2.63)	(-1.97)	(0.13)
Proportion of International students	0.290	0.863**	1.510***
	(1.23)	(2.79)	(5.76)
Proportion of MELAA students	0.0952	-0.0934	-0.0722
	(0.61)	(-0.45)	(-0.41)
Proportion of students Other ethnicity	0.428	1.043*	1.040**
	(1.35)	(2.50)	(2.92)
Minor Urban Area (Rural dropped)	0.00817	-0.0104	-0.0257
	(0.62)	(-0.60)	(-1.76)
Secondary Urban Area (Rural dropped)	-0.0189	-0.0283	-0.0416**
	(-1.33)	(-1.52)	(-2.63)
Major Urban Area (Rural dropped)	0.00890	-0.00366	-0.0186
	(0.80)	(-0.25)	(-1.50)
Boys school (co-ed dropped)	0.101***	0.118***	0.205***
	(3.90)	(3.48)	(6.87)
Girls school (co-ed dropped)	0.125***	0.181***	0.142***
	(5.60)	(6.16)	(5.44)
State school (integrated schools dropped)	-0.0246*	-0.0594***	-0.0219
	(-2.49)	(-4.58)	(-1.96)
Composite (Year 1-15) (Full Primary dropped)	-0.0221	-0.0566*	-0.0384
	(-1.14)	(-2.23)	(-1.79)
Contributing (Year 1-6) (Full Primary dropped)	0.0127	0.0345***	0.0338***
	(1.77)	(3.64)	(4.23)
Intermediate (year 7 and 8) (Full Primary dropped)	-0.0673***	-0.0974***	-0.107***
	(-6.07)	(-6.64)	(-8.65)
Secondary (Year 7-15) (Full Primary dropped)	-0.0939***	-0.110***	-0.158***
	(-6.78)	(-6.06)	(-10.15)
Boarding school	-0.00755	-0.0716**	-0.0475*
	(-0.39)	(-2.79)	(-2.01)
Constant	0.574***	0.527***	0.569***
	(15.48)	(10.75)	(13.78)
Observations	1006	996	1000
Adjusted R2	0.528	0.467	0.532
t statistics in parentheses * p < 0.05, ** p < 0.01, *** p < 0.001

Decile matters greatly. All else equal, a school one decile higher has about a four percentage point increase in pass rates. But, decile matters at a decreasing rate: moving from Decile 2 to Decile 3 correlates with a 3.3 percentage point increase in maths pass rates while moving from Decile 8 to Decile 9 only improves pass rates by one percentage point.

Class size matters: schools with more students per teacher have higher pass rates. I suspect reverse causation here: for a fixed budget, those schools that are able to run larger classes are likely those that have fewer discipline problems and so are able to put those resources to other uses.

Ethnicity matters. A standard deviation increase in the proportion of Maori students reduces aggregate pass rates by 1.3 percentage points in reading and 2.2 percentage points in math. Similar trends exist for Pacific Island student ratios. I'd be pretty cautious in interpreting this one: if you run things decile-by-decile, the effects mostly disappear. The biggest negative effect seems to hold in high decile schools, but by the time you get to Decile 10 schools, the median school has only 5.9% Maori students. Results then may be a bit sensitive to a few outliers on the right hand side. Like Luis, I'll refrain from doing much more until the official results come out.

Single sex schools seem to do well; boarding schools seem to do poorly.

I generated residuals from each of the three specifications above. The residual tells us whether a school had a higher or lower pass rate than we would have expected given its characteristics. This either tells us how good (or bad) the school is at teaching, or how good (or bad) it is at grade inflation. Without external moderation, it's hard to tell. The residuals from the three specifications correlate strongly with each other: schools that are good (or grade inflate) tend to do so across the board. The lowest pairwise correlation was 0.57; the highest was 0.63. I averaged the residuals to get a composite score. A high residual means that the school's actual pass rate was higher than what we would have expected given its characteristics.

I'm not confident enough in the model to put up my own league table of residuals. But I will put this up. This is a scatterplot of the residuals showing just how much school performance varies after we have corrected for decile, ethnicity, and everything else in the above model. That can point to its being a bad model, the underlying data being bad, strong differences in teaching quality across schools, or a combination of all three.

There are decile 1 schools providing pass rates twenty percentage points or more above what we'd expect, given their characteristics (that's the 0.2 number on the y-axis); there is one decile ten school providing pass rates more than twenty percentage points below what we would expect given its characteristics. Differences in school performance simply do not come down only to decile. Decile's the most important thing. But differences in performance among schools of the same decile by definition have to be about something other than decile. I can't tell from this data whether it's differences in stat-juking, differences in unobserved characteristics of entering students, differences in school pedagogy, or something else. But there's something here that bears explaining. 

Update: Note also Luis's very sensible cautions about selection into the dataset with the preliminary data. Schools chose whether or not to release their results to Fairfax. I'd guess the bottom tail isn't in this distribution. 

School heterogeneity

New Zealand schools have not been particularly pleased with the government's plans to release data on student performance that would allow for the construction of league tables; they've argued that heterogeneity in incoming student characteristics produce unreliable results.

A new NBER working paper by Ellison and Swanson suggests that, controlling for these kinds of incoming demographic characteristics, US schools are highly heterogeneous in producing top students. They set up production functions for top math students: those scoring very well on the AMC 12 exam. As expected, there are pretty strong demographic correlates of high-scoring schools: zip codes with more highly educated parents and with more Asian-American students produce higher AMC-12 results. But within sets of schools with similar demographic characteristics, there's still strong heterogeneity in production of top students. 

Our results suggest that there is a lot of variation among seemingly similar schools. The most notable feature of this variation is a thick upper tail of schools in which students are many times more likely to reach high achievement levels than are students in the typical school with similar demographics. This thick upper tail is present in all of our analyses of students with high AMC scores, but is not present when we look at where students with high SAT scores are coming from. This contrast suggests that that the thick tail is not because of the self-selection of high-ability students into a particular subset of schools. We suggest that a potential explanation is that almost all schools see it as their responsibility to provide English and math courses that cover material necessary to do well on the SATs, whereas there is much less uniformity in whether schools encourage gifted students to develop more advanced problem solving skills and reach the higher level of mastery of high school mathematics needed to do well on the AMC.

Relative to the literature on the gender gap in mathematics, our comparison of school effects relevant to girls suggests that schools are perhaps even more important for girls: we estimate that the 99th percentile high school in our sample is producing high-scoring girls at more than ten times the rate of an average school with comparable demographics. We also note that there are many low-performing schools that will only very rarely have girls reach the AMC performance levels we have studied. [emphasis added]

...

Our finding that there appears to be a lot of variation across schools with similar demographics could be seen as hopeful: the number of high-achieving students would increase substantially if low-achieving schools could be brought up to average; and upper-tail schools might have programs that could be emulated to produce even larger improvements. 

If it's impossible to tell which schools are high- or low-achieving given the demographic characteristics of their incoming cohort of students, it's harder to encourage the low-performers to improve. 

Fraudulent visas: de facto and de jure

Suppose that you want to come to live in New Zealand. You're young. You don't meet the criteria for a working visa. And you haven't quite the qualifications for a student visa. What happens? At least according to the Herald, you falsify your qualifications, come to New Zealand on a student visa, sign up with an education provider that doesn't look too closely at your qualifications and go and get a job at an orchard or a fish and chips shop.

That's de jure fraud. The documents used to get the visa were fraudulent.

Imagine another hypothetical situation. Something that cannot happen at Canterbury, at least not for very long, because we have progression standards where any student failing to maintain some minimal GPA is thrown out of the University. Consequently, anybody wishing to take this hypothetical path chooses a university that does not have progression standards.

Consider the case of an international student who gets a visa and enrols at a New Zealand university but who never shows up for lecture, never shows up for tutorial, never completes any piece of assessment, and never shows up for the exams. The student pays international fees but is, as far as the lecturer is concerned, a ghost. The high international fees paid go to cross-subsidize the education of those students who do attend classes.

You can pretty easily make the case that the whole deal is highly beneficial to everybody involved. The parents get to pretend that their kid has failed out of an international school because of cultural or language problems (instead of having the kid enrol in a known-to-be second or third tier school in systems with tight entrance standards), the kid gets a working holiday, and there's a massive cross-subsidy to real international students and to domestic students. But you can also pretty easily make the case that it's a de facto fraud on the spirit of the student visa.

Maybe somebody who falsifies a document to get a student visa is more likely to be dishonest on other margins and consequently is worth chasing after. But is it really that much worse than falsifying the spirit of the visa application by enrolling at a New Zealand university but never attending lectures and never submitting assignments?

I'm not sure how many ghost students the other New Zealand universities are getting. Before we had progression standards, they weren't exactly uncommon on my class lists. If we want to keep the cost to domestic students down, let's hope Immigration New Zealand sticks with the de facto cases.