NHS fails low incomes–and Canadians

Unfortunately the following note to readers from today’s release of the third and final set of data from the National Household Survey by Statistics Canada speaks for itself:

Note to readers

Comparability of low-income estimates

Low-income estimates from the 2011 National Household Survey (NHS) compared with previous censuses show markedly different trends than those derived from other surveys and administrative data such as the Survey of Labour and Income Dynamics or the T1 Family File.

Data to support quality estimates of low-income trends require a stable methodology over time that has similar response patterns. With the new methodology of the NHS, estimates of low income are not comparable with the census-based estimates produced in the past.

Previous census income releases compared low-income rates over time using the low-income cut-off (LICO). Given the lack of comparability of the trends and to prevent misleading conclusions arising from comparisons of LICO estimates from the NHS with earlier censuses, estimates of low income based on LICO are not available as a standard product from the NHS. They are available upon request.

The Harper government’s decision to drop the Census and replace it with the voluntary National Household Survey has clearly been a major failure.  As many from across the spectrum expected, the results aren’t comparable to previous Censuses or to administrative data for major sections of the population.     Kudos to Statistics Canada for at least being honest about it.  This doesn’t mean that all is lost and nothing can be considered reliable from it, but it will take some time to determine what can be considered reliable or not, especially with the technical documentation and methodology not to be released until next year.

The NHS will of course provide a greater level of detail in many areas, but I suspect analysts won’t consider most of these credible unless the general results for those populations confirmed by other surveys and data sources.  A dark cloud will remain over the NHS results wherever there isn’t a stronger light shining from other more reliable statistical sources.

Armine Yalnizyan, Kevin Milligan and others have written, blogged and tweeted much more extensively about this and I look forward to their further comments and commentary.

UPDATE:

Useful analysis over at economicjustice.ca   “The only way Statscan was able to publish the 2011 NHS data was by remarkably dropping its quality standard by doubling the acceptable global non-response rate.”

If Statscan had applied the quality standards it used in the 2006 Census, they would have only been able to release data for 19% of the Census Subdivisions (CSDs).   In comparison, 84% of the census subdivisions met this quality standard for long-form responses in the 2006 Census.  Non-response was so high in the 2011 NHS that more than a third didn’t meet these much dramatically lower statistical quality standards.

It’s sad that so much has been wasted through this.  I, like no doubt other analysts, researchers and social scientists, eagerly anticipate new sources of data, especially from the Census.  It’s our raw material, but it needs to be of a decent quality, otherwise it’s not credible and is unusable.   I was hoping for better results, but if Statscan warns about its reliability after significantly dropping their quality standards, unfortunately there’s not a lot we can do with it.  It’s a bit like tainted meat: difficult to figure out what’s worthwhile and what isn’t, especially when it gets used for other products.