Here is also a post by @swaminomics in 2011 on NSSO sample unreliability per RBI Governor's opinion swaminomics.org/?p=2021—
Shashi Shekhar (@offstumped) February 23, 2013
Here is a paper from the relevant Ministry that speaks to the issue of reliability or lack thereof wrt NSSO Samples mospi.gov.in/7cocsso_paper_…—
Shashi Shekhar (@offstumped) February 23, 2013
Originally published on Niti Central here. Updated below with Twitter debate on this blog post between Sadanand Dhume and many friends.
On July 26, 2012 a complaint was received by a court in New York. The plaintiff in that law suit was our very own New Delhi Television Ltd, or NDTV. The case was filed against, among others, most notably Nielsen Holdings and Nielsen Company. To quote the core complaint against Nielsen, Kantar and their India-based joint venture Television Audience Measurement (TAM), the case was to do with “losses” apparently suffered by NDTV on account of alleged “rampant corruption” in the Television Audience Measurement business.
To the puzzled reader wondering what this squabble between a television channel and a television viewership survey agency has to do with the debate at hand over social indicators in Gujarat and the leftist propaganda in oped columns, let me call upon your patience and direct your attention to what specifically was NDTV’s problem with the Nielsen and Kantar joint venture, TAM.
The first concern NDTV had with TAM was the survey sample size. It wanted it to be increased from 8,000 to 30,000. The lack of adequate sample size is a recurrent theme in NDTV’s complaint. The complaints get serious with added element of corruption in the data collection area. The complaint quotes unnamed senior officials of Nielsen and Kantar admitting that television audience surveys in India were generating inaccurate data due to rampant corruption. A more insightful view emerges from the findings of a committee that looked into the whole area of television audience survey in India. Specifically the committee found “sample sizes to be inadequate” and “lack of reliability” in data collection methods from sample homes.
The limitation of ‘sample sizes’ in conducting surveys in a demographically heterogeneous population is not limited to the world of Television Audience Measurement alone. As we have seen in India, in election after election exit polls and pre-poll surveys falter pitifully from the final outcomes in States with multi-polar contests and complex demographics.
Which leads us to the question: If surveying something as technologically straightforward as Television Audience Measurement can be so faulty and corrupt in a country like India, how much faith can we put into surveying something as sociologically complex as human development indicators?
I hope the patient reader now appreciates the detour from the intellectually malnourished debate on social indicators of Gujarat to the murky world of television ratings to question the very credibility and pseudo-science behind how social indicators are sampled and projected.
There are sound scientific reasons to debunk the entire approach by these leftist commentators who have been abusing lagging social indicators to make politically convenient arguments.
Take the case of ‘Sample Registration System’ or SRS that is used to measure infant mortality among other social indices. It is not a population wide measurement (Census like). Instead, it relies on about 7,000 sample units. The data is not collected by professionals and not through technology enabled automation but manually by part-timers. There is usually a six-month lag for independent audits. The sample itself is based on a 10-year Census during which demographics may have altered significantly. As an example the current sample is still based on the 2001 Census data.
Take the case of another index that was bandied about by activists of all sorts in Oped columns during 2011 and 2012 – the so called ‘Indian State Hunger Index’. It was based on 2004-2005 sample data and a statistical model developed in the West with no independent validation or relevance for Indian conditions. Even the UNDP HDI index is a lagging indicator at best.
If samples and statistical models were found to be faulty and unreliable for television viewing habits and electoral preferences, why pray should we deem them to be anymore scientifically accurate for measuring social indicators?
Lastly, let me draw attention to a recent story in The Times of India on how elite pockets of Mumbai were found to have a skewed sex ratio. The story was interesting not so much for its political import but for the granularity of the data cited, getting down to specific pockets within the city of Mumbai. This kind of granularity is far more actionable and useful as it points out specific areas where potentially welfare interventions by local Government (municipal or panchayat) could make a real difference.
Contrast this with pointless State-level aggregated social indices on infant mortality or sex ratio.
They neither pinpoint the dark zones nor the bright spots. They can’t tell you if Surat is better performing or if a certain pocket of Patna has a problem. These leftist sociological constructs are at best lagging indicators that end up making the case for Centrally-sponsored schemes and statist top-down welfare models. While these social indices help in academic analyses while keeping the Delhi-based extended ecosystem of NGOs and activists gainfully employed, they are barely actionable.
The TOI story on sex ratio disparity across the city of Mumbai is a pointer to how we must fundamentally alter the debate on social performance, taking it away from the voodoo statistics and pseudo-science of the political Left.
Today the Nielsen method of panel-based media audience measurement is being challenged by upstarts with the help of big data analytics and population-wide Census-based methods of audience measurement. Recently the Obama re-election campaign employed similar population-wide behavioral analytics to micro-target voters to ensure his re-election. There is no reason why we in India must not look to technology to devise ingenious methods for near real time data collection and population-wide analytics of social performance. This will not only help micro-target and localise welfare Interventions by local Governments (as opposed to centralised schemes) but it will also shift the focus away from agenda-driven politicking based on lagging indicators and towards a debate on actionable interventions that can make a difference here and now.
Let me end with relating an experiment the World Bank attempted in 2010 by opening up all of its data sets through open APIs for an ‘Apps for Development’ contest. There is a lesson in this experiment to our Governments on how to win back the debate from the agenda-driven Left by embracing the idea of ‘Open Government’.
It is time real facts based on near real time data made these voodoo statisticians and pseudo-social scientists redundant, perhaps even irrelevant.
Associated Twitter debate:
Interesting pow-wows between @dhume01 and many friends over this
blog.offstumped.in/2013/02/23/the…—
Shashi Shekhar (@offstumped) February 23, 2013
@ShankkarAiyar @dhume01 @offstumped @KalpanaKochhar Many limitations but the most reliable and consistent time series on cons, emplt inc etc—
Yashodhan Ghorpade (@yashodunn) February 23, 2013
@ShankkarAiyar @dhume01 @offstumped @KalpanaKochhar NSS is the best available and also most widely used by Wb economists themselves—
Yashodhan Ghorpade (@yashodunn) February 23, 2013
@dhume01 12th five year plan doc says it. @shilpitewari @manohar_sram @offstumped—
Vijay (@centerofright) February 23, 2013
@dhume01 @offstumped @swaminomics NSSO data collection methodologies are a joke. Untrained low paid teachers used as forced labor.—
UnBanker (@unbanker) February 23, 2013
@dhume01 How superior is G-20 data. Isn't most of US data incl demography, sample based? @ShankkarAiyar @offstumped @kalpanakochhar—
KK (@kk_ind) February 23, 2013
@dhume01 @amishra77 @malviyamit @manohar_sram @offstumped sorry to barge in. deshgujarat.com/2013/02/18/jap… Argument on Malnutrition para. Japanview—
(@SominduS) February 23, 2013
@dhume01 @amishra77 @malviyamit @manohar_sram @offstumped sorry to barge in. deshgujarat.com/2013/02/18/jap… Argument on Malnutrition para. Japanview—
(@SominduS) February 23, 2013
@dhume01 @amishra77 @manohar_sram @offstumped Another good read on Gujarat. Objective. articles.timesofindia.indiatimes.com/2012-09-22/edi…—
Amit Malviya (@malviyamit) February 23, 2013
@dhume01 True. I think Census, RBI do a good job.@offstumped @KalpanaKochhar—
Shankkar Aiyar (@ShankkarAiyar) February 23, 2013
@ShankkarAiyar It's all a question of degree. Is there a single institution in India where "all is kosher"? @offstumped @kalpanakochhar—
Sadanand Dhume (@dhume01) February 23, 2013
@dhume01 @shilpitewari @manohar_sram @offstumped Panagariya says Gujarat has improved on malnutition statistics. articles.economictimes.indiatimes.com/2013-01-02/new…—
Amit Malviya (@malviyamit) February 23, 2013
@malviyamit Like many institutions there's room for much improvement but it's also the best you've got. @manohar_sram @amishra77 @offstumped—
Sadanand Dhume (@dhume01) February 23, 2013
@dhume01 often their reports quote/refer to NSSO I guess there is acceptance but that doesnt mean all is kosher @offstumped @KalpanaKochhar—
Shankkar Aiyar (@ShankkarAiyar) February 23, 2013
@dhume01 @manohar_sram @amishra77 @offstumped Even Montek doesn't think much of NSSO data ! livemint.com/Politics/LsaTB…—
Amit Malviya (@malviyamit) February 23, 2013
@ShankkarAiyar Would love to know if WB/IMF view NSSO data as better/worse than in most developing countries. @offstumped @kalpanakochhar—
Sadanand Dhume (@dhume01) February 23, 2013
@dhume01 Let me add to the churn newindianexpress.com/opinion/articl… @offstumped @swaminomics—
Shankkar Aiyar (@ShankkarAiyar) February 23, 2013
Filed under: Left Liberalism, Local Governance, Narendra Modi, UPA-II Critical Appraisal
Comments