“I’m from the government and I’m here to tell you that you’re wrong.”
That was sort of the email that I got two and a half weeks or so after I published my expose of the serious problems in the Census Bureau’s reporting of online retail sales.
You remember that piece, don’t you?
It was the one in which I compared the bad reporting by the Census to the data that the Kodak folks relied on back in the day to make their decisions about investing in digital cameras.
Basically, the article, which spread like wildfire all over the Internet, reported how the Census had actually underreported data related to online sales, on average, by as much as a third.
And they had done it that way for as far back as we had looked, which was 2006. We even suspected that this error stretched back even longer than that, setting retailers up to walk into the same trap that Kodak did if they weren’t paying careful attention.
Well, it turns out I was wrong!
It’s actually even worse.
The “we” is my colleagues at Market Platform Dynamics – economists David Evans, Dick Schmalensee and Scott Murray. These are people with long-standing professional reputations, and who know their way around data. Dick is the former Dean of the MIT Sloan School and former Member of The Council of Economic Advisers. David teaches at two of the leading universities in the world, University of Chicago Law School and University College London. Both are ranked among the leading economists in their fields based on citations to their work and have published out the wazoo. Neither are exactly novices, nor data dufuses.
And, it actually wasn’t their life’s mission to prove the Census wrong.
They were just doing diligence on the chapter on retail reinvention in their upcoming Harvard Business School book, “Matchmakers,” and needed an estimate of the online share of retail sales — both a current estimate and an estimate of growth over time. After digging into Census data and looking at other sources of data, they found what they thought was an error.
But they wanted to be sure.
So, they set up a nice email exchange with a very accommodating person at the Census – the Chief of the Retail Sales Branch there. He was amazingly helpful and responsive, and confirmed, in writing, the source of the problem and acknowledged, in writing, the underreporting.
David, Dick and Scott also pored over the Census documentation that had the actual survey they used to collect the data and details on how they came up with numbers. Seven months later, after one final email exchange with Census, they thought they had figured out how to fix the undercount and came with a pretty good estimate of online sales.
Well, until my article and the technical paper documenting the error, and the fix, was published on Jan. 18 and the Census PR people got wind of it.
That triggered the nice little missive from the head honcho at the Census Public Information Office asking me to retract my article. They also issued a press release “clarifying” their position that what we reported was wrong.
There was only one problem.
What they said in the press release was flatly inconsistent with what the Census official had told the economists, and put in writing.
It was also inconsistent with what a reasonable researcher would interpret their published survey forms and methodology to say, and inconsistent with what a reasonable reader would conclude from the Census reports.
David, Dick and Scott tried to get in touch with the nice Census official who they had worked with to uncover the error but were prevented from speaking with him. They were told that they had to go through the Public Information Office on the matter (aka Census PR).
After a lot of back and forth, there was a call on Friday, Feb. 5, between said Census Public Information Officer, a few data analytics folks at the Census and Dick and Scott. The very nice and accommodating Retail Sales Branch Chief was on the line but didn’t answer any questions directed to him from the economists.
What Census contended on that call, is that they do count online sales from physical retailers, sort of, and do count online sales from non-retailers like Nike, sort of, too. That their data is correct, sort of, and that they made no mistake, absolutely.
Except, of course, for those nice email exchanges between the very nice and accommodating Retail Sales Branch Chief. Which were wrong according to Census.
And the Census documentation that they post? Well that isn’t really accurate either.
The nitty gritty of what the Census folks now say is in this short paper that David, Dick and Scott just released describing their Excellent Adventures With Census. The bottom line is that the Census data probably do understate retail sales, but given how they do things and how they’ve described how they do things, it is virtually impossible to fix the Census Bureau’s calculations.
For instance, the Census tables at the detailed industry level don’t report what they say they do, even though they could. The Census publishes a very handsome table that shows for various retail industries their eCommerce sales. It shows a row like General Merchandise (that’s where Walmart would be) and then an entry called eCommerce.
Sounds helpful, right? Wrong. It turns out the eCommerce figures are just junk data.
Why? Because most of the eCommerce sales for General Merchandise is put into Non-Store Retailers.
Now you might think that was just for pure non-store retailers like Amazon, but it is also for Walmart.com.
In other words, it’s a big fat mess. And you don’t need a Ph.D in economics to figure that part out.
As I said in my piece, this stuff is complicated and the Census has a tough job to do. And probably not as much of the resources and funding as they’d like to get this reported precisely right. Further, mistakes happen all of the time when dealing with massively complicated data sets like this. And researchers admit to them, correct them and carry on.
Apparently not if you have to work through Census PR.
So here’s a constructive thought.
We’d actually like to get the real retail sales numbers, divided by physical stores that do online and pure play online retailers.
We’d like to get that by detailed industry.
We’d like to get some assurance as to how big the undercount is and how good the coverage really is.
We bet you might too.
And we think that there’s only one way to do that since we find it a bit disturbing that the guy whose job it is to produce the numbers and who told us that they were underreported and looked at our calculations now can’t talk to us.
David, Dick and Scott aren’t exactly sure what to believe at this point. They’ve more or less thrown up their hands and have called for an independent review of the Census methodology and reporting.
I think that’s a great idea — and PYMNTS is jumping on the bandwagon.
And what the heck, I’m going to lead the charge.
So, let’s commission an Independent Audit of the Census retail numbers and reporting so that we can not only see the numbers for real – not the sort of numbers that they report today, but also better understand the methodology for how these numbers are devised.
And then figure out how the data should be reported so that it is actually useful to businesses and people who pay the nice salaries (probably too low) of the Census folks.
Census Retail Sales Office: We’re from PYMNTS.com and we’re here to audit you.
Well, not PYMNTS exactly, but a team of economists and statistical experts, independently appointed.
Consider it our contribution to making government work for everyone out there in payments and commerce land.
Plus, you have to admit, it would sort of fun to have the audit tables turned, don’t you think?
#CensusRetailSalesDataAudit – Gotta love it.