Archive

Ineffective Theory

Statistical Lies II: A Secret Survey

I admit this is pretty minor. But every day in the bathroom, I’m confronted by this poster: Infographic poster about a survey conducted by Swytch

Nice design. (Unfortunately, the poster is laminated, so I can’t put it to good use when I’m done!)

How, exactly, was that study conducted? Is “shared equipment” specifically, something that 70% of respondents wanted? Ah, thankfully there’s a footnote: “survey conducted by Swytch in 2019.” Okay, so, what is Swytch? Might they have an incentive to deliberately distort the survey findings?

The original source seems to be this article. As far as I can tell, no actual data — or methodological information — was ever released. It doesn’t seem that any more reputable news outlets ever even mentioned this survey. The poster seems to be based (perhaps indirectly — many junk news sites played a game of telephone with the results) on a mash-up of two lines. First:

More than 70% said that they were more likely to choose to work at a company with a strong environmental agenda.

Second:

Nearly 70% of respondents said that if a company had a strong sustainability plan, it would affect their decision to stay with that company long term.

You can decide for yourself if the poster is a fair interpretation of those statistics. I’m genuinely torn on how harsh I should be. Slightly more bothersome is

The survey, which was the result of conversations with 1,000 employees at large U.S. companies […]

That casts some doubt on how the results generalize to “recruiting grad students and postdocs”. None of this is nearly as bothersome, of course, as the fact that we don’t have access to the survey itself! Or any data! So it all comes down to how much I trust the honesty and competence of the folks who ran it.

There are many ways to distort a survey, to “honestly” report misleading results. Advertise it with a link that says “care about the climate? Click here!”. Ask five different versions of the same question and only report the one that gives the most favorable statistic. Lie about the plain english meaning of the questions. Ask leading questions. Disregard how much respondents actually care. And on, and on. Since no concrete information was ever released, I guess we’ll just have to trust in the integrity of Swytch.

What is Swytch? Swytch — now defunct — was a “blockchain-based clean energy incentive”. Oh.