Surveys, Statistics, and Talking Points
I’m no statistician, and chances are that neither are you. And yet with the proliferation of free online survey tools and 5 minutes of free time to throw together some questions, any one of us can pull together a semi-official looking industry survey and push it out to the web and our mailing list looking for data. Now, what set me off was reading an article over on CMSWire called Thinking of Moving to a Public Cloud? Think Again by David Roe. While I do think it’s a bit of a hit piece on the cloud, I’m not attacking the author, nor am I necessarily beating up on Bitglass, the company who put together the Cloud Security Spotlight Survey. I’ve created surveys in the past, and I’m gearing up to kick one off next month before we head off to Ignite in Chicago in early May. They are a great conversation piece, and as a source of data for future content.
But I do have some problems with this piece, and wanted to articulate them here as an example of what is often wrong with these surveys. So while I am pointing out specific issues with this survey and results, you can really abstract my complaints to the broader category of industry surveys. For example:
- Lumping all cloud services together. According to the report, the survey “found a third of respondents have experienced more security breaches with the public cloud than with on-premises applications.” Excuse me, but using Gmail is simply not the same thing as sharing documents via Office 365, and yet according to the report, 45% of the respondents stored information the cloud via email, the leading problem area identified. Later in the report, it says “About one-third have experienced more security breaches with the public cloud that with on-premises applications.” But without some kind of breakdown of the type of breaches again the type of cloud services, this sounds much more ominous that it probably really is.
Equating fear of danger with actual danger. Throughout the report, the company talks at length about fear of potential issues, but provides very little data about whether these companies have actually experienced what they fear. It states “90 percent still express concern about public cloud security” (47% very, 43% moderately concerned) - Lack of qualifying questions. The report mentions the top 3 barriers to cloud adoption being 1) general security concerns (45%), 2) Data loss and leakage risks (41%), and 3) Loss of control (31%) but provides no qualifying data, nothing to pivot on to better understand how they are lumping these problems together in an effort to generalize, and therefore make the problem seem greater than it may actually be.
- Substituting facts with feelings. Ask someone what they feel, and you’ll receive one answer. Ask them what they’ve actually experienced, and you’ll likely get a different answer. So when the survey states that “36% of respondents believe that major cloud apps are less secure than on-premise application” my reply is “So what. My cloud services comply with the major security standards and certifications, and provide a detailed SLA. What does your on prem solution provide?”
What you need to remember about these types of surveys are that they are not scientific, they are heavily biased toward the company conducting the survey (even non-profits show bias toward results), and they are rarely comprehensive. Of course, none of this means that these things aren’t directionally correct, and that you should not continue to use them as a point of reference. My point is that buyers should beware. What bothers me is when a piece like this is shared as if it WERE scientific and unbiased, without acknowledging the source of the data (usually a company pitching a product or service), and without detail behind the questions asked so that you can make your own opinions about the resulting data.
We all like data. The company that produced this survey and report are using this data – and all of its imperfections – to sell you something. David Roe used it for an article. And I am using both for a blog post. But its always good to ask questions, look at the data behind the data, and try to apply what you think you understand to what is happening within your own business. </rant>
