We typically do surveys wrong; here’s how we can do better

Introduction

The earliest known public survey was conducted in 1838, by the Statistical Society of London – a simple, door-to-door survey seeking to understand industrial and social conditions. Don’t worry, this isn’t a lesson about the history of surveys (much to the chagrin of our survey department).

Surveys have always been and remain a critical tool to gauge public perceptions and provide valuable insight into how a community feels and thinks about core issues. It has been used for decades by cities and counties to better understand their residents, inform strategic plans, set priorities, and measure performance. 

But in the 185 years since a few English lads walked around Manchester knocking on doors, not much has changed in the methodology of surveys. Door-to-door gave way to mail, telephone, and, later, online surveys, but the core assumptions and process of designing, deploying, and analyzing a large-scale public survey have remained largely unchanged.

This is not a takedown of surveys; we are huge fans of surveys. In fact, we are such big fans that we want to see more surveys, just better and more aligned with how residents in cities and counties live. Most importantly, we think that the discipline of the public survey has suffered from stagnation for too long and has lagged behind, making it a remnant of the past rather than a tool of the present, let alone the future. 

So, what are the common challenges that are still plaguing traditional public surveys, and how can we do better?

1. Sampling

Traditional Surveys use probability samples, which means identifying a list of people or houses, drawing a random sample from that list, and contacting those individuals or households. This worked well for much of the 20th century, whether by mail, telephone or in person. But times have changed. For decades now, response rates have declined. So even if you’re contacting 1000 people from your list, maybe 100-200 will respond. You can either assume that the 100-200 accurately represent the 1000 people or make some statistical adjustments, which have their own assumptions. You’re also left with a much smaller analytic sample than you set out to obtain. 

And even further, the people who appear on that list, to begin with, might be different from people who aren’t on those lists. If you have a list of households, you are more likely to miss people in multi-unit apartment buildings, people who move often, or other renters – people who tend to be younger and are more likely to be people of color. 

Surveys of the future can help fill these gaps and address these limitations by using a non-probability sample that is representative of the population of a city. Instead of randomly selecting a group of people from a list and sending the survey to them, and them alone, we invite anyone within our target population to respond by reaching them where they are – on their devices, probably refreshing their feed as they walk their dog. By inviting more people to complete our survey, we can ensure a more representative sample, reach people we wouldn’t otherwise reach, and avoid dealing with non-response. These surveys are increasingly used by survey organizations around the world and are here to stay.

2. Inclusivity

Traditional Surveys rely on distribution methods that leave many people out, particularly, as noted, people who do not have a permanent address or might be otherwise harder to reach. Furthermore, traditional surveys often ask respondents to spend a good amount of time providing their answers or ask respondents to mail back the completed form. By targeting only a specific set of people, or by making it too burdensome of a demand, surveys often alienate and exclude an important part of their community. In doing so, they also neglect to take note of crucial perceptions, opinions, and concerns.

Surveys of the future find people where they are; that is, most often, on their phones or tablets. By popping up on the apps and websites residents regularly use and taking only minutes to complete, the surveys of the future remove traditional barriers to participation and encourage more people and, crucially, more diverse groups of people, to complete the survey.

3. Effort and Resources

Traditional Surveys tend to be a heavy lift and incredibly resource-intensive. Between survey design, distribution, response collection, and analysis, by the time you get any actionable insight you can act on, the data has gone stale. It is a difficult conundrum to make and implement decisions when you’re already starting to work on the next survey. Running a public opinion poll shouldn’t feel as cumbersome as implementing an infrastructure project, but it often does.

Surveys of the future significantly reduce the human labor, attention, and effort around each survey, and place the burden on technology. Templatized surveys, digital distribution, automatic collection, AI-based response analysis, and a dashboard surfacing immediate takeaways from the data – these are all technological tools that make the job of running surveys much more efficient, and less costly or time-consuming. We often forget that the survey itself is a means to an end, not an end in itself. It is a tool meant to serve decision-making, not get decision-makers bogged down by process and methodology. Technology-reliant surveys allow decision-makers to focus on the survey’s takeaways and take action rather than spending their time on making the survey happen at all.

4. Cadence

Given the mammoth tasks that are Traditional Surveys, and in light of many local governments’ limited resources, cities and counties normally deploy a broad community survey once a year, or possibly bi-annually (in some cases, even less frequently). This cadence means that surveys, which aim to provide a snapshot of the here and now, lack the pace that would make them limber and responsive. Low cadence means that a survey that runs in January of a given year is often the source of knowledge about community sentiment and satisfaction long into the year, and beyond. It fails to capture shifts in public sentiment or perception due to large-scale events, policy changes, or performance improvements as they happen, leaving decision-makers grasping at straws until the next survey comes around.

Surveys of the future are in line with the swift pace of life and can be deployed at a higher cadence to respond to events as they happen. Rather than capturing a moment in time, the surveys of the future are always on, collecting public sentiment on an ongoing basis and doing real-time analysis to surface immediate takeaways – as the survey runs and not just when it concludes.

What the future holds

There is no future where local governments won’t have to keep their finger on the pulse of their community or won’t need data to make better decisions, measure their performance, and increase their residents’ trust. Surveys remain one of the best tools with which to gauge public sentiment and opinion.

But the world moves forward, and technology can make even the most long-standing methods better. The future of surveys is not fewer surveys, but better surveys: more efficient, light on their feet, inclusive, accessible, and representative; and more frequent and therefore more actionable. 

Zencity brings surveys to the here and now by leveraging technology and AI to re-envision public opinion surveys for local governments