Christoph Trappe Data quality in research is so vital in gathering insights. Without having accurate data, how can we make the right decisions? Conversely, when data is insufficient, we might make the wrong decision because it was based on incorrect or even fraudulent data.
“When we want to understand our customers, we have to have good data quality,” said Zoe Dowling, senior insights leader at Microsoft, on an episode of the market research podcast “Reel Talk: The Customer Insights Show.”
In this article, I cover the following:
What’s the spectrum of data quality in research?
Lisa Wilding-Brown, CEO at InnovateMR, explained that data quality and the lack thereof happen on a spectrum.
“On one end, you have these nefarious users who are doing this at scale and are trying to exploit vulnerabilities of research and panel companies,” she said on “Reel Talk.” “In the middle of the spectrum, you might have real human beings recruited by over-incentivized websites. And then, on the other end, you have innocent issues that pop up.”
So the spectrum goes from nefarious attempts to wrong intentions to unintentional mistakes, which is good to know so we can address the specific use cases of issues in data quality.
“Cyber fraud is not going away,” Lisa said. “If you measure the GDP output of cyber fraud, it would be the third largest country after the United States and China. So just put that into perspective the global damage these fraudsters can deploy.”
Read next: How to write research questions
Strategies to ensure data quality in research
Straight-up fraud in research, the one extreme of the spectrum, happens especially in an economic downturn and when technology enables bad actors to cheat the system in easier ways than would have been possible years ago.
“There’s a lot of strategies you can implement to get ahead of it,” said Lisa. “There’s a balance between technological and methodological strategies to deploy.”
She said that technical strategies can include digital fingerprinting, while methodological ones can include how questions are asked and how people respond.
In-survey strategies
Lisa said that fighting fraud in research and going for good quality responses requires good survey design from the start.
In quant surveys, the danger of survey bots can be a problem, but several strategies can weed them out, including using red herring questions that ask questions to see if people are paying attention and that are hard to answer for bots.
Read next: How to avoid survey bots
Red herring questions can also ensure that the right people are being asked the survey questions.
“They can be used to test domain expertise,” Lisa said. “Let’s say you are going after IT decision-makers. Create questions that are designed to test their domain expertise.”
In your questions, include real brand names and non-existent brand names to weed out people who don’t know and to catch bots.
“If they have an awareness of those fake brands, you know they aren’t in it for the right reasons,” Lisa said.
Before even getting into the survey, make sure to ask good screener questions to get the right people to the survey.
But survey fraud can happen anywhere, which is why it’s so important to stay ahead of trying to prevent it.
For example, in the Voxpopme video survey platform, panel manager Matthew Handegaard and the team have implemented several strategies to prevent fraud in research and ensure data quality.
“Respondents are required to show their face, which helps identify and remove duplicates,” Matthew said. “We also geofence, so anyone registered is inside the country. Then at the response level, we reject bad responses and repeat bad respondents are deprioritized, which could lead to removal.”
Read next: How to ask inclusive demographic questions in your market research
Partnerships
Relationships matter, which certainly also holds regarding assuring data quality in research. Ensure to understand what your suppliers do to strive to get a high-quality sample, Lisa said.
Consider: Use the questions for users and buyers of online samples from ESOMAR
“Look at how they recruit and incentivize people,” Lisa said.
On-staff expertise
Tactics and technologies of bad actors committing fraud in research change constantly. That’s why it’s so important to have somebody on staff that focuses on the issue, Lisa said.
“Make sure that all these mitigation tactics are deployed,” she said.
To deploy tactics, somebody needs to understand fraudsters’ tactics and how they evolve. For example, take the example of device farms discussed in this interview with a former fraudster.
Consider getting outside perspectives. That could include studying former fraudsters, talking to somebody new to the market research industry about a problem, and participating in formal mentoring.
“There’s just so many formalized mentorship programs,” Lisa said. “Get involved, and you’ll get so much benefit from it, and companies will see that benefit from that, too. So everyone wins.”
Read next: Our checklist from experts: Building a team the right way
Technology to scrutinize responses
Use technology solutions – like Natural Language Processing – to review the responses. Do they make sense? Are they in line with the question?
Consider all the options.
Keep an eye on new emerging options, and don’t put all your eggs in one basket.
“We need to use many different tools and strategies to catch these individuals,” Lisa said.
Collect the right amount of personal information
Collecting the correct information and the right amount of it can help us verify who participates in our surveys. But, of course, there’s also a level of consumer trust and privacy concerns that need to be honored here.
“There’s a halo effect that can happen when other industries not even connected to us do bad things or report a data breach,” Lisa said. “That can erode the trust of our consumers in survey research. That can have a ripple effect on our space.”
Some of that can be overcome through good communication, said Jenn Vogel, Voxpopme’s CRO and host of “Reel Talk.”
“How are we communicating that this is what we are doing, what you are opting in for, and what we are going to use your data for, and that there’s no personally identifiable information attached to it,” she said.
At the end of the day, ensuring data quality in our research is crucial. Whether the quality is threatened because of fraud, the wrong incentives, or simply mistakes, it’s a problem that we as an industry together need to keep working on fixing. And if we can’t eliminate it, at least let’s minimize the impact.
Listen to our market research podcast
