There is a lot happening in the Insights space right now and while tools, AI, and automation are having their moment, the single greatest issue remains data quality. The quality issue is complex and requires comprehensive and creative solutions across the ecosystem to ensure we get this right. It also requires continued vigilance as the variables continue to evolve.
Here are a few observations based on recent data quality reviews.
The primary line of defense for data quality lies in our choice of source partners. As standards in this space have degraded, we need to be more discerning about sample sourcing. There are a lot of ways to reach consumers for research but there are a lot of critical considerations to make with partnership decisions that can make or break your quality. Transparency is essential. Spend time on this step and ask a ton of questions.
Strategic relationships with partners who offer cutting-edge solutions for respondent validation, fraud detection, and real-time monitoring are needed. If you don’t have fraud detection running on your surveys, you have low quality data and likely fraudulent responses in your data. You’re not protected. Make sure to partner for this purpose or work with your sample partners to better understand their offerings.
The old approach of checking a few boxes for data quality – maybe a simple CAPTCHA and an attention-check question – isn’t enough. The fraudsters know how this works and have adapted. There are a lot of new ways to detect and remove bad participants but detection requires a combination of tools and human strategies. You have to stack tools – third party and internal - to combat the different ways fraud or otherwise poor-quality data seeps into your project.
Think of it as a layered security system for your data. We use a set of tech solutions for pre-survey detection that capture things like IP address spoofing, address inconsistencies, duplication, nonsensical systems configuration, VPN usage, and the presence of bots. From there we use additional solutions to detect AI and capture in-survey quality issues. Our approach for QC always includes humans in the loop throughout phases and a flagging system across multiple quality checks.
Be careful to strike the right balance of quality control and participant experience. Our objective is to get the bad data out of the system while also encouraging good data from engaged participants. When you focus too heavily on catching fraud, it’s easy to inadvertently discourage participation from those we rely on for insights.
The number of quality issues impacting B2B research is concerning and specific mitigation is needed. The level of fraud we’re seeing has been increasing the past couple of years and is showing no signs of slowing down. The bad actors know where to invest their time and AI has made it easier.
Depending on which industry source you reference, removal rates tend to range between 15% and 25% for most consumer research. For B2B, the average removal rates when recruiting from panels can exceed 80%. Panel sample alone is no longer a sufficient strategy for this type of work. We’ve seen more B2B projects utilize executive recruitment as a way to extend our reach with audiences and also to validate the credentials of the participants.
A few years ago, we talked a lot about the inherent problems we see from open-end responses. Participants don’t like answering open-end questions and their answers aren’t as useful as we would like. Fast forward a couple years and we now feel open-ends are essential for detecting fraud and assessing data quality.
In addition to gauging the quality of the open-end response, this is a great way to capture meta data/tech info and behavioral flags for removal. When bad actors employ auto-responders and AI to evade detection, the open-ended questions can be a great way to uncover truly fraudulent behavior. We encourage the use of at least one open-end, even one very general in nature and unrelated to the recruitment, for this specific purpose.
Data quality is the single most important issue impacting our industry. The playbook we used two years ago for data quality isn't cutting it anymore. We're moving toward more adaptive solutions to keep up with the changes. The threats are more sophisticated and complex and ultimately harder to detect. Collaboration across the ecosystem – suppliers, agencies, brands – is critical to align on key quality metrics and ensure sufficient steps are being taken to mitigate the threats to quality.
