I’ve wanted to write this article for nearly a decade. Many in my firm cautioned against it, not wanting to endanger our relationships with existing and potential clients. “Don’t get on your soapbox,” they pleaded. Perhaps they’re right, but as the president of a mid-sized, full-service research company with more than 30 years in the research industry, I’ve seen just about everything and believe there are a few issues that still need to be addressed on the client side.
Some of you might think “Who is this guy to tell us all the things we’re doing wrong?” Well, like most good advice, it comes from a place of concern and caring. Know this: at MDC Research we love our clients, we love research, and we constantly strive to make it better. If you recognize yourself in even one of the bad traits listed in this article, and want to improve the value of your research, we want to work with you.
Every one of these ‘mistakes’ is described in order to make your client-side research budgets more effective. Yes, research firms would benefit as well from more efficient workflow to more exciting methodologies. But ultimately, we exist to make our clients more effective and successful, and therefore I submit this list.
1. Sending all research projects out for bid
There is a common belief that competitive bidding is the best way to review and select vendors for upcoming research projects. In reality, nothing could be further from the truth. Researchers are often hesitant to submit their very best, most creative research design, for fear that an “expensive” design may not even make the first cut.
Corollary: Frequently, a second round of proposals/bids is requested, using the intellectual properties of one or more of the vendors who originally submitted a design. This is often couched as: “we’ve re-imagined the project and would now like you to bid on…” The original—creative, experienced, knowledgeable—vendor is now faced with a situation in which they must re-bid against vendors who were incapable of conceiving the sophisticated design in the first place. The other vendors now bid on the original design but are not familiar with its nuances and requirements, and therefore submit bids far below the cost proposed by the original vendor. The result is the client getting ‘cheap’ research from a company that doesn’t fully understand the design and is less capable of implementation.
Market research companies should be up for periodic review, exactly like advertising agencies and outside legal counsel. You’ll never get the very best from a research vendor without building a true working partnership with them, and that can’t be done while sending each individual project out to bid.
There is a reason your company doesn’t send every advertising campaign out for bid, nor do they ask attorneys to bid on writing every contract or handling every negotiation. Heck, your company probably doesn’t even ask suppliers of consumable commodities (paper/water/toner) to bid on every shipment. Instead, most companies have annual or multi-year contracts for these services. Market research companies should be up for periodic review, exactly like advertising agencies and outside legal counsel. But you’ll never get the very best from a research vendor without building a true working partnership with them, and that can’t be done while sending each individual project out to bid.
If you’re worried the removal of the per-project bidding process will result in increased costs, simply demand transparent pricing. You should be able to calculate the cost of a project before you receive a proposal from any research vendor that is your true partner. (Full disclosure: we share our cost calculators directly with our clients and encourage their use). Review the designs your research partner submits and don’t be afraid to ask them to rethink any design you’re not completely satisfied with (much like you would do with an ad agency’s creative pitches).
Still not convinced? Have a heart-to-heart with any of your current research vendors, and they will tell you they provide their very best pricing to those clients with whom they have the strongest connection and understanding of what to expect in the working relationship.
2. Not getting the most out of your research vendor
It’s a fact: in today’s world of frequent (and sometimes scheduled) company reorganizations, we, as research vendors, often find ourselves with more knowledge of and history with a client’s products, services, or divisions than those within the company. It is the norm for us to have decades long relationships with our clients, often spanning at least a half dozen primary contacts on the client’s side. On ours? We still have the original account team, mostly comprised of principals in our firm.
Our knowledge of what research has worked for your company and its products, our understanding of your market and its segments, and even our ability to navigate your internal politics is encyclopedic.
Our knowledge of what research has worked for your company and its products, our understanding of your market and its segments, and even our ability to navigate your internal politics is encyclopedic. You’d be shocked at how often a new team asks us to conduct research that has already been done. We’ve lost sight of the number of times we’ve seen a research request along the lines of: “We need to better understand our customers and more fully integrate with their business model,” yet we know the exact date/year that very project was conducted, its outcomes and (often, but not always) what was done internally with the findings. But, that knowledge falls on deaf ears all too often, because we’re not considered part of the team, and all the people in the new positions want to do is pick the lowest cost provider to conduct the (redundant) research for which they currently have budget.
Fully engaging your research vendor means you can leverage existing data (that no one on your team even knows exists) while spending that same budget to further flesh out the original findings or on another initiative altogether. Here’s a secret: we hate wasting time and money doing the same research over and over even more than you do. Let us help you get more out of your research budget. That’s when we can really contribute value above and beyond the parameters of the current project under consideration.
3. Thinking of research in terms of discrete projects, not as a process
The very best use of corporate research is an ongoing process that constantly improves its subject. Most research requests that hit my desk are ‘one-offs,’ basically designed in a vacuum with a ‘start’ and a ‘finish.’ There may be some reference to an older study (think two years, not two months old) but this is usually connected to a desire to ‘update’ the information, rather than building on and internalizing the findings from the first study. Additionally, we’ve all seen our share of tracking studies that just chug along spitting out the same banal ratings/scores month after month, year after year (see mistake #10). There are sometimes projects that are ‘multi-phased’ with both quantitative and qualitative elements. But what we rarely see is the type of meaningful, ongoing commitment to a research program that truly produces actionable results to excite and motivate upper management.
Issues raised by the initial study deserve deep follow-up to fully understand, but were unknown at the time of project initiation.
We’ve all heard the expression: “every good answer begs two additional questions,” and in the cost/benefit world of market research this is often very true. Those additional questions are nearly always “why is it that way?” and “how can we change/improve it?” While those questions are probably asked superficially in your original study, they do not attain the depth of what you really need to know. More commonly, issues raised by the initial study deserve deep follow-up to fully understand, but were unknown at the time of project initiation. Without a research process to provide continuous learning, this critical information will never see the light of day.
Survey questionnaire real estate is a finite resource – we’re limited by money and respondent time/interest constraints that apply to each project, but can be effectively addressed in an ongoing research process.
A well designed, on-going commitment to market research for a particular product, service, or message delivers that nuance, leaves fewer questions unanswered, parallels the development of said product/service/message (i.e., adapts to the changes inherent in any production roadmap to stay relevant and up-to-date), and can even change the very personality of the development team/company. A true commitment to always knowing how the market will react to your development is critical, and reduces the potentially exorbitant expense of missing the mark because your research was out-of-date, shallow, or didn’t reflect your current build.
4. Getting your vendor involved too late in the process
Our company is nearly always asked to provide examples of our experience in research, in our client’s industry/business, and in a specific methodology. After nearly 40 years in the business, this experience is usually quite impressive to our clients. However, sometimes the very next thing that occurs is that we’re given an RFP that has a fully baked out methodology that doesn’t match (or optimize) the goals of the project. And when we attempt to point this out, we’re often met with silence or deflected with “that’s what our CMO/VP Marketing/Marketing Manager wants,” which immediately negates the years of experience and thousands of successful projects which made the client want to work with us in the first place. Rather than helping the company craft a project that most accurately and efficiently meets their informational goals, we’re reduced to the role of order taker/fulfiller.
Your vendor can improve turn-around time, remove extraneous and expensive design features, and crystallize the data delivered into actionable insights.
Instead, choose your research vendor based upon trust, knowledge, experience, and ability to deliver and then include them in the early stages of research design. I guarantee, if you’ve chosen wisely, the vendor can improve turn-around time, remove extraneous and expensive design features, and crystallize the data delivered into actionable insights. Perhaps more importantly, the vendor will understand every aspect of the reasons the project is being undertaken, the politics and personalities of those on the project team, and can design research to better deliver the specific data required so your business can move forward.
We see more surveys and other research projects in a month (about 75 – 100) than many companies conduct in a decade. Let us put that experience to work from you from the outset; don’t handcuff us to a design that we know we can improve upon if given the chance.
5. Your data collection doesn’t focus strongly enough on negative issues
Sure, everyone on your company’s product team wants the survey to produce positive ratings for the new widget/service/message. But it’s very difficult to take only positive feedback and tell your C-suite or front-line staff: “our customers already like our product, now let’s do better.” Instead, focus on finding every opportunity to make your products and services better.
Honing in on potential obstacles to your offering doesn’t mean the research is trying to ‘kill it’—in fact, quite the opposite is true. Well-designed research intended to tease out all potential roadblocks creates data that is a veritable roadmap to success.
6. Conducting too much research in-house
Bluntly put, in-house market research professionals are often too close to the subject to produce fully objective research results. It could be as ‘innocent’ as allowing a few marketing generated adjectives to remain in your product description (that came directly from the marketing department/product group/engineering team). It could be leading and/or biased questions or even the omission of entire lines of critical questioning for the sake of expediency. Worse still, sometimes internal pressures and politics mean the research has virtually no chance of being objective.
We all understand the value of independent, outside counsel in legal matters, and the same is true for research.
Whatever the case, an objective outside vendor is better positioned to scrub all bias out of the survey design, questionnaire document, and data analysis. We all understand the value of independent, outside counsel in legal matters, and we all place more value on objective, third party white papers and case studies than we do on those created in-house with an explicit or implicit bias. The same is true for research. At its core, the goal of market research is to reduce risk. Don’t compromise that by allowing any bias to creep in. The risk on the back end is too great and if unchecked it will defeat the entire purpose of the research.
7. Excluding your research consultants from the implementation of the findings
Lots of companies have positions or even departments dedicated to ‘customer insights’ or ‘customer advocacy’ or, at the very least, like to include their ‘customer’s POV’ in internal training and marketing discussions. But politics, inertia, and incomplete understanding of all the market’s segments often cause this message to become diluted or even abandoned. If you’re serious about always including your customer’s perspective, you should include your market research consultant on all ‘customer voice’ task forces and initiatives. Like any objective third party, we will ensure that your message remains on-target and will help real change occur. Don’t let the job stop when customer data is delivered; continue it until it becomes part of your company’s DNA.
8. Conducting research that is broad, but shallow
Your customer insights department puts the word out that they’re conducting research and the requests come flying in from all quarters, with an array of disparate suggestions for “nice to know” questions that stray far from the core goals of the project. Resist the urge to add items which broaden the scope, but subtract from the profound data that a targeted, deep dive project can deliver. The very best—and most actionable—research depends on a design that focuses on the subtle and possibly complicated customer attitudes and behaviors around the issue at hand. Focus laser-like on the core objectives and ignore peripheral questions. If you find that some of the “off topic” questions are critically important to your company, then they probably deserve their own deep-dive research.
9. Over-reliance on online methodologies
The advent of online research capabilities has seduced many companies (both client and vendor) into using it nearly exclusively. At first blush, it’s easy to see why: it’s usually less expensive and faster to conduct. But dig a little deeper and troublesome underlying problems are raised.
Professional respondents. Straightliners, speeders, non-committed respondents: we’ve seen them all and they sprout like weeds within online panels. Don’t believe me? Just notice how much time online panel providers dedicate to assuring you they have the issue under control. This is a problem that gets worse over time, as more and more people succumb to the come-ons proliferating all over the internet. (Fun activity: Google “Get Paid to Do Surveys” and sift through just the top 20+ results.) Anybody remember ads and promises to get paid to take telephone surveys? Didn’t think so.
Probing and clarifying open-ended responses. Market research companies spend hours every day, every shift training their interviewers to probe for more open-ended responses, as well as to better clarify them into meaningful, useful answers. This simply doesn’t occur online. In fact, one of the ways every market research firm cleans online data is to search for meaningless drivel and/or random typing in open ended response boxes.
It is true that some online, open-ended responses can be detailed and on-point. But the vast majority are simplistic (“Why didn’t you like this concept?” Response: “Just didn’t like it.”) or off-topic (“Why didn’t you like this ad?” Response: “Your company sucks.”). Trained interviewers know to probe past those responses and/or turn them back on track. Additionally, real-time monitoring and verification assures that respondents are focused on the topics being researched. Neither of these occur with online research.
Difficult audiences are almost impossible to reach online. Don’t be fooled that business-to-business audiences, moderate-to-low incidence audiences, and high-level decision makers can be reached cheaply online, no matter what your panel provider says. Another industry side-effect of professional online respondents is that they very quickly figure out how to ‘fake’ their way past survey screeners. Unless screeners are very carefully written by experienced market research vendors (and alas, even sometimes then) these respondents can suss out the correct path to gain entry. And, because these difficult/low incidence audiences also typically pay the highest online survey fees (either real cash or points to be redeemed) there is greater incentive to do so. Once again, trained professional interviewers, real-time monitoring, and respondent verification are all designed to weed out these ‘fake’ respondents.
Will an online methodology be effective and not just low-cost?
This is not to say that all online research is bad. Rather, I’m suggesting we tilt the scale back towards the center, and encourage thoughtful investigation of the research methodology chosen, instead of blithely pushing ahead with an online panel/survey. The goal is always to be cost effective, but you can’t simply ignore the second word in that term: will the chosen methodology be effective and not just low-cost?
10. Tracking studies that are too long in the tooth
Most of our clients have some form of long-term tracking studies in place for customer satisfaction, awareness, market share, etc. Often, the results from these studies are boiled down into a couple of measurement scores that find their way onto management dashboards (and sometimes, even management compensation) and therefore become entrenched into customer research. While these trackers can be useful, there are a few important caveats.
Staleness over time. While changing the questions asked and metrics used to calculate scores every wave would defeat the entire purpose of a tracking study, it is also not effective to leave them in place too long. If you company has one (or more) of these trackers in place, chances are your products or services look significantly different now than they did when the tracker was originated. There may have been a few changes along the way—a question added here, or a wording change there—but most questions stay the same for trending/consistency reasons, and don’t necessarily reflect the current product environment.
Not the entire picture. Sometimes, too much relevance is put on the output from a deeply embedded tracking study, and no one—least of all upper management—is questioning what may be driving that (potentially watered down) calculation. Off-setting actions—a price decrease around the same time customer service response times are increasing, for example—can produce the same overall score, but miss both sides of the changes in the market. If you aren’t spinning off ad-hoc studies based upon the findings of your trackers, you a) aren’t using the tracker to the fullest, and/or b) you need to update your tracking questions to be more meaningful.
Like other company processes and procedures. tracking studies –need to evolve over time. Changes are made proactively to the product or service in order to adapt to external forces/changes, and tracking measurements/scores must be transitioned as well. Market research vendors are masters at this transition. Tracking studies can provide critical, ongoing data to help steer your company in the right direction, but only if they can adapt and change to accurately reflect your company’s needs right now, and not the situation that existed when the study was conceived.
Bonus observation: If you’ve read this far, you deserve a little something extra.
Procurement systems/agencies are the bane of the service industry. I don’t know enough about the use of procurement systems to speak to their value at saving costs across commodities or consumable products. What I do know is that they aren’t doing your company any favors when purchasing consultative services. The ‘one size fits everything’ approach is ludicrous, cumbersome, maddening and inefficient. Ever try to fit survey sample size options into a unit price table to calculate total price? Doesn’t work. Just imagine how hard it is to communicate a creative, non-traditional design via a mechanism that, by definition, forces every submission to look the same. I can’t imagine a scenario in which it is better to have another layer inserted between both sides in an on-going, working consultative relationship. Buying pencils? Maybe. Trying to explain the various components of a multi-phase, dual methodology research project? Nope, not even close. And of course, this additional layer inserts procurement people who have little to no research experience or expertise into what is now a three-party collaboration.
We know our clients share some of our frustration on this issue, and it is typically far removed from their purview. Still, it is important to be aware that you may need to help your most trusted, creative, and loyal research vendors navigate this obstacle in order to provide an accurate, effective proposal.
There you have it—the view from my seat. Remember, I present these from an authentic desire to improve your company’s research program. After all, the better your return from research efforts, the more your company will commit to research in the future.
Don’t agree with these? Did I miss some points? Want to tell us the things that drive you crazy about research vendors? Start a dialog and let’s discuss how we can work on eliminating these issues from your research plan. 503.977.6762 or firstname.lastname@example.org