How to get the most out of your product testing

In this article, MMR’s Director of Product Testing Excellence, Simon Harris shares his considerable experience solving product testing conundrums for every conceivable category - in all corners of the world.

1. There’s no one-size fits all solution. People often ask me what the ideal approach will be for their particular set of circumstances. The solution is rarely ‘off-the-peg’. As product testing specialists, it is our responsibility to deeply understand the client’s objectives and weigh up the alternatives.

A recent client briefing stipulated “we always test products branded” – implying that we should do the same. The client was interested to learn more about sensory drivers. In this instance, we were compelled to point out that running the test branded risked results integrity, because people’s perceptions and experiences of a brand will impact their judgement. Instead, we proposed a blind approach which we know will generate higher quality data. To the client’s credit, they went with the recommendation, which eventually led to a significant uplift in appeal for a revised product formulation.

My advice is resist specifying a particular approach. Focus on what you need to learn, listing any limitations that we need to be aware of. Let us do the thinking around methodology!

2. People find it hard to describe a product experience. We often test products to provide R&D teams with incisive feedback that supports product improvement goals. To do that, R&D teams need granular detail. However, the average consumer – who doesn’t consider your category for more than a few moments, isn’t capable of articulating what you probably want to know - let alone be fully aware of what is driving their opinions.

My advice is that you keep quantitative consumer surveys simple! The increasing use of chat bots and video provide effective new ways of collecting spontaneous feedback – and in everyday language. We often recommend that clients boost understanding with qualitative deep dives or tap into the power of sensory heightened consumers who are better able to explain what is really going on.

3. You get richer data from comparisons. It’s a well proven fact. People find it easier to talk about Product A when there is Product B to compare it with. For instances when a consumer rates a product in isolation, you will pick up a lot of generic category perceptions. When a second product is fielded, consumers will find new energy and will latch onto the sensory characteristics that justify why they might prefer A over B.

There are circumstances where monadic testing will be the way forward. However, in cases of product improvement or optimization we would typically recommend sequential testing because it is more sensitive – and that is what we all want!

4. Don’t use significance testing for everything. I’ll leave the technicalities of sig testing for another day, but if sig testing is sought, then it is important that we apply some critical thinking. The 95% two-tailed test is the industry standard across the CPG industry. However, I believe that it isn’t always the most appropriate - and that a lack of due diligence here could risk undermining your investment.

My advice is that you think very hard about what you are trying to prove and then use an appropriate level of significance.

• If you want superiority, then a 95% one-tailed superiority test will provide you with robust results.
• If you want a new formulation to achieve parity with current, then an 80% one-tailed inferiority test will give you the confidence you are not worse off than current.
• If you are just exploring differences, then a two-tailed test is best. However, imposing a 95% level of confidence means that you risk not finding any actionable differences. Therefore, consider employing a lower level of confidence to allow you to uncover more trends if the nature of the test is explorative. This is especially true for benchmarking studies and programs. In most sectors - and in most countries - the standard of product competition is strong, so we need to allow for this when setting confidence levels.

5. Test the full user experience. Where the standard of existing products is high – and let’s face it, this is pretty much the case everywhere nowadays - it’s no surprise to learn that your chances of achieving product preference or superiority are slim. But all is not lost. There are opportunities to pull ahead on the overall user experience. Sensory science tells us about cross-modal sensory perception – which basically means that information received by one sense (say, product appearance) impacts how information from another sense (say, product taste) is decoded. It means that people’s experience of your product is impacted by their experience of your pack (and more besides). How many product tests account for the full user journey? Only a few. Testing a liquid in isolation fails to pick up on opportunities to drive up perception of taste by making changes to everything that happens before the sip test!

So, when testing your product, think about expanding your understanding by measuring as much of the ‘real world’ user experience as you possibly can. You might unearth clever new ways to drive up perceived superiority. It’s still all to play for!

In 2022, MMR is pulling out all the stops to help client innovation make more impact. If you want to elevate your next product test, and tap into opportunities across the full user experience, then please get in touch for a discussion on your product testing program can be improved 👇