Blog Layout


Quantifying impact: The art & science of measuring the value of design

This article is based on a recent presentation that Cole Armstrong and Suzi Goodwin from NeuroSpot gave to the CX Collective, a global community of customer advocates run out of Aotearoa New Zealand. We greatly appreciate the input from Sarah Clearwater and Baruk Jacob from the CX Collective, and from the participants on the day.

A key reason for tackling this topic was because we know the work that goes on in organisations across the country, work delivered by people passionate about the customer experience, work that provides real value to organisations. But work which is at times hard to justify, seemingly impossible to quantify and difficult to see the tangible evidence of positive shifts in behaviour. Work that at times can be viewed as a cost centre rather than a value driver. 


CX as a function is not always well understood at all levels of a company – McKinsey reports show that companies with an embedded design practice are growing at nearly twice the rate of their peers, yet only a third of CEO's and their direct reports could state confidently how design delivers on business metrics. We wanted to share ways that you could actively measure not only the impact you have delivered, but the potential of testing different assumptions and interventions from the beginning of a project, giving you opportunities to set every project up for success, and highlighting the value you provide.

You need to quantify your impact

In the fast-paced world where organisations are constantly moving onto the next thing, we rarely get a chance to look back and see how what we delivered made a difference. Retrospectives are great, but not always well timed or they can be rushed as a box ticking exercise as we all try to focus on our current challenge. If quantifying your impact takes extra effort, why bother?


  • We want to make a positive difference. If we don't measure our intervention, we can't say for sure that it worked. Maybe all that resource resulted in no change, or potentially, having made things worse?


  • Quantifying impact helps mitigate risk. Every new innovation or experience brings a measure of risk, and it takes up a lot of stakeholder headspace. How can you alleviate this?


  • Look at the cost/benefit. Experimentation helps you see if the expected gain justifies the resources required.


  • Build your professional standing. Building trust with your stakeholders, showing that you deliver well executed interventions with measurable outcomes can help with internal support for your next challenge.

Use experimentation to articulate value

Hand drawn diagram of a hypothetical test, showing users going into the test enviroment, being directed to either Option A or B, and then the results.

Experimentation goes beyond A|B tests, and you're probably familiar with the basics from high school science - you have your hypothesis, a control and treatment group, there's some analysis and an outcome. That's pretty much how it works in design too. 


You'll need the following:


Objective: a clear sense of what you want to achieve - the behaviour or response that you expect to see. E.g. greater satisfaction with our insurance claims process.


Metric: something observable, reliable and related to your objective. Preferably actual behaviour, and ideally already collected - e.g. people tagging onto public transport, sales records. Or maybe you need to send someone out with a clipboard to count people. Other metric types can be used as well - claimed behaviour (however this can introduce human bias) or perceptions (e.g. via a survey).

Users: you need a group of people to test on. How many depends on a number of factors but for some projects 100-200 per group could be enough.


Intervention: this is the thing you actually want to test, the change you are wanting to make to a process, product or experience.


Randomisation: your users need to be randomly assigned to your control and treatment groups. Randomisation means you can be confident you're not inadvertently introducing bias (e.g. males vs females). While straightforward in the digital space, in other settings this could involve splitting a call centre in two, sending two different letters or changing instore displays from one hour or day to the next.


What will the decision drive?: consider all the potential outcomes of the experiment. What decisions would it drive if your outcome metric shot up, or down - or stayed the same?


Having these clear in your head will help you both articulate the experiment and its intent to stakeholders, as well as provide an instruction manual of sorts to the team implementing it.

Experimentation adds value earlier in the design process

One of the key takeout’s from our presentation is this: don't wait till you're delivering the work to experiment. There is often far greater value in incorporating experimentation towards the start of the design process. Why not test:


  • Stakeholder assumptions. Experimentation at this stage helps ensure you focus on the issues that matter, and allows you to engage stakeholders into the process, getting buy in for down the track. 



  • Alternative concepts. How do you objectively know the intervention your team is looking at is better than the one tossed aside? Or, how can you test your intervention at a small scale, to estimate the impact it could have when rolled out? Experimenting at this stage gives you the opportunity to test some of the crazier, counter-intuitive ideas that could be a huge competitive advantage for your organisation.

This is not to say there is no merit in experimentation at the end of a project - it can help ensure you've delivered a project of value and should absolutely be used. You'll want to be able to share those results with stakeholders, and add them to your CV...

Finally, your experiments need to be ethical

Ensuring you have some ethical boundaries in your experiments means you're avoiding brand damage and a loss of trust to your organisation, and hopefully not keeping yourself awake at night with a moral dilemma. Some questions to keep in mind:


  • Is the objective ethical? This should be universal to all projects.


  • Does it infringe on your user's welfare? The objective might be worthwhile, but it shouldn't leave people worse off.

 

  • Is there anything more than minimal risk? Holding a conversation for example, carries a risk - what's the risk from your experiment?


And at the end of the day, is it ethical to offer a new product or service that makes people worse off?

This is just a summary of what we covered, the video of our presentation and the resources discussed in the session are all available from the CX Collective Learning Library. But we're here to help too, so if you have any questions about what we've discussed or have a challenge you need a sounding board on - get in touch. You can also check out some of our other work here.

Remember, there are no bad ideas - just unproven ones.

By Cole Armstrong September 4, 2024
In recent discussions about customer behaviour, a recurring theme has emerged: the belief that providing more information will lead to better decision-making (“If only they knew … then they would …”). While this perspective isn't entirely misplaced, it overlooks a crucial nuance. It's not merely what people know that drives their behaviour, but rather what information is most salient and readily springs to mind in the moment of decision. The Limitation of Knowledge Alone It's easy to assume that if customers simply knew more, they would make better decisions. For instance, knowing how to budget effectively or save money can indeed be useful. Knowing that I should save more for retirement or for investments. That I should eat less red meat for health or sustainability reasons. That I should be careful about what information I share online. These are all real-life examples of issues I’ve worked on where educating customers (or users) might have been a solution – and where helping people ‘know’ why they should do it failed to shift the needle. The critical factor isn't just having the information, but rather how accessible and prominent that information is when it’s needed. The question then is not do I know something, but rather does it spring to mind when it can actually shape my decisions? The Salience of Information in Decision-Making Consider a scenario where a customer interacts with a product or service. The decision-making process is often instantaneous and intuitive, driven by what immediately comes to mind rather than a comprehensive evaluation of all known benefits. This means that the information most salient at the moment—whether it's the immediate cost or a specific feature—has a more significant impact than the general knowledge a customer might have. As an example, I was involved in a project where my client was responding to new environmental regulations – and wanted to find out how to do this without upsetting their customers. The initial solution didn’t work because what was salient – and thus impacted customers responses – was the benefit to the brand not the benefit to the environment. While both benefits were ‘known’ by customers, it wasn’t till small tweaks were made to subtly emphasise and customer support for such actions, that complaints dropped off. A Case in Point: Council Rates Another example where this idea is relevant are the rates (tax) bills that New Zealanders receive from their local council. While residents might be well aware of the benefits their rates support—such as parks, libraries, and community services—this knowledge doesn’t always translate into a positive reaction when the bill arrives. For many, the first thing that springs to mind is the expense, not the benefits. In my case, receiving a $1,000 council rates bill (these are billed quarterly) was a stark reminder of this phenomenon. Despite understanding the value provided by these rates, the immediate focus was on the financial burden. If the benefits were more salient at the time of billing, the experience could be more positive. For instance, if the rates bill included a summary of the value received from various services—like a breakdown showing the value I obtain from the library based on the number of books borrowed—this could shift the focus from the cost to the benefits. In my case, borrowing 35 books over three months at an estimated value of $30 each amount to $1,050 worth of benefits, which highlights the value received far beyond the cost. And of course there are other services that I use as well – my local park that I go running in, the playgrounds that I take my kids to, the roads I drive along or the public transport that is subsidised. I ‘know’ these things, but do they immediately spring to mind when I see my rates bill? Seizing the Opportunity This concept of salience extends beyond council rates. In various customer interactions—such as bills, invoices, loyalty schemes, and product renewals—the opportunity lies in enhancing the salience of positive attributes at the critical moment of engagement. To effectively leverage this understanding, organisations should focus on making the benefits of their products or services more prominent when customers are most engaged and in a way that is relevant to the context. This means designing communications and touchpoints that highlight the value received, not just the cost or features.  While knowledge is important, it's the salience of that knowledge at the moment of decision that truly influences behaviour. By ensuring that the most relevant and positive information is top-of-mind, organisations can improve customer satisfaction and decision-making outcomes.
By Cole Armstrong July 24, 2024
Navigating the Shifting Landscape of Design
By Cole Armstrong March 15, 2024
How do we create persuasive touchpoints that make a difference? By considering how simple ways of reframing our messages, using insights from psychology and behavioural science, can create greater motivation to act.
Show More
Share by: