Intermediaries loyalty: Is NPS the best metric to measure experience with a particular tool?

I found myself trying to answer this question while working on a project for re-designing a web platform for an insurance company broker’s. We wanted to know how good was their experience and if we were improving it, at all.

NPS is now The Queen of CX metrics and the consultancy company that was helping us in the project, immediately told us to measure it. After the MVP was release we activate a Hotjar survey with the famous question, How likely is for you to recommend this tool to a colleague? We couldn’t ask for friends and family, since this is a working tool… Unless your family is all about brokers, otherwise it didn’t make sense.

The other thing that wasn’t quite accurate, was that brokers are sort of obligate to use the platform, otherwise they just can’t work with that insurance company or they have to go to one of their brunches, in very reduce opening hours, and wait for an executive to do the whatever they need to do for them… so how could that question be useful for us?

Eventually we stop the survey and I will go deeper into the whys later, but first I’ll like to point out some learnings we had while doing this question that way:

I LOVE THE COMPANY, NOT THE TOOL

NPS score wasn’t bad (23 points), but they would leave unhappy comments and send us emails about how bad the platform was working. So how could the NPS be that good? We understood that they are very loyal to the company, even if the experience with the platform sucks. They wanted to support the company they love, but they were struggling to use the new tool; so NPS did worked for loyalty measurement, but it didn’t correspond with a good experience.

DON’T MAKE ME THINK

One of the main reasons they were unhappy with the platform, was that they were very much used to the old one; the experience was destroy in the exact moment they couldn’t find a particular feature. They would abandoned and return to the old one after one or two failures. By the way, we had both platform working for a long time, because it was unthinkable to leave this guys without a tool to complete their sales; that didn’t help adoption, at all.

THIS QUESTION? AGAIN?

Having a survey every time they would submit a policy through the new platform was annoying; this is a working tool, remember? Imagine if you would get a question every time you send and email from Gmail or you create a new bill from a SAP platform? Not a good idea.

Here are the main reasons for why we decided to stop measuring NPS to understand the experience:

1. The experience we need to measure happens in an specific moment, and for that we need a here and now result like a C-SAT survey can offer. NPS should be consider to evaluate the hole experience with the company, and not only through the interactions with the platform or any other tool in particular.

2. NPS measures loyalty with the company and not with the company’s tools or particular features. Considering this, it would only be correct to use NPS for this project, if we were a software company that develops broker’s management platforms.

3. The experience the broker have with the platform, it will affect its total experience with the company, and it’s loyalty, but only in an indirect way. It’s necessary to measure NPS at an organizational level, in order to understand how this or any particular project impacts in the global NPS. The question here should then be, How likely is for you to recommend to work with this company to a colleague?

We then studied a variety of possible indicators for measuring experience among which we consider…

• CES (Customer Effort Score): It’s a survey that asks the customer “How much effort did you have to expend to handle your request?” It helps to understand if the action we take actually makes the task easier to the user.
• C-SAT (Customer Satisfaction): This survey asks a customer how satisfied they are with a recent interaction–often a purchase or a customer service call–on a rating scale. is the most popular transactional metric.
• CHURN RATE: amount of customers or subscribers who cut ties with your service or company during a given time period.
• PROBLEM RESOLUTION TIME: How fast can you solve a customer’s problem? This question also tries to understand how many people does the customer speak with before the issue is solved?
• FIRST RESPONSE TIME: This is the amount of time it takes a company to respond to a customer query.

We decided to go for C-SAT and Problem Resolution Time, as a satisfaction metric, that we would triangulate with usability information from Google Analytics to provide a complete Experience report.

The results were now much consistent with the direct feedback we had from the brokers and we could track their feelings every time we would release a new feature.

CONCLUSIONS

Be careful with metrics fashions; not every KPI is suitable for your project. Metrics can and sometimes have to be customized, considering the goal and the user we are dealing with.

Intermediaries and final customers, are not the same, either is their experience: for companies who work with this model, considering an NPS metric for each of them could be important; one is loyal to the way of doing business and the other to the product or service.

Tools and other touchpoints, are one part of a hole experience, leveraging a question like NPS in any of those in particular, won’t give you experience insights. At least not in a consistent way.