The Problem With Customer Opinion Surveys and Strategy

For the past few years I’ve noticed that more and more banking and banking technology articles about the future of banking seem to be informed by dubious claims. The writers derive these claims from customer opinion polls and immediately jump to conclusions without understanding survey methodology. Some researchers and consultants conduct customer opinion surveys with biased questions and limited responses. The results support pre-determined positions.

For instance, I recently came across a customer opinion survey that says that if the top ten American banks don’t address “customer frustrations,” the researchers project 11% of their consumer deposit customers will flee. They further indicate that this will result in a loss of 5% of the total deposits held by those banks, all within one year. The researchers call these customer actions “Historic Levels of Defections.” Based on the list of potential customer frustrations researchers put together, they determined that 11% of the customers plan to leave their banks. I don’t contest that 11% of people answering the survey are unhappy. However, there is a major leap between people saying that they consider leaving to acting on it. For better or for worse, people find it very difficult to switch banks. This widely known fact has protected banks from each other and from new entrants.

Further, I question the researchers’ analysis when they note that they had successfully predicted similar defections in 2011, 2013, and 2015. However, the Fed deposit data for the top ten banks show growth year after year. In fact, between September 2010 and September 2018, the top ten retail banks’ deposits grew by 50.2%. During the same time, all banks had an increase in deposits of 46.5%. Therefore, during the time-period in question, the top ten banks grew faster than the overall banking sector. The top ten banks grew 11%, 7.3% and 2.4% each year 2011, 2013 and 2015 respectively. The rate of increase for the top ten banks, and all banks, has slowed down. Regardless, in none of the years between 2010 and 2018 did deposits decrease. This trend indicates that a decrease of 5% in deposits in one year, and 11% in customers, is highly unlikely for the top ten retail banks.

The vast gap between customers’ opinions, recollections, and perceptions and facts represents a barrier to understanding data. Unfortunately, organizations often fail to make the distinction between these two different data sets. A few years ago, I conducted a customer usage survey designed to understand perceived channel use. Banking customers reported using the branch channel significantly more than any channel, followed by ATMs, digital, call center, and automated telephone. When I compared the customers’ recollections to the same customers’ channel use data I saw stark differences. Channel data revealed strongest use of digital followed by automated phone, ATMs, branches, and call center.

What could explain such a difference? We took great care in designing questions and defining the terms contained in the survey. For instance, we defined using a branch ATM not as a branch visit, but as an ATM visit. However, when we interviewed some of the survey takers this distinction had apparently confused them. Some thought that visiting an ATM only at a branch counted as both, or as a branch visit only. It was immaterial to the customers how we defined the channels. Similarly, if a customer called the call center and interacted with the automated system only, some customers viewed this method as a call center interaction. We were also surprised to find that some survey takers admitted to answering the questions incorrectly. These people claimed to use the branch or the call center because they didn’t want the bank to close branches or call centers if they used them sparingly or not at all. Even though we felt we had explained the objective of the survey to those taking it the customers still made their own assumptions.

Value exists in understanding customers’ perceptions, opinions, and recollections. People’s attitudes towards brands, product, services, and experiences should be one of the data points considered by an organization, particularly in their design efforts. The danger lies in using these type of surveys and rationale for building strategy. Consultants should not present these sorts of findings as fact or use their expertise to push ideas without supporting factual data. Decision makers must be aware of the information’s source and methodologies used to obtaining it, including full copy of survey questions and directions. If a consultant or researcher doesn’t share this information or cannot defend their analysis, companies should exercise caution in their approach to the information.