SAP is at the centre of a fierce controversy following a US based advertising campaign which claims that companies that run SAP are 32 percent more profitable than peers that do not run SAP. A slanging match broke out following counterclaims by Nucleus Research Inc, a small US based research company, that SAP users are in fact 20 percent less profitable than their peers – a swing of 52 percent! Gary Simon, FSN's managing editor looks at the arguments.
A fierce debate is raging on the internet following claims by SAP that their customers have a strong track record of outperforming their peers. The claim that companies that run SAP are 32 percent more profitable is based on a 2005 Stratascope Inc. analysis of the operating margin of all non-financial services industry companies listed on the NASDAQ and NYSE exchanges. The counterclaim by Nucleus Research Inc is based on an analysis of the Return on Equity, (ROE) of quoted companies listed as customers on SAP's web site.
The SAP commissioned study considered 4,600 listed organisations but eliminated 800 financial services companies, (which measure performance differently from other trading concerns), enterprises with less than $50 million turnover and companies which were recently de-listed. This resulted in a SAP population of 587 companies versus a non-SAP population of 2,769 companies. The average operating profit margin for SAP users was 10.3%, compared to 7.8% for non-SAP companies, or a 32% difference.
Nucleus did not have access to the SAP study so it sifted through the ROE of all of the SAP customers it could glean from the SAP web-site using Bloomberg as its source of financial information. It then compared those customers' ROE with industry average ROEs from Hoovers available through Yahoo! Finance. Based on its analysis of 81 SAP customers, Nucleus found that SAP customers had an ROE of 12.6 percent, compared to an "industry average" ROE of 15.7 percent, i.e. 20 percent lower. But it is this averaging of average sector ROEs that is most hotly contested by SAP.
William A. Wohl, Sr., Vice President, SAP AG told FSN, "Anyone straight out of high school knows the danger of averaging an average. It is ridiculous, shoddy work which compares apples to rotten oranges," he said.
Clearly the methodologies used by the two companies (in so far as they are published) are quite different in scale and content. The SAP study is an exhaustive one based on a very large sample population, whereas the Nucleus research is based on a smaller number of SAP customers. Furthermore, the Nucleus approach is based on ROE, a very broad brush measure of performance, whereas the SAP research relies on operating margin which some would argue is a more direct measure of profit.
Ian Campbell, CEO of Nucleus Research defended his use of the ROE metric as being a pragmatic measure that is published and readily available. But one company unwittingly included in the Nucleus study criticised the use of ROE as potentially very misleading because it included acquisition-related, one-time charges and expenses.
Notwithstanding the limitations of the "industry average" the Nucleus table of results appears to be correct, though SAP takes issue with the industry classifications used. However, whether Nucleus has satisfactorily shown "cause and effect" is open to question. Without access to SAP's knowledge base, their study of necessity draws on very limited information about the applications being used, for example, how long they have been in service and the extent of their deployment, for example, in-house, shared service centre or outsourcing arrangement. Furthermore, William Wohl told FSN that some of the customers included in Nucleus' study were not yet live on SAP. Additionally, companies using CRM, SCM and PLM applications appear to be lumped together to produce the overall headline grabbing finding that "SAP customers are 20 percent less profitable than their peers." However, regardless of its possible shortcomings the data that Nucleus Research used is at least available on their web site so that readers can reach their own conclusions about its validity.
But the SAP study also has vulnerabilities. Ian Campbell points to the fact that SAP refuses to publish its data. "As far as I am concerned, until it's published it doesn't exist," he told FSN, "adding that the research was not independent but funded by SAP."
However, Wohl told FSN that leading companies in all sectors fund research to support their advertising campaigns. "We are not saying that companies should base their purchasing decisions on our advertising campaign. These are complex and sophisticated products used by complex multinationals. The advertising campaign is simply a way for us to engage with potential customers but we fully expect and encourage them to do their own investigations using professional assistance."
Fundamentally, what should be at issue in this debate is whether the best run companies use SAP or not, but it has disintegrated into an argument about who has the best methodology. Wohl told FSN, "We believe that an ERP system deployed on enterprise wide basis gives management a strong handle on performance and our research shows that the best run companies happen to run SAP."
But the debate has at least been valuable in raising broader questions about the contribution of ERP systems to overall corporate performance and the validity of a single measure as blunt as ROE or operating margin. Jeremy Roche , CEO of CODA Group Plc, the financials software house, agrees. "I suspect that the actual number of customers that are more profitable as a result of using SAP could be anywhere between the numbers quoted by SAP and Nucleus or even outside the range. There is more to looking at the performance of a company than a single instance of its ERP software. The problem that this debate really highlights is that companies are unable to measure the Return on Investment (ROI) from their ERP because of the scale of their ERP programme. By the time that they have completed the implementation, the business has moved on, entered new markets and made significant acquisitions and disposals so that the initial business case is buried in the detail."
Given the complexity of calculating economic value added, any attempt to simply correlate ERP investment with "well run companies" quickly runs into difficulty because it ignores the relative contribution of everything else, such as markets, people, geographies, performance management applications, business intelligence software, customers and economic cycles to name but a few. Nevertheless, many ERP systems are justified on the basis that they boost productivity, profitability and competitiveness and it would be surprising if the benefit of an ERP system in the longer term did not show through somewhere in the financials. Wohl adds, "It is also unlikely that SAP could enjoy its present level of market dominance and growth if customers did not believe in its merits." The idea that companies that use SAP consistently under perform relative to their peers seems wildly inconsistent with its long track record of success, share price and market gains. Unless one believes that it is possible to hoodwink a whole market on a grand scale, this alone should cause thinking people to treat Nucleus' research with some caution.
Competitors have been quick to take advantage of SAP's predicament but have no reason to be gleeful since the debate highlights an industry-wide issue of significant importance. According to CODA's Roche, the problem of measuring returns from ERP systems is beginning to have a longer term effect. "I don't think that we are going to see so many companies upgrading their ERP systems or moving to ERP systems in the future." In a move reflecting the latest trends in outsourcing he told FSN, "I believe we are beginning to see a shift to smaller more focused IT projects around which it is easier to measure a return on investment."
Whatever Nucleus Research's motives for producing the research or the wisdom of the methods it applied it has been helpful in one respect. It does expose the real gap in our knowledge about the contribution that ERP systems, not just SAP, make to business performance and make us question the appropriateness of simple financial measures. In the meantime, perhaps all software CD's should carry the following health warning. "ERP performance measures can go up as well as down depending on the metrics employed. Past performance is no guide to future performance. Do your own research and obtain professional advice before making purchasing decisions!"
