FSN: Enterprise Search – the next big thing in Business Intelligence?

Enterprise Search – the next big thing in Business Intelligence?

24th September 2007
Since the appearance of ‘Query and Analysis' tools in the mid-eighties business users have become accustomed to slicing and dicing data, mining for information and generating reports. But all of this effort has been concentrated on structured matter, essentially numeric accounting data, which lends itself to a methodical and logical path of enquiry.

However, the last decade has seen an explosion in non-structured information, for example commentary, ad-hoc explanations, emails, PowerPoint presentations and PDF documents. Many of these so called unstructured sources of information –essentially text, provide a rich repository of information which in a corporate sense has a very short shelf life. We read and review documents or emails and then by and large discard or archive the content but have no ready means of retrieving it. But incredibly, 70% of the world's information is locked up in corporate databases. Hidden from view and guarded behind firewalls, valuable information lies dormant, inaccessible and difficult to query using conventional tools.

Yet combination of structured and unstructured data is capable of yielding powerful synergies. Many of us make decisions based on numbers but it is often supplemented with informal data sources and documents that provide context and texture to the decision. In other words the coalescence of numeric and non-numeric data can create powerful insights that neither category of data can necessarily provide on its own.

At the same time the Internet provides a striking example of the ease with which unstructured information can be retrieved. Some would undoubtedly argue that much of the information is worthless and it is sometimes difficult to distinguish the ‘wood from the trees', nevertheless for many users, Google represents a transformational and liberating technology that allows anybody to search for information in a relatively painless way.

So how come information is so difficult to search across business systems? Why can't business users simply have a search tool that allows them to retrieve relevant information from across the business with the same apparent ease as Google?

These are not unreasonable questions according to Don Campbell, Chief Technology Officer of Cognos, who took time out to talk to FSN when he was travelling through London last week. He is convinced that a Google style search capability for corporate information is tremendously important and that's noteworthy coming from the man that leads the research effort at one of the world's largest BI vendors. “The value of search increases as the complexity of the issue grows. In other words the more difficult it is to find information the more one values the answer,” he told FSN.

With less than 20 percent adoption of systems by end users there is clearly a need to provide better enabling tools. “Search helps users get to know the data,” says Campbell . But he sees the value of search as being much more than simply the retrieval of existing information. “You have to ask the question, where to next?” he adds. “Once you've identified the problem one might want to ask what else has similar dimensions, i.e. can insight be gained from something with a similar context?”, he challenges.

“Search shouldn't just answer questions about the data that exists in say a performance management system but it should also create answers that are new. Search becomes an authoring opportunity allowing end users to be their own information authors. One of the difficulties at the moment is that searches of highly valuable unstructured data are not integrated into reporting.”

“Another issue is that Google is statistically based on key words that match but is not linguistically based. So it does not understand the question – it just matches the key words. The key to search is to understand what the enquirer is really asking,” Campbell told FSN.

So what is Cognos doing to advance developments around enterprise search? “We are working in our labs and have grown technology with the aid of partners and through investments in research in academia. I think that we will reach an inflection point within a year beyond which we will start to show the value of unstructured data within a performance management system. The solution is likely to be based on standard technology but will be applied in a proprietary way.”

“The challenge is how to retrieve and display information in a valuable way. We have lots of people working on it and it is definitely one of our top level projects and I believe that the value of search will be accepted and meeting user expectations within 5 years,” says Campbell .

But it would be a mistake to believe that the obstacles to enterprise search are purely technical. Better search capability will throw the spotlight on data quality and the consistency of data definition across the enterprise. Retrieving data from across the enterprise could highlight stark differences in interpretation of data used in different business segments. There will also be some knotty issues around data security. For example, how does one limit access to key words in certain contexts but not others.

Despite these difficulties, search offers the prospect of unleashing a step change in BI capability. More than a year has elapsed since Google's announcement of Google OneBox for Enterprise , but progress is being made and the ultimate prize is probably worth waiting for.

About Us