Why Search Still Matters

I was chatting with Jeff Fried over at BA Insight yesterday, and I was reminded of a conversation we had a few years back while in either New York City or Washington DC (honestly, I can’t remember which) where Jeff was teaching a workshop on architecting enterprise search, or something along those lines. What I remember from the conversation, however, was one simple truth: most searches fail because people don’t know what to look for.

This may sound like a no-brainer, but there’s some complexity behind every successful search result:

1. You need to be able to somewhat accurately describe what it is you’re looking for
2. Someone needed to upload relevant content to your search
3. The content needs to be tagged appropriately to match your description
4. Even if all of those things fall into place, you still need to have the right permissions to access that content
5. And in this day and age of distributed systems, where some of the data you require could be sitting on-premises somewhere rather than out in the cloud, the network connection to that information resource must be good

One topic I often talk about is the power of social as another layer of the search experience — and how the social graph and machine-learning can improve on these steps. Arguably, one of the reasons we reach out to others within our vast collaborative landscapes is so that we can share what we know, then discuss and dissect and ultimately uncover that unknown. A major roadblock to this basic premise is how to easily access and synthesize what it is we already know, which is often buried within our various data silos.

Search is critical to every enterprise collaboration platform. Whether it be a structured enterprise collaboration management (ECM) platform, our email system, or even our social networking tools, information workers increasingly rely on these systems of record to help them connect, relate and communicate. Some of their features make communication with geographically remote partners and peers almost seamless, as if they were right across the hall. Yet, the best social or content management features in the world cannot make up for the fact that search, for most organizations, is fundamentally broken.

The ties between search and collaboration are simple: if you can’t find your data or content, collaboration will be limited. How can you collaborate if you cannot correlate the dialogue with your content? So I go back to my original premise, that most searches fail because people don’t know what to look for. That isn’t to say that they don’t know what they want — they may just be using the wrong description, the content that might be useful has not been properly shared, the content may not have been properly classified and tagged, their permissions may not allow them visibility or access to the content, or the server on which the document sits might be down.

When you think about it, there are multiple points of failure for finding the content you need.

A few years back, my former consulting company did a lot of pro bono work for startups and entrepreneurs coming out of UC Berkeley and Stanford. Sometimes these projects led to consulting projects, but most of the time not. Our decisions to take on these projects were mostly based on how interesting we found the technologies involved. One startup had raised some angel investment and was aggressively pursuing venture capital, but had received feedback that their product roadmap, as well as their overall marketing strategy, needed some help. They had developed algorithms and a novel approach to building a semantic search solution.

For those unfamiliar with the concept of semantic search, it is the idea that you can improve on search by interpreting the intent and contextual meaning of your content, generating more meaningful results than predictive relevancy in traditional search.

While the technology was fascinating and the discussions with the company’s founder were deeply engaging, I questioned the ability of this technology to adequately interpret and intelligently map end user sentiment to content and metadata, or “data about data,” improving the overall search experience. Sentiment analysis is an incredibly difficult thing to automate, much less deliver within mainstream platforms.

That was then, but we now have a method for providing a robust, sentiment-based layer to our structured collaboration platforms: social collaboration. Even the search leaders recognize that they cannot completely replace human interaction (at least not yet) as the ultimate semantic classification mechanism. Google and Bing include some elements of semantic search within their platforms, but also recognize the need for integrated social capabilities to provide deeper sentiment awareness.

Collaboration — especially within the enterprise — tends to be around content, whether content within documents and other artifacts, or content within lists, comments and conversations. As people interact through social collaboration, we consume, assign, filter, share, comment and edit that content.  In effect, we are adding metadata as we go, building into our collaboration platforms semantic relationships between what is being discussed and the content being stored within these systems. We are moving from “systems of record,” traditional data infrastructures and content stores to “systems of engagement,” where the focus is not so concerned about where your data and the resulting conversation are being stored. The argument of structured collaboration versus unstructured collaboration is irrelevant here as the lines have become blurred. The focus has shifted from the system to the needs of the business, and to the specific requirements of the employee or customer.

This highly personal view of the collaboration landscape brings to the forefront the importance of search. On a daily basis, we rely on different types of search: a browser-based query to answer a question, a quick navigation tool on the desktop to bypass a few clicks to find content, files or programs, or for more advanced questions out to our knowledgebase and our personal network. However, search alone cannot solve our more complex search criteria – which is where the contextual benefits of social interactions can enrich our search experience. Collaboration improves when the content (whether structured content or unstructured data) you want to discuss, edit, rate, like or share is at your fingertips, and comes included with the comments, edits, ratings, likes and connections of people within and outside of your network. If you can’t find the data or documents that you need, your ability to collaborate with your team or with customers becomes limited, because the conversation happens out of context of the content. The more your content is associated with social collaboration activities, the more searchable – and findable – your content becomes.

Search amplifies the reach and impact of collaboration by making relevant content and conversations easy to find, although it is most effective when joined by the contextual outputs of social interactions. The actual search experience may be nothing like the traditional, consumer journey of entering keywords and bringing back results from a previously crawled subset of data. For most, it is a much more dynamic experience, combining cached and crawled data with real-time analysis to provide a more personalized, and contextual, search experience within the scope of the collaboration platform your organization uses to communicate.

And if people can find the right content, they are more likely to take action on that content, such as sharing, commenting, or linking it to other relevant content – all of which make that content more accessible to others. Organizations that put search at the forefront of their collaboration strategy will find a quicker path to end user adoption, engagement and overall platform satisfaction.