Data Silos and Systems of Engagement
Research conducted by Columbia University and others has shown that when confronted with too many choices, customers struggle to make any decisions — and end up purchasing less. Digital marketers have learned this lesson (well, the smart ones) through decades of email marketing campaigns, understanding that a crowded field of links (otherwise referred to as the Call to Action or CTA) reduces overall clicks. Many fast food restaurants have found that limiting the number of choices in their menus actually drives sales. Additionally, a limited menu can mean a more streamlined and efficient kitchen, a much less complex supply chain for products and services (fewer suppliers might mean lower costs), and the ability to run operations with fewer personnel.
Limiting Scope = Good
In my early years as a technical project manager working in the data warehouse space, I quickly learned about the impact a limited data set could have on system performance while also improving reporting speed and accuracy. The limited scope allowed users to refine and focus on specific data types and functions, and specialize the types of solutions I was able to deliver. A system or tool utilizing data that has been organized and optimized for specific queries, retrieving specific results, will perform better than a system that must query across multiple databases to find and present the relevant data set. The downside to this specialization, of course, is that the cost of expanding beyond these focused solutions can be far greater.
I spent much of my first decade in information technology working closely with internal customers to slice and dice their data so that the specific results they needed were shown quickly. However, much of my time was inevitably spent going back into the system to add additional data (geographic, biographic, or psychographic data), and to join or index various data sources to improve query performance and to leverage relevant customer and product data. And because the data was becoming more and more complex, the front-end tools used to access the data became more complex — we were constantly adding third-party and custom tools and reports to improve access and get more value out of this data.
The almost constant reworking and shaping of data and systems came at a high cost in people, hardware, and time. Historically, the larger the data set, the slower the performance of our data queries (compute and processing time). Because of this, what tends to happen is that end users become frustrated with the performance and request specialized, focused solutions for their own team or project, creating yet another data silo focused on short-term needs. And the process repeats itself.
Focusing is Hard
Human nature is to solve the problems right in front of us. More difficult is to look at our problems more holistically — to step back and think about these issues long-term, and how else this data might be used. An operations team will look at solving their own needs, regardless of the potential benefits to support or engineering. Architecting our systems (and planning our businesses using systems-based thinking) can be difficult and time-consuming, but are necessary if we are to prepare ourselves for the unknowns.
We are siloed in our thinking about business problems, not just the data. It is understandable, since one team is not typically measured on the activities or optimization of other teams. From this perspective, not all data silo issues are bad. A smaller sub-set of data, or a spot-solution, allow us to act more quickly, developing responses that solve the problems at hand. But we can’t lose sight of the broader goals while chasing after short-term solutions.
We are at a major inflection point in the history of information technology, with pressures around decreasing costs and headcount, pushing data and functions to the cloud, to enable more social capabilities, and to build out mobility solutions. The danger is not to fall into the data silo gap, and build out solutions that do not truly scale the enterprise, that do not look beyond solving the smaller problems before us without considering the long-term requirements of the business. Much of the focus of company leadership is on solving the short-term problems: lowering capital expenditures, reducing operational costs, increasing revenue, and so forth, through cloud, social, and mobile strategies. But many of these decisions are made without fully understanding the long-term costs of these decisions.
Don’t Short-Change the Long-Term Benefits
In some ways, collaboration technology is a great example of short-term thinking at work around our data silos. We are drawn to these platforms because they best mirror the ways in which we increasingly communicate in our personal lives. They allow us to connect in synchronous (real-time) and asynchronous conversations to help us collaborate and share content and ideas more than ever before. Unlike traditional intranets and other knowledge management repositories, this new slate of collaboration technology provides us with ever-expanding “systems of engagement” that extend and enrich our systems of record (where we store our content, such as our intranets). Unfortunately, the data collected through these dynamic interactions are often disconnected from our other systems of record. We are embracing this new category without fully understanding the data, process, and even cultural siloes that we are creating. Organizations, by and large, are not yet seeing the lost opportunity cost of aligning this data to other systems.
Short-term thinking around collaboration focuses on communication, when the true opportunity is in adding context to our content — in connecting the data and context created around a document or a threaded conversation out to the broader system, building a web of individual experiences that enhance the entire system.
Each interaction we have through these social collaboration capabilities adds metadata, which makes the related content more searchable, findable. For example, two documents of equal value and complementary content are added to a repository with similar tags or taxonomy applied, but one of the documents is then shared with a few peers. Some of those peers read the document and find it relevant to other project they’re working on, and so they ‘Like’ it, rate it, comment on it, tag it with keywords relevant to their projects, and share it with others. As a result, people who are unconnected to the original author or project, and who may have never searched for those original keywords, are able to consume this content. Social helps surface content and context, but only if those social activities are linked into the broader collaborative system and search architecture.
In the early days of my SharePoint technology career, I was often the only voice at a conference or SharePoint Saturday community event talking about the gap between systems of engagement and systems of record, and how social collaboration (conversational) technology could fill the gap, providing context to our content. I was happy (and frustrated, at times) to see progress made within the SharePoint platform, through the acquisition of Yammer, the launch of Microsoft Teams, and the incremental improvements to the out-of-the-box social and sharing capabilities of the Microsoft Office suite of productivity tools to where we are today. There is still work to do — and a way to go to integrate these systems of record with our systems of engagement — but I finally feel that the right steps are being made.
The long-term benefits of a “highly-engaged system of record” is that our content and data will begin to truly reflect the effort that goes into their creation, and the conversations that fill the gaps between documents, meetings, and deliverables. That’s where true innovation happens — in the spaces between the documents, meetings, and deliverables. By capturing this tacit knowledge, we can begin to identify the patterns of innovation and duplicate them.
