Measuring Sentiment and Engagement

Since last fall, I’ve been partnering with the team over at tyGraph (www.tyGraph.com) around the metrics for my monthly CollabTalk tweetjams, helping them tweak their analytics to improve upon some of the over-priced and questionable-value Twitter tools out there. They improved their measurements, which are now all that I use for my online events — and it also resulted in tyGraph signing on as an official sponsor back in August 2016 (or thereabouts). It was out of those initial conversations that made me realize how difficult it is to measure end user sentiment and engagement.

And yet engagement, in my opinion, is the #1 metric organizations should be tracking. While the platforms we use for enterprise collaboration (such as SharePoint) contain much of the data we need to identify, track and measure end user usage patterns, activity metrics do not equal engagement. Engagement, unfortunately for all of those hoping for a quick fix, is not a clearly defined (or understood) concept, and few organizations have taken the time to define it, much less figure out how to track the metrics.

Organizations track basic measurements: number of posts, comments, downloads, and number of “Likes,” but we do not take the next step of trying to understand how these statistics affect collaboration. Metrics are the key to driving end user adoption. Unfortunately, collaboration has an analytics gap. Managers who own the success of collaboration across their organizations know this intuitively, yet they rarely go beyond the out-of-the-box data points and leaderboards that come with any social collaboration solution. There is very little visibility into what people are doing, and much less data on why they are doing it.

Why Out-of-the-Box Is Not Enough

SharePoint provides many data points, capturing details about each social interaction (content access, content shared, status updates, etc), but does not provide a simple, out-of-the-box method for presenting this data to administrators. Within the Office 365 admin console for SharePoint, the default view is your storage usage per site collection. With the platform’s broad feature set, the complexity of its data, and the customizations required to access and report on this data, few organizations have taken the steps to pull this data out of the platform in a way that can help them improve key social collaboration metrics.

Yammer has the opposite problem: an easy to use and understand interface and a comparatively simple feature set, but it lacks the depth in the data from which to build these same metrics. Even when paired with SharePoint Online, which is the version of SharePoint available through the Office365 platform, Microsoft has simplified the reporting and analytics available through their tools rather than use the pairing as an opportunity to provide a federated view of social activities across Yammer and SharePoint.

The danger with not being able to measure and adequately manage the collaborative activities within the enterprise is that administrators have no way to determine which features, which teams or which collaboration initiatives have been successful. As a result, many organizations will find their end user engagement gains temporary, as users very quickly move past the novelty of anything new if it does not also bring perceived value.

Developing a Successful Engagement Strategy

A key to success in enterprise collaboration has been to align collaborative capabilities with specific business activities, and then track the changes over time. An example might be incorporating polling, threaded messaging and ratings systems common in most enterprise collaboration platforms into the product development processes, and then to measure the level of activity before and after adding these capabilities — and their continued usage over time. The benefit, of course, is that you’re giving the extended team the ability to provide input into the identification and prioritization of features in a company’s product roadmap, but you’ve now also create discrete and measurable collaboration activities that will help your organization better track team involvement (engagement) in the product design and review process. As you track this over time, you’ll be able to identify sentiment (level of interest) around specific features, or various product releases, as well as an overall trend (downward or upward) of participation.

For the CollabTalk tweetjams, the tyGraph metrics give us this measurement of sentiment. A tweetjam is a great example, because it’s a discrete event (it takes place over the course of an hour) with a MAJOR spike in activity, but allowing us to compare sentiment around the different questions asked, as well as the level of participation and influence of any single participant. If you’ve not seen the tyGraph metrics for my tweetjams, which are shared via Power BI, you definitely need to check it out: bit.ly/CollabTweets

As a category, collaboration analytics still has a lot of growing to do. While there are many platforms out there that provide some additional reporting and analytics capability for SharePoint and other solutions, the fact remains that the major platforms do not yet provide turn-key tools for measuring the impact of social in relation to the structured collaboration activities that enterprises rely on. Until these vendors, and their vast partner communities step up and close this analytics gap, organizations must build out their own solutions with the data that is available.

I’m still a big fan of the engagement metric. Of course, no metric at a single point in time is as valuable as the same metric shown over time. The value of the information is rarely in a snapshot, but in the upwards or downward trends over time. As I saw while working as an operations manager in my last role at Microsoft, when people know they’re being measured, they tend to adjust their activities toward those metrics to make themselves look good — which is all perfectly normal. Metrics and KPIs should be regularly reviewed and optimized. You’ll want to continually refine your analysis, review your data sources, and adjust your metrics to ensure you’re making the right assessments.

Enterprise collaboration continues to grow and mature. And with that maturity, vendors are stepping forward to provide tools and methods for improving information worker productivity. I’m definitely a tyGraph fanboy, but there are other vendors with great solutions that can help your organization fill the analytics gaps, such as BA Insight and CardioLog Analytics — both of which I have worked with from time to time, and will be doing more with later this year.

You cannot improve what you do not measure. So get to it.