Basic Tips and Advanced Tricks for Implementing Client-side Analytics
Most companies these days claim to be Data-driven. But how you collect, store, and analyze that data impacts your ability to use it to make decisions. And as a Product Manager, you are responsible for making sure your team can make decisions from this data.
There are two types of data: state data and clickstream data. These different data types tell you two very different things. State data is what you get out of the database and tells you the current state of things: How many tickets did you sell to the event? How many workers completed jobs on Monday? How many messages a user sent last month? These are questions that are easily answered by state date. Whereas clickstream data will tell you the order that things were done: Did the user select the t-shirt from the main page or from the checkout promo? Was the job accepted from the list view or the map view? These are the questions that are more easily answered by clickstream data. State data is stacked and stored based on your engineering team's approach. But clickstream data can be implemented in a variety of ways. It is critical that the implementation of clickstream data is done in a way to set the team up for success.
As the Product Manager the team will look to you for implementation decisions on the client-side analytics that produce clickstream data. This process must be taken seriously or else you won't be able to answer simple questions later on. For ApptheGame - the app from the first company I co-founded - I did not understand the importance of correct analytics implementation. When the app rose to #7 in the app store I reviewed the reporting to understand what was driving engagement. To my horror although we tracked every pageview there was no way to answer the simplest question. For my next app, Swoopt Daily Fantasy Sports, I invested my time in the analytics instrumentation and it paid off. When Active Users increased I isolated the engagement actions most correlated with retention. I used this data to steer development towards these features and increased retention.
The north-star of analytics is that it should let you answer simple questions. While this seems intuitive, inexperienced Product Managers underestimate the amount of clickstream data. Generally this flaw is found out after launching the product. Thus the team has to re-instrument before putting themselves in a position for success. Don't waste time - instead make sure you can answer these basic questions before launch:
How did the user get to the feature/product?
What did the user do once they got to the feature/product?
When did the user come back to the feature?
After you’ve instrumented your new feature or product to capture the basics you need to make sure that it works. The first thing to do is to see how you can intercept any kind of analytics reporting in real-time to better debug and help your team out. If you’re using Segment there’s both a handy debugger in the platform as well as a simple Chrome Extension. Since clickstream analytics use string matching it is critical that the team agrees on the data format up front. Details like snake_case vs camelCase matter. Check this on your dev environment. Check this on production right after launch. And check this 2 days after launch. I will say it again: make sure your data stacks and you'll be a happy camper.
Thus far we've discussed the foundations for success. Now let’s move onto four advanced tips and tricks for instrumenting analytics, to help you answer more advanced questions.
1. A/B tests
It is possible to use platforms like Optimizely or Launchdarkly to do the analysis for A/B tests. But what if you have a proprietary A/B testing system without these capabilities? While at Eventbrite, we actually used an internal A/B testing framework we developed called Janus. Although the enrollment and exposure functionality worked well the platform lacked robust analytics. I was charged with extending the platform to the mobile apps and found that the reporting quickly broke down. The lead Android engineer and I searched for answers. Eventually it occured to us that we could send the enrollment value back to Google Analytics as a custom property on events. To keep it simple we used a format of test_name_enrollment_value eg vertical_feed_b. This allowed for the engagement and retention analysis to be done directly in Google Analytics. This pattern saved countless hours of SQL query time later on. This answers the simple question: what was different between my test and control groups?
2. Date range components
Another trick I’ve developed over the years is the best way to track a date range component. Again, start with what questions you most likely want to answer. Since there is a date range component this means your team has likely has to set a default date range. But was the default range you picked correct? The trick is to track the difference between the original date range and the new date range selected. Track the difference in each dateTime input component with either a positive day count (into the future) or a negative day count (into the past). This answers the simple question: is my default date range correct?
3. Multi-select components
Tracking the dreaded multi-select component. At Wonolo, we use Segment and Amplitude for client-side analytics and it works quite well. But there is one component that took us several tries to instrument so that we could answer simple questions. The multi-select component. On the surface, this seems simple: we want to understand which options the user selects. But since there are multiple options, what is the best way to store and report on this data? There are three points to consider for success: 1) although interesting, order does not matter. 2) send the multi-selected inputs in the same, standardized order. 3) send the properties in string_list format instead of an array. Point 3 is important: platforms like Amplitude and Mixpanel don’t handle data arrays the same way you want to visualize them. But follow these three points and you'll be able to answer the simple question: what are the most common values selected?
4. Loader and spinners
The last trick to share is that you can use Segment or Amplitude to track and visualize render times. Since clickstream events are based on what happens in the client these platforms can determine that an item has rendered on the page. Round to the nearest 100 milliseconds so that your stacked data can convey patterns. This answers the simple question: how long does my user wait until they see what they requested?
Follow these basic tips and you'll put your team in a position to answer simple questions. Follow the advanced tricks if you come across these use-cases. But whatever you do, invest time in instrumenting client-side analytics. Your team and end-users will thank you for it.