Measuring ROI and Success with Analytics

Our 11th Analytics class was about ‘how’ to measure an online strategy’s return on investment (ROI) — which sounds much simpler than it often is.

Some of the many challenges are:

• Assessing SEO  – It’s hard to estimate the SEO’s ROI because you can’t estimate the full impact of the long tail. A workaround is to multiply the search volume for top key words by 3.3 (or 30% of the possible clicks) but this is just an estimate.
• Defining Social Media ROI – You can’t always definitively tie a social conversion to a specific financial metric.  Subsequently, you need to assign a financial value for each social engagement metric by testing and validating social activity. For example, if you generate a $1,000 sale for every 10,000 Twitter re-tweets (RT), each RT is worth $10
• Attribution Analysis Quandary – There is no definitive answer on which brand/campaign touchpoint ‘wins a visitor over’ and prompts them to complete the desired action (e.g. purchase, sign-up or send an email inquiry). It may be the fourth visit to the website or seeing the item in the store after after reading about it once on the website. As an alternative, Sofia outlined a blend of Media Mix Modeling and Marginal Attribution Analysis. Specifically, this means measuring your baseline, allocating part of the budget to one marketing channel, run tests with/without it and track impact until the value declines. Repeat for each channel until you find the most cost-effective combination with the highest returns.
• Offline Impacts – Offline promotions and external events (e.g. new competition, scandal, market crash) can skew results. You need to take extra steps to track offline promotions, such as setting-up vanity URLs to track coupons, and find rationale for traffic patterns.
• App ROI – Apps aren’t cheap to make or run. To measure their ROI, you need to consider the Total Cost of Ownership (TCO), including development and ongoing operation costs, which can fluctuate over time.

So while measurement is a finite science, measuring online ROI is part art because it requires some subjective decisions.

Sofia provided comprehensive best practices and a number of formulas, which I think will give me long-term value — or ROI on the cost of her college course. However each campaign and scenario is different and there is no ‘cookie cutter’ solution.

What does this mean to me?

For example, the business model for my senior project generates revenue by selling online advertising. If I need 10,000 page views per month to sell a $1,000 ad but I currently only get 9,000 views per month, I might use a keyword upgrade to drive traffic to my site. The resulting formulas might look like:

For SEO Revenue:

18,000 (people searching for my keywords) X 10% (average success rate, where I define success as clickthroughs with page views) = 1,800 new page views

$1,000/1,800 = $0.50 value per page view

For ROI:

$1,000 (July SEO Ad Revenue)/$200 (Contractor to identify keywords and incorporate them into site) = ROI of 5

Illustration Source: Lisa Solonynko /Haml via Morgue File.

Analytics Success Lies in Digesting Related Segments

Our 10th analytics class focused on how to best analyze and understand data.

The top-level message I came away with is a data metric shouldn’t be evaluated in isolation, which is meaningless.  For example, knowing only that users spent an average of five minutes on a page offers no insight to validate or adjust your strategy. It may mean only users from a paid referral on one day of the year spent this much time on the site and on other days or referrals from organic/social/direct bounced after 10 seconds.  This would call for course corrections to improve results but without looking at the ‘big picture,’ these issues might go undetected.

Specifically, you should assess data metrics:

• In context with the organization’s goals, what competitors are doing, industry information, internal initiatives, external events/trends…
• Ideally in related segments, such as: referral sources (paid/social/organic/direct), days of the week (weekends and weekdays), times of the day or platforms used to access a site.

Sofia cautioned against comparing unrelated data metrics, such as tablet use with social referrals or creating compound or ‘super’ combinations like Alexa page rank with inbound links.

I found it interesting to note that higher numeric data (e.g. 10,000 contact page exits on every 100,00 visits), might deliver exactly the same statistical significance as smaller data (e.g. 1,000 contact page exits out of 10,000 visits) but is more effective when expressed as a clear outcome, such as 10%, versus a ‘muddy’ ratio like 10,000:100,000 or 1,000:10,000. However, I think higher numbers do strengthen a metric’s value.

For the class exercise, we re-examined the metrics set for our senior project, identified segments to be assessed and ways they might be visually depicted.

For my project, here are some KPI segmentation and visualization options:

• Percentage of conversions by each referral source, such as social (fb, Twitter, Reddit), organic, paid and direct in a specific month, compared to non-conversions by referral sources in the same month.  (Visual depiction – 2 pie charts or a bar graph)
• Specific Senior Care Share modules engaged in a specific month by referral sources. (Visual depiction – segmented bar chart)
• Page views for a specific month, segmented by weekdays versus weekends. (Visual depiction – segmented bar chart, pie chart or even infographic with other metrics)
• Page views in a specific month, segmented by referral sources. (Visual depiction – segmented bar chart)

However, as with all analysis, you should:

• Find out what’s happening across the organization, such as other initiatives; business changes or help desk calls.
• Consider external events that might impact data, such as holidays, market trends or even weather/power outages.
• Possibly do surveys, click density analysis or other research….

….to get the full picture and extract maximum value from your data.

Illustration Source: Haml via Morgue File.

Telling the Visual Story of Data

In our ninth analytics class, we focused on data visualization, which is used to visually depict a project’s results or analytics.  Seeing a visual representation of data helps people to understand its meaning. It also makes the information more memorable as it helps to tell a story.

Data visualization is particularly effective for conveying a project’s ongoing results, particularly KPIs, to stakeholders and showing them where they need to pay attention or course correct.

There is a huge variety of data visualization formats you can use to tell the visual story of data — from conventional bar and line graphs to golden ratio depleting charts and intricate infographics.  Fortunately, there are a range of tools available for creating them. We took some time to experiment with Tableau. I think this is a really versatile tool but it comes with a steep learning curve.

I think it would be really useful for showing KPIs for my senior project, such as conversion rates from various social and organic referrals, possibly in a pie chart, to show which are most effective.  I might also use a bar graphic to compare the time users spent on each of the niche social media network’s pages.

However, in the interim, I wanted to try this tool on existing data. Last summer, I managed a youth shelter’s eight week fundraising campaign that was heavily promoted on social media. I used a tool called ‘SumAll’ to track Twitter results for this campaign and exported them into an excel sheet. I imported this into Tableau and experimented with the tool.

One challenge was getting the weeks to display from oldest to latest. I re-labelled the weeks from a ‘week of Mon 08 Sept’ format to ‘week 8 (08 Sept)’ format in Excel to make the sequencing clearer. Latterly, I also found that in some views, you can click on the Dimension value to reverse the order.

There is much more formatting I need to do but, the above bar graphic is a start at showing the stats from this campaign. In this example, I used a side-by-side bar chart format to show the growing momentum of three measures (tweets, mentions reach and retweets reach) over the last six weeks of the campaign leading up to a September 6 event date. I filtered out the first two weeks, as the Mentions and Retweet reaches were minimal during this early part of the campaign.

A graph like this could be used to demonstrate to a stakeholder the value of a campaign of several weeks, versus expecting comparable results with a one or two week period (which some might think adequate).

I need to learn much more — but can’t even begin to visually imagine the data I must absorb to proficiently use these tools.

Why Catch Key Words and the Long Tail?

Our eighth Analytics class focused on Search Engine Optimization (SEO), which determines how prominent (or where) your website site will be listed in unpaid/organic/natural search results via Google or other online search tools.

A key part of attaining optimal SEO for your site is understanding your target audience’s search behaviours.

We learned that people usually access websites to solve one of the following types of queries:

• Navigational – to find a specific URL
• Informational – to find any information from an address to an in-depth research study.
• Transactional – to complete a task, usually a purchase

A key part of mastering SEO seems to be identifying the keywords people use to find your site. Two main types for keywords are: one word to short terms (2 – 5 words) called Popular/Lead keywords; and longer, detailed phrases, called Long-tail keywords.  You need both for success.

As expected, transactional searches are the highest value because they usually generate revenue but I found it interesting that these are also the more detailed, long-tail words or phrases.

There are also other psuedo keywords, such as ‘Trophy Term-Keywords,’ which may drive traffic to your website but this traffic doesn’t convert or behave how you want it to, likely because it’s not a good fit between your audience and your site. For example, a term like ‘call to arms’ might drive military oriented people to a site but they will quickly leave if it’s about prosthetic limbs.

Sofia provided a great list of links and tips for maximizing SEO. Even pointers like setting URLs in the country of your target audience and putting keywords in JPG titles helps.

To compile a list for my senior project, I used Google’s keyword generator tool for my site, then searched my competitors’ sites and ran the generator for their sites.  I found many popular terms, like: senior care, help for caregivers and help for elderly.  I also found the following particularly useful long-tail options:

• practical tips from real caregivers
• outside resources
• here to share advice from the community
• online community for family & professional #caregivers
• Where Caregivers Survive & Thrive
• Caregiver’s Survival Network
• family caregivers taking care of a loved one

As my project progresses and the content is clearer, I will re-do and refine my searches to find more.

(Photo: Taken by Gabor from Hungary via Morguefile) 

Adding and Measuring the Impact of Mobile Apps

For week 7 of analytics, we focused on how to measure the impact of an app — be it for entertainment, practical or promotion purposes.

Given the development work behind each app, I think it’s particularly important to measure their analytics so that you can course correct, as required, to maximize the investment. This means using app analytics to learn: who downloads the app,  how they discover it  and how frequently they use it (if at all).

Although app analytics measures many of the same metrics as web analytics, like number of visits and sources, I thought it was interesting to note some of the subtle differences between the two.  For example, I noticed that app analytics metrics include:

• Screen resolutions (as well as screen size and orientation)
• Varied types of mobile devices used
• A stronger focus on groups of users — including user’s friends who also use the app and ‘cohorts’ (i.e., segments of customers that downloaded the app during a common time period).
• “Usage” instances and frequency, since downloads/installs don’t correlate with engagement/use.

Like web analytics, you need to define metrics and KPIs. For example, you need to define an ‘active’ user’s level of engagement (frequency, duration).

We also looked at various tools for measuring app analytics. However, Sofia pointed out that the metrics they measure are not evenly distributed among the categories of apps available. Specifically, current tools are more focused on marketing metrics, like search engine keywords used to discover them, versus content metrics, such as the app’s longevity/life cycle and number of app/widget installs a user has.

Of the tools we looked at, I particularly liked Flurry Analytics because it provides benchmark measures and several distinct metrics, such as user personas. However, it didn’t look like it provided search engine keywords, which you needed to measure through other tools, such as App Annie, as its Key word ranks feature.

For the exercise, I selected a made-up ‘package ingredient analysis app’ promoted through an Instagram campaign. I identified KPIs and measuring tools as:

• „Number of active users (Flurry- Event- Use)
• „Average increase in users (over 2 week time periods) – (Flurry – Funnel Analysis)
• Number of people using the app at 2x per week or more (Flurry – Retention – Return Rate & Event – Use)

I identified metrics and their tools as:

• „Search engine words used to find app (App Annie – Key word ranks)
• „Number of installs (Flurry – Funnel –Downloads)
• „Number of users who have installed the app vs. Instagram users (app installs and Instagram audience)
• „Personas using app (Flurry – Audience Interest – Personas)
• „Use of this App compared to other lifestyle apps (Flurry – Audience Interest – My Apps Interest)

We also explored and discovered several other worthwhile app analytics tools, including:

• Amplitude – Focuses on user engagement, including: segmentation, cohorts and churn.
• AppSee – Indepth analysis of users and everything they do with app (for a steep price)
• Ninja Metrics – Focuses on customers and is well suited to game apps
• Localytics – Unique tool that integrates analytics with marketing to help produce push campaigns

This class opened a new world of analytics — and there may be an app for that too.

Linking to Success With Social Media

For week 5 of analytics, we dug deeper into social media with a focus on the value of Google+.  Although Google+ may seem less intuitive to use, you need to evaluate its use in terms of your goals. Specifically, if you want ROI or engagement, Google+ may be well worth the time it takes to set up because it controls ‘who’ sees what you share by controlling how high your organization comes up in Google searches.  Sofia showed us a couple of examples from her own business, when she posted on Google+ and how it boosted the company’s online prominence.  I thought this was effective and drove the point home.  With my senior project, I think it may be worthwhile looking into Google+ ‘communities’ that pertain to senior care but also I think it will be important to have a presence on  Google+ so that might key words and organization will be found.

We also learned than ‘not all links are created equal.’  A key way to optimize search engine results to your site (SEO) is by having links from other website’s connect to yours. However, we learned the strength of these links varies according to various criteria, such as:

• Authority or credibility of each site linking to you. (It’s better to have links from established, high profile sites like Buzzfeed or the Huffington Post than a blog run by one person with a few subscribers)
• Number of other sites each site links to. (If it links to a high number of other sites, its link is of less value.)
• Where your link appears (The Jackpot to have your link in the title or anchor text but this is rare.)
• Relevance of the sites linking to you. (If you pay for links from an essentially ‘content-free’ site or link farm, there is little value)
• Authority and Page Rank of the linking page

I also learned the word ‘Social Signals,’ which is when someone likes, recommends or shares your post.  So with my senior project, I ideally want many ‘high value’ inbound links, ideally from relevant healthcare/senior care sites and to generate many ‘social signals.’

Drilling Down to Social Strategies

Our fourth Analytics class focused on the first of two parts on analytics for social media. This first part highlighted facebook, twitter and LinkedIn in terms of what users can do on these sites and “how” you can use these platforms to measure online, as well as offline, initiatives.

As with all analytics, you need to identify your SMART goals, KPIs, metrics and measurement methods in advance to ensure you capture all the relevant data and don’t waste time or lose important stats.

I think it’s interesting to note that if you are running a campaign that includes an offline tactic (e.g. a coupon or redeemable voucher), you can measure its impact online — IF you plan in advance.  The reverse also applies. For example, you can measure the offline response of a Facebook ad by using “offer claims.”

For this week’s in class exercise, we had to develop a social media strategy for promoting Parks Canada’s “Unplugged” campaign. For this, we had to identify SMART Goals and tactics for achieving them, as well as KPIs and metrics to measure progress.  I found this exercise a little confusing because we had just learned that traditionally KPIs are metrics that relate directly to your goals and generally impact the business.  Specifically, KPIs traditionally apply to revenue-related metrics (e.g. cost per lead, return on invest) or direct conversions that impact the business (new memberships).  However, I learned that the KPIs for a social media campaign are an exception to this. That is, they are social, as are the SMART Goals that support the campaign (e.g. generating X number of tweets with a specific hashtag).  This is good to note, as ‘when’ I launch my senior project, I will likely do it with a social campaign to drive traffic to the site and ideally generate conversions (member sign-ups).

In the exercise, we had to create a strategy for Twitter, Facebook and LinkedIn. The first two were straightforward, as the campaign was consumer-facing, which both Facebook and Twitter can be.  However, LinkedIn forced us to think ‘outside the box’ as it’s more of a B2B channel.  This was a challenge but good because sometimes clients decide they want to use a specific social media channel and you need to find a way to make it relevant to your marketing needs.

Many Web Analytics Metrics and Ways to Measure

Our third Analytics class focused on the various types of website traffic data points and ways to measure them, both qualitatively and quantitatively.

I learned that you can measure a vast quantity of web data points or metrics. I think it helps to try to remember them by categories, which include: conversions (rate, by source..), traffic ‘referral’ sources, geographic location, visits, visitors (new, return, unique…), time on page, time on site, bounce rate, exit rate, engagement and purchase habits (cart abandonment, days/visits to purchase…).

It was also worthwhile learning that analytics aren’t infallible and that loopholes exist. For example, a visitor may spend five minutes on a page but if they don’t go to a new page, their visit is tracked as a ‘bounce,’ which is inaccurately perceived as negative.

I think a good point to remember from the class is also that the impact or positive/negative attribute of a metric varies according to the type of site and its purpose.  For example, for a service business site, like Sofia’s, you want first time visitors who scan the site, exit on the contact page and follow through to enquire about purchasing a service.  Repeat visitors who never follow through offer negligible value.  In contrast, for a social network (like the like the caregivers one I’m proposing for my senior project), repeat visitors are critical for the site’s longevity; you also want them to come to the site, ‘engage’ (by posting or responding to a post) and leave, most likely without going to the contact page. Ideally, you want repeat visits from those who’ve signed on as members (i.e., converted) but even those who check the site a few times before committing are ok, particularly as the site evolves.

As for ways to measure web analytics, we learned about various methods. It sounds like one of the key ways to quantitatively measure analytics is using JavaScript tags. I’ve used this method to measure another wordpress.com blog by embedding the tag via a CloudFare workaround. Unfortunately, I haven’t blogged much since I set this up in Oct. 2013. However, my page views for Nov. 2013 were 1.47, which doesn’t sound as bad when I consider that page views are not as relevant on a blog because people usually visit the most recent post and bypass the archives. We also learned about various qualitative measurement methods, including heuristic evaluations, site visits, usability testing, surveys, web enabled research, experience testing and collecting competitive intelligence data (e.g. panel based measurement, similar to AC Nielsen for  TV).

The we did an in class exercise to clarify our understanding and from this I learned more about methods.  My takeaways from this assignment, (which I ended up presenting) were:

• When trying to determine a website’s goal, always search for ‘how’ it is making revenue (unless it’s clearly run by a not-for-profit or clearly funded by another source).
• Be very specific when citing KPIs or metrics. For example, don’t just say “measure return on investment” but clarify ‘what’ you are measuring (e.g. orders), against what (e.g. operational costs) and how frequently (e.g. weekly).
• Heuristic Evaluations, such as Site Visits and Remote Testing, do not usually have a sufficient number of participants to be statistically significant, even if you are testing a metric that can be quantified.

KPIs and Metrics: The Signposts and Footsteps to Success

In our second Analytics class, Sofia introduced the value of Key Performance Indicators (KPIs) and Metrics to a project’s success.

My understanding from the class is that KPIs are like signposts. That is, they are: important measures that show how your project is progressing toward achieving its SMART goals with business implications and can inform course corrective actions, where required. We also learned that KPIs are specific to the project and as such, the number of KPIs varies but should be  enough ‘to do the job.’ This is a good way to assess if you have enough KPIs but still challenging, as I think it’s easy to be over zealous when you’re just starting.

An important KPI criteria that stood out to me is that they should deliver useful information in a timely manner, ideally in less than two months, unless you’re in a slower industry. For example, if you have an established, broad appeal, social network that promotes and sells premium services, a KPI might be your cost per order/sale and can be attained within a month.  In contrast, if you have a new social network in a niche market, (e.g. the social network for caregivers of senior citizen family member I’m considering for my senior project), it may take two to three months to attain this KPI.

Depending on the KPIs selected, I thought it noteworthy that part of the task may include defining key components, such as valuable exits or successful events, tailored to your project. For each example, you need to define ‘what’  a user needs to do before they leave the site, to assess whether the visit exit is valuable or the event successful.

Since KPIs have a major impact on the business, such as impacting revenue, costs or conversions, it makes sense to reference them when reporting up to internal executives or clients. To this end, Sofia discussed the importance of segmenting your selected KPIs into custom reports, possibly separate ones, for reporting to executives, as well as project team members.

I also found it interesting to learn about KPI’s various categories, including: Actionable Outcome KPIs; Calculated KPIs; Engagement KPIs; and Business KPIs. There are also Social Media KPIs and Conversion KPIs but many of these measures are ‘Metrics’ and not KPIs. Metrics, Sofia explained, are timely qualitative or quantitative data points that help inform your strategy but don’t directly impact the business. They remind me of footprints.

We discussed social media measures, such as the number of people accessing a site via a social network, and while important, they’re rarely KPIs. The reason is visits via social links just offer more ‘opportunity’ for people to consider a service/product/offer.  This opportunity may prompt some to sign-up or make a purchase. Since the second action directly impacts the business, its associated measures (e.g. orders per social acquisitions) would be the KPI. The exception might be a campaign where a key goal is to have an advertiser’s hashtag mentioned 100 times a week. In this case, hashtag mentions might be a KPI.

If I pursue a social network for seniors’ caregivers as my senior project, one KPI might be: engagements that include a successful event, defined as ‘joining’ the network,’ compared to overall site visits. Metrics could include specific pages visited and abandonments during registration. Sofia also suggested you should move from macro to micro insights to figure out why users are behaving a specific way online. In following this, these metrics for my project could be used to learn which pages are compelling or need improvement and how well the registration process is working.

The Heart of Analytic Success: SMART Goals and the Trinity Strategy

For our first analytics class, Sofia provided an overview of what analytics is, its value and how every successful project must start with SMART goals. Sofia defined analytics as the use of data to gain insights and make better decisions. I agree with this but would add that analytics helps you report up to decision-makers on how investment in your initiative has impacted behaviour and user experience to advance business goals, as reflected in outcomes. Analytics also helps you gain executive (or client) buy-in for future projects that build on a prior campaign’s analytics or results.

I think analytics is becoming increasingly important as we continuously have more volume and variety of information or ‘big data.’ Sofia defined big data as “a collection of data from traditional and digital sources inside and outside a company,” which companies are increasingly looking at for ongoing discovery and analysis to inform their decisions. This data includes many things we can measure and analyze from digital sources, such as how many people access a website through specific social media platforms and from what countries, as well as traditional sources, such as how much money an event raises or how many people attend it. What I found particularly new was learning the difference between ‘structured’ data, which is quantitative, such as how many visitors access your site via Twitter, and ‘unstructured’ data, which is more qualitative, such as comments posted on your company’s facebook page.

Before you can attain analytics, you need to set goals for a website or digital property. Sofia explained that you summarize each goal in a sentence that includes Specific, Measurable, Attainable, Realistic and Time-bound (SMART) attributes. This is a slight variation from PR campaigns (which I’m familiar with), where goals are broad and objectives have SMART attributes. In PR, I’m also more used to setting goals and objectives for campaigns with a definitive end point.  In this first class, we did an exercise to identify a website’s goals. The feedback from this exercise gave me the impression that SMART goals can be for an overall website or company (such as the Toronto Star) versus a set campaign. If that’s the case: how can you make these goals time-bound?  I think you might set them for an initial period goal, such as six or three months, and then reassess but it would be good to know for certain.

Sofia told us the grandfather of analytics is Avinash Kaushik, who discusses his theories extensively in his blog: Occam’s Razor.  The blog is named after a 14th-century English logician and one of the alternate translations of his principle is: plurality should not be posited without necessity.  I think this suits analytics because it provides a methodology for specifically identifying what big data an organization needs to measure and why (necessity), versus trying to measure all the data it can access (plurality).

We also learned about Kaushik’s Trinity Strategy. This is a strategic approach to extract insights and metrics from a website/other platform, based on the users’ behaviour and experience, as well as the overall outcomes.  It’s imperative that these insights and metrics can be ‘acted on,’ that is used to make decisions that alter the organization’s approach or to design future initiatives.

For example, let’s say you implement a promotional campaign to sell featured books highlighted on a page in your ecommerce site. You also promote it through direct mail flyers and social media. Your goal may be to sell 50 copies of each featured book within one month. The overall process begins with the clickstream data, which is the data collected through the site. I can include: who is accessing the book page, from where, via what devices or user path, etc. From this data, you can measure and assess users’ behaviour in response to the promotion, such as:

• How many came to the site via each social media platform?
• How many were driven there by the flyer?
• Which book image or caption attracted users when they first landed on the page or made no impact for the duration (as measured by a heat map)

You then measure outcomes, such how much revenue was generated through specific online book sales. The third element is the experience, which tells why the users behaved the way they did. My understanding is this data is often accessed through additional steps, such user testing to measure effectiveness of user paths, experimenting  with site changes (e.g. trying book purchasing buttons in different positions) and customer/user surveys to assess how they feel about the site. These three elements, combined with competitive intelligence, help you uncover insights about what is attracting users to the site and getting them to buy the books, as well as what changes might improve outcomes. For example, in assessing this data, you may want to adjust the page layout, if users consistently miss a specific book displayed.