User Engagement is a Depth, not a Breadth, metric (and # UV's / # Facebook App users are crappy UE metrics)

I don't have all my thoughts on this organized very well yet, but regardless of whether i have the stats to back it up yet, i'd like to make a rather sweeping statement about measuring user engagement :

User Engagement is a depth, not a breadth, metric.

what do i mean by that?  i mean that to truly understand if someone is using your product / website / facebook app / feature, you probably want to measure adoption based on more than just initial exposure.  that means  >1 page view OR >1 click OR >10 seconds per session, and >1 visit per 30-90 day period.  forced to generalize, let's say "active user" is indicated by session behavior >=3 page views, >=3 clicks, >=30 seconds, and repeat visit behavior >=3 visits per month (note: your mileage may vary). anything less might not indicate true user activation; it could just mean someone was just being a tire kicker.

to use the disease analogy: if 1M people catch cold, but everyone "recovers" in 2-3 days and no one dies / has to stay home from work... you probably don't have the Black Death on your hands.  similarly, if lots of people walk into your store and look around, but don't stay more than 2 minutes and don't buy anything, i wouldn't go crowing about all the foot traffic you generated.  your conversion still SUCKS.

now true, if you get 1M people to catch *ANYTHING* -- like say, a Facebook App -- and you can tweak the cold into a REAL virus, you might have something.  similarly, if you use all the foot traffic to your door (or website), and give away a door prize or taste test (or landing page), things might get more exciting.

still, as a product manager / web developer / product marketer, it's a lot more relevant to your job to note depth of adoption or conversion % to a more significant metric that helps you decide a) what features to build or modify, or b) what marketing channels to emphasize / buy traffic from.  ultimately, isn't that what you should be using those metrics for, really?  (ok, so i'm ignoring metrics for advertising  or fundraising purposes... but i hate those use cases anyway. and i'll leave monetization to a later discussion, once we've determined you actually have usage adoption first).  more on this after we come up with an appropriate Lab Experiment.

---

so here are 3 rough observations i'll make about user engagement metrics:

OBSERVATION #1: top-line # Unique Visitors (for websites) or # "Active Users" (for Facebook Apps) are crappy measures of user engagement.  while they may indicate some level of initial usage (= acquisition) and perhaps some level of virality (= referral), they are not very good metrics for measuring depth of interest (= activation) or repeat usage (= retention), and can lead to incorrect conclusions for product managers or marketers about how active and valuable a user base you really have.

OBSERVATION #2: # page views is probably a better user engagement metric but still kinda sucks; due to Ajax, other rich media / applets, and also the use of video, these metrics are also not the best ways to determine user engagement. that's why even ComScore is moving away from this metric. if measured accurately, time-on-site is probably a better way to determine engagement, altho it may still leave the question of whether the user was doing something else (on the computer or off), or whether they were truly watching/thinking about the page in question.

OBSERVATION #3: active clickstream time within a session (= time-on-site combined with a "minimum time between clicks" (MTBC?) =60-180 seconds or something like that), and repeat sessions over time (= multiple visits per week / month) are likely best-case broad-use user engagement metrics.  however, for most product & marketing decisions, you'll probably still want to use more custom metrics relevant to your business & customers, such as conversion % from initial states of user activity to deeper states of user activity. 

---

like i said, this still feels like too sketchy definitions for me to be comfortable with, so i'll keep working on some better ways to describe this that can be applied practically. in the meantime, i've provided an example of a sample conversion dashboard report along with some pretty pictures in my previous post: Startup Metrics for Pirates (AARRR!). 

there's also a rough powerpoint below here (updated w/ long version):

other links / sites / reports (please add notables i've missed in comments)

• Appsaholic on Facebook
• Appaholic.com on the web / described on Inside Facebook
• Jeremy Liew's reecent blog posts on user engagement
• Jeremiah Owyangs blog posts on social media measurement
• Jeremiah's interview of Eric Petersen on attention & engagement
• Compete.com's Attention Metric
• Comscore new "Visits" metric
• GrokDotCom on Web 2.0 Metrics / Persuasion Metrics

Update: VentureBeat has an interesting piece up on how Slide, RockYou, & HotOrNot are converting Facebook app usage into traffic back to their own sites... again, the top-line #'s are decent indicators that something is going on, however not clear how those users are converting to deeper engagement. 

Update2: Jeremy Liew also has an interesting post on VB about Behavioral Targeting.

Comments

[Balaji Sowmyanarayanan]

Hey Dave,
I was able to appreciate your Gnomedex presentation better through this blog post. [ Thanks to Choppy ustream and other factors]

The techniques you listed can be put to use to empower the users too.
Attention being the emerging new currency, in the not-so-distant future, users will start demanding weekly/monthly 'Attention Account' statement from each of the services they are using. That way, they can direct their attention more effectively.
Empowering the users(through transparency) and let the wisdom of crowd dynamics takeover might be good direction to pursue.

Will refine my thoughts and post a followup if something interesting/concrete emerges.

Cheers.
-Balaji S.
Chennai.

[Damon Billian]

Hi Dave,

As always, interesting stuff. Some quick thoughts...

1. Time on site is probably a good indication of engagement (as you state). If anything, the Facebook apps should lead to increased engagement.

2. Product managers need to be aware of what customers are doing on the site. If you require email confirmation, how many actually confirm? How many folks actually click on a link for a specific product? How many click on an FAQ for a particular product?

3. Initial exposure: Awesome thoughts! What percentage of these initial exposures=an increase in adoption? I would also try to find out why people aren't adopting a product after this exposure.

[Peter C]

Great blog entry, Dave. I'd love for you to segue into that revenue per user, that is indicated at the bottom of the graph, for Facebook. I'm not talking about exact numbers but maybe some generalities around monetizing the user (can one ALWAYS monetize the user and is Facebook just not doing it right (or unleashing the tiger)?). Cheers.

[Dawud Miracle]

Interesting...I was just talking to a friend about this very thing yesterday. I'm sending him the link now. Thanks.

[Jeremiah Owyang]

Dave

I love your product marketing mind! What you've done above is a great example of a 'web marketing funnel'. Or, if you're a User Experience professional, you'd call this a use case, or user flow.

To me, user engagement can happen at any of those levels, and is a combination of interaction, interest, and attention.

The ending factorial results in what I call "Apparent Interest". Meaning, interest that we can measure.

Your graphs suggest that users need to click to be engaged. I don't think that's accurate, as users could be highly engaged by just reading.

I have a white paper coming out that I co-authored with Dow Jones on social media measurement, I'll point it your way when it goes live.