I don't have all my thoughts on this organized very well yet, but regardless of whether i have the stats to back it up yet, i'd like to make a rather sweeping statement about measuring user engagement :
User Engagement is a depth, not a breadth, metric.
what do i mean by that? i mean that to truly understand if someone is using your product / website / facebook app / feature, you probably want to measure adoption based on more than just initial exposure. that means >1 page view OR >1 click OR >10 seconds per session, and >1 visit per 30-90 day period. forced to generalize, let's say "active user" is indicated by session behavior >=3 page views, >=3 clicks, >=30 seconds, and repeat visit behavior >=3 visits per month (note: your mileage may vary). anything less might not indicate true user activation; it could just mean someone was just being a tire kicker.
to use the disease analogy: if 1M people catch cold, but everyone "recovers" in 2-3 days and no one dies / has to stay home from work... you probably don't have the Black Death on your hands. similarly, if lots of people walk into your store and look around, but don't stay more than 2 minutes and don't buy anything, i wouldn't go crowing about all the foot traffic you generated. your conversion still SUCKS.
now true, if you get 1M people to catch *ANYTHING* -- like say, a Facebook App -- and you can tweak the cold into a REAL virus, you might have something. similarly, if you use all the foot traffic to your door (or website), and give away a door prize or taste test (or landing page), things might get more exciting.
still, as a product manager / web developer / product marketer, it's a lot more relevant to your job to note depth of adoption or conversion % to a more significant metric that helps you decide a) what features to build or modify, or b) what marketing channels to emphasize / buy traffic from. ultimately, isn't that what you should be using those metrics for, really? (ok, so i'm ignoring metrics for advertising or fundraising purposes... but i hate those use cases anyway. and i'll leave monetization to a later discussion, once we've determined you actually have usage adoption first). more on this after we come up with an appropriate Lab Experiment.
so here are 3 rough observations i'll make about user engagement metrics:
OBSERVATION #1: top-line # Unique Visitors (for websites) or # "Active Users" (for Facebook Apps) are crappy measures of user engagement. while they may indicate some level of initial usage (= acquisition) and perhaps some level of virality (= referral), they are not very good metrics for measuring depth of interest (= activation) or repeat usage (= retention), and can lead to incorrect conclusions for product managers or marketers about how active and valuable a user base you really have.
OBSERVATION #2: # page views is probably a better user engagement metric but still kinda sucks; due to Ajax, other rich media / applets, and also the use of video, these metrics are also not the best ways to determine user engagement. that's why even ComScore is moving away from this metric. if measured accurately, time-on-site is probably a better way to determine engagement, altho it may still leave the question of whether the user was doing something else (on the computer or off), or whether they were truly watching/thinking about the page in question.
OBSERVATION #3: active clickstream time within a session (= time-on-site combined with a "minimum time between clicks" (MTBC?) =60-180 seconds or something like that), and repeat sessions over time (= multiple visits per week / month) are likely best-case broad-use user engagement metrics. however, for most product & marketing decisions, you'll probably still want to use more custom metrics relevant to your business & customers, such as conversion % from initial states of user activity to deeper states of user activity.
like i said, this still feels like too sketchy definitions for me to be comfortable with, so i'll keep working on some better ways to describe this that can be applied practically. in the meantime, i've provided an example of a sample conversion dashboard report along with some pretty pictures in my previous post: Startup Metrics for Pirates (AARRR!).
there's also a rough powerpoint below here (updated w/ long version):
other links / sites / reports (please add notables i've missed in comments)
- Appsaholic on Facebook
- Appaholic.com on the web / described on Inside Facebook
- Jeremy Liew's reecent blog posts on user engagement
- Jeremiah Owyangs blog posts on social media measurement
- Jeremiah's interview of Eric Petersen on attention & engagement
- Compete.com's Attention Metric
- Comscore new "Visits" metric
- GrokDotCom on Web 2.0 Metrics / Persuasion Metrics
Update: VentureBeat has an interesting piece up on how Slide, RockYou, & HotOrNot are converting Facebook app usage into traffic back to their own sites... again, the top-line #'s are decent indicators that something is going on, however not clear how those users are converting to deeper engagement.
Update2: Jeremy Liew also has an interesting post on VB about Behavioral Targeting.