About a year ago, I was selected as Tech Director for Philly Startup Leaders, the largest startup entrepreneurs community in the Philadelphia region. Our first project was to build a small tool that would help us figure out how well we’re engaging the PSL membership. PSL provides two core functions for the startup community, an extremely resourceful mailing list connecting over 700 local startup entrepreneurs as well as an annual series of highly-respected educational and networking events.
We hypothesized that of the 500+ members (at the time), only a smallish handful were really engaged within the community, and we wanted to find better ways to engage those on the “fringes,” i.e., those members who neither participate on the mailing list nor come to any events. We want to know, how can we help more of our members in a more meaningful way?
First things first, like any good nerds (whether engineering, business, or otherwise), we set out to better understand the problem and to test our hypotheses. We wanted to quantify the activity on the mailing list, gauging it with analytics and statistics to determine mailing list engagement. The end result: the Engagement Dashboard! (We’ve open sourced the software we built to create this dashboard.)
As you can see in the beautiful screen shot below, we’ve provided some interesting (we think) graphs and stats about the interaction on our list. We looked at engagement at three levels: per member, message frequency, and conversation (thread) length**.
These can be easily read after a moment; for example, about 260 threads of size 1 (no replies) have been sent since February, or 75 threads of size 2 (one reply), maybe 12 threads of size 5, and it continues to decrease. Similarly, almost 300 people have sent exactly one message, about 75 have sent two messages, and so forth. (The little negative blip at N people sent 0 total messages is because this graph doesn’t take into account the list size, and thus doesn’t have the number of people who didn’t send a message; just ignore that.)
The more interesting thing in this screenshot is that we more-or-less disproved our hypothesis; according to the stats shown above, 20% of the list sent at least one message (new or reply) in February 2010, up to 26% in March (which is pretty damn good, I think. I suspect tax season questions drove much of this), and then steadily declined as it warmed up outside til 15% in June. (We haven’t crunched new numbers since last June, but I thought I’d share the basic idea anyway).
Who cares? How do we use this? There’s a lot of things that can be learned. By looking at the number of people who only send new messages but never reply, we can see how many folks are sharing but not really discussing. By looking at the number of original (non-reply) messages versus replies, we can tell how much actual conversation is going on on the list (as opposed to “check this out” or “here’s a job posting”, etc). By looking at the total number of messages in a thread, it’ll give us some rough estimate of the quality of the conversation (generally, the better the convo, the more replies in it). By looking at the number of people who are only sending one message, versus the number who are sending five or more, we can see how many people are regularly active within the community versus those who may listen frequently but only participate on occasion. And there are other tons of other ways we can look at this information or other data we can extract, we just haven’t thought of it yet; let me know your ideas, what you’re looking for, and we can help get the data needed to make good decisions in running PSL.
**Â First, we looked at how many messages each member has sent; then, we aggregated it to see the frequency at which 1, or 2, or 15 messages were sent (how many people sent exactly this number of messages), noting whether they were new messages or replies; finally, we took each conversation (message thread) and counted the number of messages inside it, to determine the number of conversations which contained 1, or 2, or 15 messages (frequency of thread lengths).
0 Responses
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.